You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by MikeThomsen <gi...@git.apache.org> on 2017/08/26 14:38:46 UTC
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
GitHub user MikeThomsen opened a pull request:
https://github.com/apache/nifi/pull/2113
NIFI-4325 Added new processor that uses the JSON DSL.
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/MikeThomsen/nifi NIFI-4325
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/2113.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2113
----
commit 4859d821bfd03f1b3999c163bb9d9bf5e6e0597e
Author: Mike Thomsen <mi...@gmail.com>
Date: 2017-08-18T17:27:29Z
NIFI-4325 Added new processor that uses the JSON DSL.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on the issue:
https://github.com/apache/nifi/pull/2113
If you're looking to run integration tests against a ES instance (from the Maven plugin), perhaps you can split out the integration tests into a new file called ITJsonQueryElasticsearch5, rather than having the @Ignore or running integration tests during the unit test phase. I realize the others aren't done this way, but if that Maven plugin works well, I think we should consider a separate Jira/PR to refactor the tests into proper integration vs unit tests, so they can be run during their corresponding lifecycle (integration tests aren't run during a normal test phase, e.g.)
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r157903403
--- Diff: nifi-nar-bundles/nifi-standard-services/nifi-elasticsearch-client-service-bundle/nifi-elasticsearch-client-service-nar/src/main/resources/META-INF/NOTICE ---
@@ -0,0 +1,332 @@
+nifi-hbase_1_1_2-client-service-nar
--- End diff --
This appears to be a copy from a different NAR ;) The first line should be updated, and the rest of the NOTICES below should only be included if we are including the component(s) in the artifacts.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r149121878
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+ public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship. " +
+ "This applies even if you select the \"break up hits\" option to send individual hits to the " +
+ "\"hits\" relationship.").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el5-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el5-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .defaultValue("")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(Validator.VALID)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el5-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax.")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final AllowableValue BREAK_UP_YES = new AllowableValue(
+ "breakup-yes",
+ "Yes",
+ "Break up results."
+ );
+ public static final AllowableValue BREAK_UP_HITS_NO = new AllowableValue(
+ "breakup-no",
+ "No",
+ "Don't break up results."
+ );
+
+ public static final PropertyDescriptor BREAK_UP_HITS = new PropertyDescriptor.Builder()
+ .name("el5-break-up-hits")
+ .displayName("Break up search results")
+ .description("Break up search results into one flowfile per result.")
+ .allowableValues(BREAK_UP_HITS_NO, BREAK_UP_YES)
+ .defaultValue(BREAK_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+ public static final PropertyDescriptor BREAK_UP_AGGREGATIONS = new PropertyDescriptor.Builder()
+ .name("el5-break-up-aggregations")
+ .displayName("Break up aggregation results")
+ .description("Break up aggregation results into one flowfile per result.")
+ .allowableValues(BREAK_UP_HITS_NO, BREAK_UP_YES)
+ .defaultValue(BREAK_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+
+ public static final PropertyDescriptor CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name("el5-client-service")
+ .displayName("Client Service")
+ .description("An ElasticSearch client service to use for running queries.")
+ .identifiesControllerService(ElasticSearchClientService.class)
+ .required(true)
+ .build();
+
+ private static final Set<Relationship> relationships;
+ private static final List<PropertyDescriptor> propertyDescriptors;
+
+ private ElasticSearchClientService clientService;
+
+ static {
+ final Set<Relationship> _rels = new HashSet<>();
+ _rels.add(REL_SUCCESS);
+ _rels.add(REL_FAILURE);
+ _rels.add(REL_HITS);
+ _rels.add(REL_AGGREGATIONS);
+ relationships = Collections.unmodifiableSet(_rels);
+
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(QUERY);
+ descriptors.add(INDEX);
+ descriptors.add(TYPE);
+ descriptors.add(CLIENT_SERVICE);
+ descriptors.add(BREAK_UP_HITS);
+ descriptors.add(BREAK_UP_AGGREGATIONS);
+
+ propertyDescriptors = Collections.unmodifiableList(descriptors);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return propertyDescriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ clientService = context.getProperty(CLIENT_SERVICE).asControllerService(ElasticSearchClientService.class);
+ }
+
+ @OnUnscheduled
+ public void onUnscheduled() {
+ this.clientService = null;
+ }
+
+
+ private ObjectMapper mapper = new ObjectMapper();
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ return;
+ }
+
+ try {
+
+ String query = context.getProperty(QUERY).evaluateAttributeExpressions(flowFile).getValue();
+ String index = context.getProperty(INDEX).evaluateAttributeExpressions(flowFile).getValue();
+ String type = context.getProperty(TYPE).evaluateAttributeExpressions(flowFile).getValue();
+
+ Optional<ElasticSearchResponse> resp = clientService.search(query, index, type);
+ ElasticSearchResponse response = resp.get();
+
+ List<FlowFile> hitsFlowFiles = handleHits(response.getHits(), context, session, flowFile);
+ List<FlowFile> aggsFlowFiles = handleAggregations(response.getAggregations(), context, session, flowFile);
+
+ if (hitsFlowFiles.size() > 0) {
+ session.transfer(hitsFlowFiles, REL_HITS);
+ for (FlowFile ff : hitsFlowFiles) {
+ session.getProvenanceReporter().create(ff);
+ }
+ }
+
+ if (aggsFlowFiles.size() > 0) {
+ session.transfer(aggsFlowFiles, REL_AGGREGATIONS);
+ for (FlowFile ff : aggsFlowFiles) {
+ session.getProvenanceReporter().create(ff);
+ }
+ }
+ session.transfer(flowFile, REL_SUCCESS);
+ } catch (Exception ex) {
+ getLogger().error("Error processing flowfile.", ex);
+ session.transfer(flowFile, REL_FAILURE);
+ }
+ }
+
+ private FlowFile writeAggregationFlowFileContents(String name, String json, ProcessSession session, FlowFile aggFlowFile) {
+ aggFlowFile = session.write(aggFlowFile, new OutputStreamCallback() {
+ @Override
+ public void process(OutputStream out) throws IOException {
+ out.write(json.getBytes());
+ }
+ });
+ if (name != null) {
+ aggFlowFile = session.putAttribute(aggFlowFile, "aggregation.name", name);
--- End diff --
Recommend adding a @WritesAttributes annotation to the processor class containing any attributes like aggregation.name, mime.type, etc. that may be added/updated by this processor
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r149120354
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+ public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship. " +
+ "This applies even if you select the \"break up hits\" option to send individual hits to the " +
+ "\"hits\" relationship.").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el5-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el5-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .defaultValue("")
--- End diff --
If this is not a required field, you can remove the defaultValue() and replace the VALID validator with StandardValidators.NON_EMPTY_VALIDATOR. Not a requirement, just a suggestion to make it more consistent with other properties like this.
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
Thanks. I'll get to these some time today or tomorrow.
FYI Elastic has some official statements on the future direction of the Java client here: https://www.elastic.co/blog/state-of-the-official-elasticsearch-java-clients
The other processors will need reviewing to make sure they look good.
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
Ok. It should all be there now.
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on the issue:
https://github.com/apache/nifi/pull/2113
Thanks @MikeThomsen I should be able to take another look later tonight
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
I discovered a slick plugin for Maven that lets you cleanly run ElasticSearch in the background during integration tests. I added that to the test setup under a profile. So I have some cleanup to do on this branch to bring that in. I think it'll be worth it.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175271738
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java ---
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.PrintWriter;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends AbstractControllerService implements ElasticSearchClientService {
+ private ObjectMapper mapper = new ObjectMapper();
+
+ private List<PropertyDescriptor> properties;
--- End diff --
Done.
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
@JPercivall changes checked in.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r157902770
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+ @WritesAttribute(attribute = "mime.type", description = "application/json"),
+ @WritesAttribute(attribute = "aggregation.name", description = "The name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder().name("original")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship. " +
+ "This applies even if you select the \"split up hits\" option to send individual hits to the " +
+ "\"hits\" relationship.").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el5-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el5-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el5-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax. Ex: " +
+ "{\n" +
+ "\t\"query\": {\n" +
+ "\t\t\"match\": {\n" +
+ "\t\t\t\"name\": \"John Smith\"\n" +
+ "\t\t}\n" +
+ "\t}\n" +
+ "}")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final AllowableValue SPLIT_UP_YES = new AllowableValue(
+ "splitUp-yes",
+ "Yes",
+ "Split up results."
+ );
+ public static final AllowableValue SPLIT_UP_HITS_NO = new AllowableValue(
+ "splitUp-no",
+ "No",
+ "Don't split up results."
+ );
+
+ public static final PropertyDescriptor SPLIT_UP_HITS = new PropertyDescriptor.Builder()
+ .name("el5-split-up-hits")
+ .displayName("Split up search results")
+ .description("Split up search results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+ public static final PropertyDescriptor SPLIT_UP_AGGREGATIONS = new PropertyDescriptor.Builder()
+ .name("el5-split-up-aggregations")
+ .displayName("Split up aggregation results")
+ .description("Split up aggregation results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+
+ public static final PropertyDescriptor CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name("el5-client-service")
+ .displayName("Client Service")
+ .description("An ElasticSearch client service to use for running queries.")
+ .identifiesControllerService(ElasticSearchClientService.class)
+ .required(true)
+ .build();
+
+ private static final Set<Relationship> relationships;
+ private static final List<PropertyDescriptor> propertyDescriptors;
+
+ private ElasticSearchClientService clientService;
+
+ static {
+ final Set<Relationship> _rels = new HashSet<>();
+ _rels.add(REL_ORIGINAL);
+ _rels.add(REL_FAILURE);
+ _rels.add(REL_HITS);
+ _rels.add(REL_AGGREGATIONS);
+ relationships = Collections.unmodifiableSet(_rels);
+
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(QUERY);
+ descriptors.add(INDEX);
+ descriptors.add(TYPE);
+ descriptors.add(CLIENT_SERVICE);
+ descriptors.add(SPLIT_UP_HITS);
+ descriptors.add(SPLIT_UP_AGGREGATIONS);
+
+ propertyDescriptors = Collections.unmodifiableList(descriptors);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return propertyDescriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ clientService = context.getProperty(CLIENT_SERVICE).asControllerService(ElasticSearchClientService.class);
+ }
+
+ @OnUnscheduled
+ public void onUnscheduled() {
+ this.clientService = null;
+ }
+
+
+ private ObjectMapper mapper = new ObjectMapper();
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
--- End diff --
Please see my other comment on allowing optional input flow files :)
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r150726237
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+ @WritesAttribute(attribute = "mime.type"),
--- End diff --
This should have a description that the mime.type will be set to "application/json"
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
@mattyb149 I think everything should be up to date now.
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
Thanks.
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
@JPercivall Thanks for the feedback. I'll get to working on these. WRT:
> Lastly, it's preferred if you don't squash your commits every time. If you don't, that allows the reviewer to more easily see exactly what changed since they last reviewed it. Also allows reviewers to see how the PR evolved over time in response to different comments.
Ok, I can do that from now on. Shouldn't they be squashed before a merge into master?
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175264784
--- Diff: nifi-nar-bundles/nifi-standard-services/pom.xml ---
@@ -15,6 +15,18 @@
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
+ <dependencies>
--- End diff --
This dependencies section shouldn't be needed here. This is the parent pom for all of the standard services.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r143453629
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,451 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.impl.nio.client.HttpAsyncClientBuilder;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.codehaus.jackson.map.ObjectMapper;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
--- End diff --
Since the newer Java client is just a wrapper around the REST API, that should make it pretty robust to work against many ES 5 clusters, right? We've been telling folks that the other ES5 processors only support 5.0.x clusters due to the native nature of the transport client, and the ESHttp processors are right for other cluster versions. If this new processor should work against many ES5 clusters, and is using the REST API under the hood, I wonder if we should call it JsonQueryElasticsearch5Http for consistency, what do you think?
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
@joewitt alright, I'll see what trouble I can get into.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r143454165
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,451 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.impl.nio.client.HttpAsyncClientBuilder;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.codehaus.jackson.map.ObjectMapper;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+
+ private RestClient client;
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el5-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el5-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .defaultValue("")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(Validator.VALID)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el5-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax.")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor HTTP_HOSTS = new PropertyDescriptor.Builder()
+ .name("el5-http-hosts")
+ .displayName("HTTP Hosts")
+ .description("A comma-separated list of HTTP hosts that host ElasticSearch query nodes.")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PROP_SSL_CONTEXT_SERVICE = new PropertyDescriptor.Builder()
+ .name("el5-ssl-context-service")
+ .displayName("SSL Context Service")
+ .description("The SSL Context Service used to provide client certificate information for TLS/SSL "
+ + "connections. This service only applies if the Elasticsearch endpoint(s) have been secured with TLS/SSL.")
+ .required(false)
+ .identifiesControllerService(SSLContextService.class)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .name("el5-username")
+ .displayName("Username")
+ .description("The username to use with XPack security.")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .name("el5-password")
+ .displayName("Password")
+ .description("The password to use with XPack security.")
+ .required(false)
+ .sensitive(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor CONNECT_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("el5-connect-timeout")
+ .displayName("Connect timeout")
+ .description("Controls the amount of time, in milliseconds, before a timeout occurs when trying to connect.")
+ .required(true)
+ .defaultValue("5000")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor SOCKET_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("el5-socket-timeout")
+ .displayName("Read timeout")
+ .description("Controls the amount of time, in milliseconds, before a timeout occurs when waiting for a response.")
+ .required(true)
+ .defaultValue("60000")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor RETRY_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("el5-retry-timeout")
+ .displayName("Retry timeout")
+ .description("Controls the amount of time, in milliseconds, before a timeout occurs when retrying the operation.")
+ .required(true)
+ .defaultValue("60000")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final AllowableValue BREAK_UP_YES = new AllowableValue(
+ "breakup-yes",
+ "Yes",
+ "Break up results."
+ );
+ public static final AllowableValue BREAK_UP_HITS_NO = new AllowableValue(
+ "breakup-no",
+ "No",
+ "Don't break up results."
+ );
+
+ public static final PropertyDescriptor BREAK_UP_HITS = new PropertyDescriptor.Builder()
+ .name("el5-break-up-hits")
+ .displayName("Break up search results")
+ .description("Break up search results into one flowfile per result.")
+ .allowableValues(BREAK_UP_HITS_NO, BREAK_UP_YES)
+ .defaultValue(BREAK_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+ public static final PropertyDescriptor BREAK_UP_AGGREGATIONS = new PropertyDescriptor.Builder()
+ .name("el5-break-up-aggregations")
+ .displayName("Break up aggregation results")
+ .description("Break up aggregation results into one flowfile per result.")
+ .allowableValues(BREAK_UP_HITS_NO, BREAK_UP_YES)
+ .defaultValue(BREAK_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+
+ private static final Set<Relationship> relationships;
+ private static final List<PropertyDescriptor> propertyDescriptors;
+
+ static {
+ final Set<Relationship> _rels = new HashSet<>();
+ _rels.add(REL_SUCCESS);
+ _rels.add(REL_FAILURE);
+ _rels.add(REL_HITS);
+ _rels.add(REL_AGGREGATIONS);
+ relationships = Collections.unmodifiableSet(_rels);
+
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(PROP_SSL_CONTEXT_SERVICE);
+ descriptors.add(CONNECT_TIMEOUT);
+ descriptors.add(SOCKET_TIMEOUT);
+ descriptors.add(RETRY_TIMEOUT);
+ descriptors.add(USERNAME);
+ descriptors.add(PASSWORD);
+ descriptors.add(HTTP_HOSTS);
+ descriptors.add(QUERY);
+ descriptors.add(INDEX);
+ descriptors.add(TYPE);
+ descriptors.add(BREAK_UP_HITS);
+ descriptors.add(BREAK_UP_AGGREGATIONS);
+
+ propertyDescriptors = Collections.unmodifiableList(descriptors);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return propertyDescriptors;
+ }
+
+
+ @OnUnscheduled
+ public void onUnscheduled() {
+ this.client = null;
+ }
+
+ private void setupClient(ProcessContext context) throws Exception {
+ final String hosts = context.getProperty(HTTP_HOSTS).evaluateAttributeExpressions().getValue();
+ String[] hostsSplit = hosts.split(",[\\s]*");
+ final SSLContextService sslService =
+ context.getProperty(PROP_SSL_CONTEXT_SERVICE).asControllerService(SSLContextService.class);
+ final String username = context.getProperty(USERNAME).evaluateAttributeExpressions().getValue();
+ final String password = context.getProperty(PASSWORD).evaluateAttributeExpressions().getValue();
+
+ final Integer connectTimeout = context.getProperty(CONNECT_TIMEOUT).asInteger();
+ final Integer readTimeout = context.getProperty(SOCKET_TIMEOUT).asInteger();
+ final Integer retryTimeout = context.getProperty(RETRY_TIMEOUT).asInteger();
+
+ HttpHost[] hh = new HttpHost[hostsSplit.length];
+ for (int x = 0; x < hh.length; x++) {
+ URL u = new URL(hostsSplit[x]);
+ hh[x] = new HttpHost(u.getHost(), u.getPort(), u.getProtocol());
+ }
+
+ RestClientBuilder builder = RestClient.builder(hh)
+ .setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
+ @Override
+ public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
+ if (sslService != null) {
+ try {
+ KeyStore keyStore = KeyStore.getInstance(sslService.getKeyStoreType());
+ KeyStore trustStore = KeyStore.getInstance("JKS");
+
+ try (final InputStream is = new FileInputStream(sslService.getKeyStoreFile())) {
+ keyStore.load(is, sslService.getKeyStorePassword().toCharArray());
+ }
+
+ try (final InputStream is = new FileInputStream(sslService.getTrustStoreFile())) {
+ trustStore.load(is, sslService.getTrustStorePassword().toCharArray());
+ }
+
+ final KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory
+ .getDefaultAlgorithm());
+ kmf.init(keyStore, sslService.getKeyStorePassword().toCharArray());
+ final TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory
+ .getDefaultAlgorithm());
+ tmf.init(keyStore);
+ SSLContext context = SSLContext.getInstance(sslService.getSslAlgorithm());
+ context.init(kmf.getKeyManagers(), tmf.getTrustManagers(), new SecureRandom());
+ httpClientBuilder = httpClientBuilder.setSSLContext(context);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ if (username != null && password != null) {
+ final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
+ credentialsProvider.setCredentials(AuthScope.ANY,
+ new UsernamePasswordCredentials(username, password));
+ httpClientBuilder = httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
+ }
+
+ return httpClientBuilder;
+ }
+ })
+ .setRequestConfigCallback(requestConfigBuilder -> requestConfigBuilder.setConnectTimeout(connectTimeout)
+ .setSocketTimeout(readTimeout))
+ .setMaxRetryTimeoutMillis(retryTimeout);
+
+ this.client = builder.build();
+ }
+
+ private Response runQuery(String query, String index, String type) throws Exception {
+ StringBuilder sb = new StringBuilder()
+ .append("/")
+ .append(index);
+ if (type != null && !type.equals("")) {
+ sb.append("/")
+ .append(type);
+ }
+
+ sb.append("/_search");
+
+ HttpEntity queryEntity = new NStringEntity(query, ContentType.APPLICATION_JSON);
+
+ return client.performRequest("POST", sb.toString(), Collections.emptyMap(), queryEntity);
+ }
+
+ private ObjectMapper mapper = new ObjectMapper();
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ return;
+ }
+
+ try {
+ if (this.client == null) {
+ setupClient(context);
+ }
+
+ String query = context.getProperty(QUERY).evaluateAttributeExpressions(flowFile).getValue();
+ String index = context.getProperty(INDEX).evaluateAttributeExpressions(flowFile).getValue();
+ String type = context.getProperty(TYPE).evaluateAttributeExpressions(flowFile).getValue();
+
+ Response resp = runQuery(query, index, type);
+
+ InputStream content = resp.getEntity().getContent();
+ String rawJson = IOUtils.toString(content, "UTF-8");
--- End diff --
Is the response guaranteed to be in UTF-8 format?
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175265212
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java ---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+ @WritesAttribute(attribute = "mime.type", description = "application/json"),
+ @WritesAttribute(attribute = "aggregation.name", description = "The name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
--- End diff --
Related to the note about pagination, should add something indicating that very large searches should be done as the response is loaded into memory.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r157902546
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+ @WritesAttribute(attribute = "mime.type", description = "application/json"),
+ @WritesAttribute(attribute = "aggregation.name", description = "The name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
--- End diff --
It looks like you're not using the incoming flow file contents, in which case we might relax this constraint to INPUT_ALLOWED, and if no flow file is available (you can evaluate EL against a null FF) you can create one, what do you think?
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r157903679
--- Diff: nifi-nar-bundles/nifi-standard-services/nifi-elasticsearch-client-service-bundle/nifi-elasticsearch-client-service-nar/src/main/resources/META-INF/LICENSE ---
@@ -0,0 +1,357 @@
+
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+
+APACHE NIFI SUBCOMPONENTS:
+
+The Apache NiFi project contains subcomponents with separate copyright
+notices and license terms. Your use of the source code for the these
+subcomponents is subject to the terms and conditions of the following
+licenses.
+
+The binary distribution of this product bundles 'Jcodings' under an MIT style
--- End diff --
The rest of this file should be inspected / updated depending on which components are actually included in the artifact(s). Please apply this to any NAR/module that has a LICENSE, and see my other comment on NOTICEs, thanks!
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r143454076
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,451 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.impl.nio.client.HttpAsyncClientBuilder;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.codehaus.jackson.map.ObjectMapper;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+
+ private RestClient client;
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el5-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el5-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .defaultValue("")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(Validator.VALID)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el5-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax.")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor HTTP_HOSTS = new PropertyDescriptor.Builder()
+ .name("el5-http-hosts")
+ .displayName("HTTP Hosts")
+ .description("A comma-separated list of HTTP hosts that host ElasticSearch query nodes.")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PROP_SSL_CONTEXT_SERVICE = new PropertyDescriptor.Builder()
+ .name("el5-ssl-context-service")
+ .displayName("SSL Context Service")
+ .description("The SSL Context Service used to provide client certificate information for TLS/SSL "
+ + "connections. This service only applies if the Elasticsearch endpoint(s) have been secured with TLS/SSL.")
+ .required(false)
+ .identifiesControllerService(SSLContextService.class)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .name("el5-username")
+ .displayName("Username")
+ .description("The username to use with XPack security.")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .name("el5-password")
+ .displayName("Password")
+ .description("The password to use with XPack security.")
+ .required(false)
+ .sensitive(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor CONNECT_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("el5-connect-timeout")
+ .displayName("Connect timeout")
+ .description("Controls the amount of time, in milliseconds, before a timeout occurs when trying to connect.")
+ .required(true)
+ .defaultValue("5000")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor SOCKET_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("el5-socket-timeout")
+ .displayName("Read timeout")
+ .description("Controls the amount of time, in milliseconds, before a timeout occurs when waiting for a response.")
+ .required(true)
+ .defaultValue("60000")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor RETRY_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("el5-retry-timeout")
+ .displayName("Retry timeout")
+ .description("Controls the amount of time, in milliseconds, before a timeout occurs when retrying the operation.")
+ .required(true)
+ .defaultValue("60000")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final AllowableValue BREAK_UP_YES = new AllowableValue(
+ "breakup-yes",
+ "Yes",
+ "Break up results."
+ );
+ public static final AllowableValue BREAK_UP_HITS_NO = new AllowableValue(
+ "breakup-no",
+ "No",
+ "Don't break up results."
+ );
+
+ public static final PropertyDescriptor BREAK_UP_HITS = new PropertyDescriptor.Builder()
+ .name("el5-break-up-hits")
+ .displayName("Break up search results")
+ .description("Break up search results into one flowfile per result.")
+ .allowableValues(BREAK_UP_HITS_NO, BREAK_UP_YES)
+ .defaultValue(BREAK_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+ public static final PropertyDescriptor BREAK_UP_AGGREGATIONS = new PropertyDescriptor.Builder()
+ .name("el5-break-up-aggregations")
+ .displayName("Break up aggregation results")
+ .description("Break up aggregation results into one flowfile per result.")
+ .allowableValues(BREAK_UP_HITS_NO, BREAK_UP_YES)
+ .defaultValue(BREAK_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+
+ private static final Set<Relationship> relationships;
+ private static final List<PropertyDescriptor> propertyDescriptors;
+
+ static {
+ final Set<Relationship> _rels = new HashSet<>();
+ _rels.add(REL_SUCCESS);
+ _rels.add(REL_FAILURE);
+ _rels.add(REL_HITS);
+ _rels.add(REL_AGGREGATIONS);
+ relationships = Collections.unmodifiableSet(_rels);
+
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(PROP_SSL_CONTEXT_SERVICE);
+ descriptors.add(CONNECT_TIMEOUT);
+ descriptors.add(SOCKET_TIMEOUT);
+ descriptors.add(RETRY_TIMEOUT);
+ descriptors.add(USERNAME);
+ descriptors.add(PASSWORD);
+ descriptors.add(HTTP_HOSTS);
+ descriptors.add(QUERY);
+ descriptors.add(INDEX);
+ descriptors.add(TYPE);
+ descriptors.add(BREAK_UP_HITS);
+ descriptors.add(BREAK_UP_AGGREGATIONS);
+
+ propertyDescriptors = Collections.unmodifiableList(descriptors);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return propertyDescriptors;
+ }
+
+
+ @OnUnscheduled
+ public void onUnscheduled() {
+ this.client = null;
+ }
+
+ private void setupClient(ProcessContext context) throws Exception {
+ final String hosts = context.getProperty(HTTP_HOSTS).evaluateAttributeExpressions().getValue();
+ String[] hostsSplit = hosts.split(",[\\s]*");
+ final SSLContextService sslService =
+ context.getProperty(PROP_SSL_CONTEXT_SERVICE).asControllerService(SSLContextService.class);
+ final String username = context.getProperty(USERNAME).evaluateAttributeExpressions().getValue();
+ final String password = context.getProperty(PASSWORD).evaluateAttributeExpressions().getValue();
+
+ final Integer connectTimeout = context.getProperty(CONNECT_TIMEOUT).asInteger();
+ final Integer readTimeout = context.getProperty(SOCKET_TIMEOUT).asInteger();
+ final Integer retryTimeout = context.getProperty(RETRY_TIMEOUT).asInteger();
+
+ HttpHost[] hh = new HttpHost[hostsSplit.length];
+ for (int x = 0; x < hh.length; x++) {
+ URL u = new URL(hostsSplit[x]);
+ hh[x] = new HttpHost(u.getHost(), u.getPort(), u.getProtocol());
+ }
+
+ RestClientBuilder builder = RestClient.builder(hh)
+ .setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
+ @Override
+ public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
+ if (sslService != null) {
+ try {
+ KeyStore keyStore = KeyStore.getInstance(sslService.getKeyStoreType());
+ KeyStore trustStore = KeyStore.getInstance("JKS");
--- End diff --
I think we should be using sslService.getTrustStoreType() here, in case it is set to something other than JKS
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
@mattyb149 I think all of the change requests are done.
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on the issue:
https://github.com/apache/nifi/pull/2113
Looks good, was able to do a couple simple queries and aggregations. I wasn't able to test HTTPS though as I don't have access to the X-Pack. Were you able to do so @MikeThomsen?
The only lingering issue, that I've mentioned multiple times now, is to add a note about the entire body of the response being loaded into memory.
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on the issue:
https://github.com/apache/nifi/pull/2113
Hey @MikeThomsen, just finished up another round of reviewing. Again, mostly focused on the error handling portion.
I hope my review doesn't come off in the wrong way, there is a lot of great work here and I just want to make sure the usability is top notch. Error handling is a huge focus in NiFi because we're writing up a toolbox of processors for potentially non-technical users to work with however they want. Those users need to be able to see what they messed up and what went wrong for efficient and effective rectification.
Thanks for the hard work, and the willingness to iterate and learn!
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175304809
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java ---
@@ -121,7 +119,7 @@ private void setupClient(ConfigurationContext context) throws Exception {
RestClientBuilder builder = RestClient.builder(hh)
.setHttpClientConfigCallback(httpClientBuilder -> {
- if (sslService != null) {
+ if (sslService != null && sslService.isKeyStoreConfigured() && sslService.isTrustStoreConfigured()) {
try {
--- End diff --
Given that this is a callback, we probably want to move as much of the logic out of here as possible. That way we can better handle any errors and reduce performance cost if this is called multiple times. Currently, if there is an error which hits line 145, the error will be caught and logged but it will continue to create the http client as if nothing went wrong. What we should probably do is have a final SSLContext which gets created above, bubbling up any errors so that the CS doesn't enable in a bad state, and set the SSLContext same way as below in the context.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175265070
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java ---
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.PrintWriter;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends AbstractControllerService implements ElasticSearchClientService {
+ private ObjectMapper mapper = new ObjectMapper();
+
+ private List<PropertyDescriptor> properties;
+
+ private RestClient client;
+
+ private String url;
+
+ @Override
+ protected void init(ControllerServiceInitializationContext config) {
+ properties = new ArrayList<>();
+ properties.add(ElasticSearchClientService.HTTP_HOSTS);
+ properties.add(ElasticSearchClientService.USERNAME);
+ properties.add(ElasticSearchClientService.PASSWORD);
+ properties.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+ properties.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+ properties.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+ properties.add(ElasticSearchClientService.RETRY_TIMEOUT);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return properties;
+ }
+
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws InitializationException {
+ try {
+ setupClient(context);
+ } catch (Exception ex) {
+ getLogger().error("Could not initialize ElasticSearch client.", ex);
+ throw new InitializationException(ex);
+ }
+ }
+
+ @OnDisabled
+ public void onDisabled() throws IOException {
+ this.client.close();
+ this.url = null;
+ }
+
+ private void setupClient(ConfigurationContext context) throws Exception {
+ final String hosts = context.getProperty(HTTP_HOSTS).evaluateAttributeExpressions().getValue();
+ String[] hostsSplit = hosts.split(",[\\s]*");
+ this.url = hostsSplit[0];
+ final SSLContextService sslService =
+ context.getProperty(PROP_SSL_CONTEXT_SERVICE).asControllerService(SSLContextService.class);
+ final String username = context.getProperty(USERNAME).evaluateAttributeExpressions().getValue();
+ final String password = context.getProperty(PASSWORD).evaluateAttributeExpressions().getValue();
+
+ final Integer connectTimeout = context.getProperty(CONNECT_TIMEOUT).asInteger();
+ final Integer readTimeout = context.getProperty(SOCKET_TIMEOUT).asInteger();
+ final Integer retryTimeout = context.getProperty(RETRY_TIMEOUT).asInteger();
+
+ HttpHost[] hh = new HttpHost[hostsSplit.length];
+ for (int x = 0; x < hh.length; x++) {
+ URL u = new URL(hostsSplit[x]);
+ hh[x] = new HttpHost(u.getHost(), u.getPort(), u.getProtocol());
+ }
+
+ RestClientBuilder builder = RestClient.builder(hh)
+ .setHttpClientConfigCallback(httpClientBuilder -> {
+ if (sslService != null) {
+ try {
+ KeyStore keyStore = KeyStore.getInstance(sslService.getKeyStoreType());
+ KeyStore trustStore = KeyStore.getInstance("JKS");
+
+ try (final InputStream is = new FileInputStream(sslService.getKeyStoreFile())) {
+ keyStore.load(is, sslService.getKeyStorePassword().toCharArray());
+ }
+
+ try (final InputStream is = new FileInputStream(sslService.getTrustStoreFile())) {
+ trustStore.load(is, sslService.getTrustStorePassword().toCharArray());
+ }
+
+ final KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory
+ .getDefaultAlgorithm());
+ kmf.init(keyStore, sslService.getKeyStorePassword().toCharArray());
+ final TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory
+ .getDefaultAlgorithm());
+ tmf.init(keyStore);
+ SSLContext context1 = SSLContext.getInstance(sslService.getSslAlgorithm());
+ context1.init(kmf.getKeyManagers(), tmf.getTrustManagers(), new SecureRandom());
+ httpClientBuilder = httpClientBuilder.setSSLContext(context1);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
--- End diff --
Why throw RTE here? (and below)
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175264845
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java ---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+ @WritesAttribute(attribute = "mime.type", description = "application/json"),
+ @WritesAttribute(attribute = "aggregation.name", description = "The name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder().name("original")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship. " +
+ "This applies even if you select the \"split up hits\" option to send individual hits to the " +
+ "\"hits\" relationship.").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el-rest-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el-rest-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el-rest-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax. Ex: " +
+ "{\n" +
+ "\t\"query\": {\n" +
+ "\t\t\"match\": {\n" +
+ "\t\t\t\"name\": \"John Smith\"\n" +
+ "\t\t}\n" +
+ "\t}\n" +
+ "}")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUERY_ATTRIBUTE = new PropertyDescriptor.Builder()
+ .name("el-query-attribute")
+ .displayName("Query Attribute")
+ .description("If set, the executed query will be set on each result flowfile in the specified attribute.")
+ .expressionLanguageSupported(true)
+ .addValidator(Validator.VALID)
+ .required(false)
+ .build();
+
+ public static final AllowableValue SPLIT_UP_YES = new AllowableValue(
+ "splitUp-yes",
+ "Yes",
+ "Split up results."
+ );
+ public static final AllowableValue SPLIT_UP_HITS_NO = new AllowableValue(
+ "splitUp-no",
+ "No",
+ "Don't split up results."
+ );
+
+ public static final PropertyDescriptor SPLIT_UP_HITS = new PropertyDescriptor.Builder()
+ .name("el-rest-split-up-hits")
+ .displayName("Split up search results")
+ .description("Split up search results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+ public static final PropertyDescriptor SPLIT_UP_AGGREGATIONS = new PropertyDescriptor.Builder()
+ .name("el-rest-split-up-aggregations")
+ .displayName("Split up aggregation results")
+ .description("Split up aggregation results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+
+ public static final PropertyDescriptor CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name("el-rest-client-service")
+ .displayName("Client Service")
+ .description("An ElasticSearch client service to use for running queries.")
+ .identifiesControllerService(ElasticSearchClientService.class)
+ .required(true)
+ .build();
+
+ private static final Set<Relationship> relationships;
+ private static final List<PropertyDescriptor> propertyDescriptors;
+
+ private ElasticSearchClientService clientService;
+
+ static {
+ final Set<Relationship> _rels = new HashSet<>();
+ _rels.add(REL_ORIGINAL);
+ _rels.add(REL_FAILURE);
+ _rels.add(REL_HITS);
+ _rels.add(REL_AGGREGATIONS);
+ relationships = Collections.unmodifiableSet(_rels);
+
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(QUERY);
+ descriptors.add(QUERY_ATTRIBUTE);
+ descriptors.add(INDEX);
+ descriptors.add(TYPE);
+ descriptors.add(CLIENT_SERVICE);
+ descriptors.add(SPLIT_UP_HITS);
+ descriptors.add(SPLIT_UP_AGGREGATIONS);
+
+ propertyDescriptors = Collections.unmodifiableList(descriptors);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return propertyDescriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ clientService = context.getProperty(CLIENT_SERVICE).asControllerService(ElasticSearchClientService.class);
+ }
+
+ @OnStopped
+ public void onStopped() {
+ this.clientService = null;
+ }
+
+
+ private final ObjectMapper mapper = new ObjectMapper();
+
+ private String getQuery(FlowFile input, ProcessContext context, ProcessSession session) throws IOException {
+ String retVal = null;
+ if (context.getProperty(QUERY).isSet()) {
+ retVal = context.getProperty(QUERY).evaluateAttributeExpressions(input).getValue();
+ } else if (input != null) {
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+ session.exportTo(input, out);
+ out.close();
+
+ retVal = new String(out.toByteArray());
+ }
+
+ return retVal;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile input = null;
+ if (context.hasIncomingConnection()) {
+ input = session.get();
+
+ if (input == null && context.hasNonLoopConnection()) {
+ return;
+ }
+ }
+
+ try {
+
+ final String query = getQuery(input, context, session);
+ final String index = context.getProperty(INDEX).evaluateAttributeExpressions(input).getValue();
+ final String type = context.getProperty(TYPE).evaluateAttributeExpressions(input).getValue();
+
+ Optional<SearchResponse> resp = clientService.search(query, index, type);
+ if (resp.isPresent()) {
+ SearchResponse response = resp.get();
+
+ List<FlowFile> hitsFlowFiles = handleHits(response.getHits(), context, session, input);
+ List<FlowFile> aggsFlowFiles = handleAggregations(response.getAggregations(), context, session, input);
+
+ if (hitsFlowFiles.size() > 0) {
+ for (FlowFile ff : hitsFlowFiles) {
+ session.transfer(ff, REL_HITS);
+ session.getProvenanceReporter().send(ff, clientService.getTransitUrl(index, type));
--- End diff --
This could be improved a bit by setting a single String and re-using instead it of recalculating for every flowfile hit (and agg hit).
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
2/3 builds pass now.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r157902041
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
--- End diff --
Lines 38 and 42 are unused imports (i.e. CheckStyle violations), but I can fix these on final merge if there are no other issues :)
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
@JPercivall Any chance we can close this out?
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
@mattyb149 I moved it to a new NAR and rebased against the latest master last night. So once this is committed, I'll start working on the new processors and service methods.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r150726016
--- Diff: nifi-assembly/pom.xml ---
@@ -338,6 +338,16 @@
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-elasticsearch-client-service-nar</artifactId>
+ <type>nar</type>
+ </dependency>
+<!-- <dependency>
--- End diff --
I mean the commented out dependency, looks like dead code and/or you moved it to the processor's POM?
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175271522
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java ---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+ @WritesAttribute(attribute = "mime.type", description = "application/json"),
+ @WritesAttribute(attribute = "aggregation.name", description = "The name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder().name("original")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship. " +
+ "This applies even if you select the \"split up hits\" option to send individual hits to the " +
+ "\"hits\" relationship.").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el-rest-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el-rest-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el-rest-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax. Ex: " +
+ "{\n" +
+ "\t\"query\": {\n" +
+ "\t\t\"match\": {\n" +
+ "\t\t\t\"name\": \"John Smith\"\n" +
+ "\t\t}\n" +
+ "\t}\n" +
+ "}")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUERY_ATTRIBUTE = new PropertyDescriptor.Builder()
+ .name("el-query-attribute")
+ .displayName("Query Attribute")
+ .description("If set, the executed query will be set on each result flowfile in the specified attribute.")
+ .expressionLanguageSupported(true)
+ .addValidator(Validator.VALID)
+ .required(false)
+ .build();
+
+ public static final AllowableValue SPLIT_UP_YES = new AllowableValue(
+ "splitUp-yes",
+ "Yes",
+ "Split up results."
+ );
+ public static final AllowableValue SPLIT_UP_HITS_NO = new AllowableValue(
+ "splitUp-no",
+ "No",
+ "Don't split up results."
+ );
+
+ public static final PropertyDescriptor SPLIT_UP_HITS = new PropertyDescriptor.Builder()
+ .name("el-rest-split-up-hits")
+ .displayName("Split up search results")
+ .description("Split up search results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+ public static final PropertyDescriptor SPLIT_UP_AGGREGATIONS = new PropertyDescriptor.Builder()
+ .name("el-rest-split-up-aggregations")
+ .displayName("Split up aggregation results")
+ .description("Split up aggregation results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+
+ public static final PropertyDescriptor CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name("el-rest-client-service")
+ .displayName("Client Service")
+ .description("An ElasticSearch client service to use for running queries.")
+ .identifiesControllerService(ElasticSearchClientService.class)
+ .required(true)
+ .build();
+
+ private static final Set<Relationship> relationships;
+ private static final List<PropertyDescriptor> propertyDescriptors;
+
+ private ElasticSearchClientService clientService;
+
+ static {
+ final Set<Relationship> _rels = new HashSet<>();
+ _rels.add(REL_ORIGINAL);
+ _rels.add(REL_FAILURE);
+ _rels.add(REL_HITS);
+ _rels.add(REL_AGGREGATIONS);
+ relationships = Collections.unmodifiableSet(_rels);
+
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(QUERY);
+ descriptors.add(QUERY_ATTRIBUTE);
+ descriptors.add(INDEX);
+ descriptors.add(TYPE);
+ descriptors.add(CLIENT_SERVICE);
+ descriptors.add(SPLIT_UP_HITS);
+ descriptors.add(SPLIT_UP_AGGREGATIONS);
+
+ propertyDescriptors = Collections.unmodifiableList(descriptors);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return propertyDescriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ clientService = context.getProperty(CLIENT_SERVICE).asControllerService(ElasticSearchClientService.class);
+ }
+
+ @OnStopped
+ public void onStopped() {
+ this.clientService = null;
+ }
+
+
+ private final ObjectMapper mapper = new ObjectMapper();
+
+ private String getQuery(FlowFile input, ProcessContext context, ProcessSession session) throws IOException {
+ String retVal = null;
+ if (context.getProperty(QUERY).isSet()) {
+ retVal = context.getProperty(QUERY).evaluateAttributeExpressions(input).getValue();
+ } else if (input != null) {
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+ session.exportTo(input, out);
+ out.close();
+
+ retVal = new String(out.toByteArray());
+ }
+
+ return retVal;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile input = null;
+ if (context.hasIncomingConnection()) {
+ input = session.get();
+
+ if (input == null && context.hasNonLoopConnection()) {
+ return;
+ }
+ }
+
+ try {
+
+ final String query = getQuery(input, context, session);
+ final String index = context.getProperty(INDEX).evaluateAttributeExpressions(input).getValue();
+ final String type = context.getProperty(TYPE).evaluateAttributeExpressions(input).getValue();
+
+ Optional<SearchResponse> resp = clientService.search(query, index, type);
+ if (resp.isPresent()) {
+ SearchResponse response = resp.get();
+
+ List<FlowFile> hitsFlowFiles = handleHits(response.getHits(), context, session, input);
+ List<FlowFile> aggsFlowFiles = handleAggregations(response.getAggregations(), context, session, input);
+
+ if (hitsFlowFiles.size() > 0) {
+ for (FlowFile ff : hitsFlowFiles) {
+ session.transfer(ff, REL_HITS);
+ session.getProvenanceReporter().send(ff, clientService.getTransitUrl(index, type));
--- End diff --
Done.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175271759
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java ---
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.PrintWriter;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends AbstractControllerService implements ElasticSearchClientService {
+ private ObjectMapper mapper = new ObjectMapper();
+
+ private List<PropertyDescriptor> properties;
+
+ private RestClient client;
+
+ private String url;
+
+ @Override
+ protected void init(ControllerServiceInitializationContext config) {
+ properties = new ArrayList<>();
+ properties.add(ElasticSearchClientService.HTTP_HOSTS);
+ properties.add(ElasticSearchClientService.USERNAME);
+ properties.add(ElasticSearchClientService.PASSWORD);
+ properties.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+ properties.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+ properties.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+ properties.add(ElasticSearchClientService.RETRY_TIMEOUT);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return properties;
+ }
+
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws InitializationException {
+ try {
+ setupClient(context);
+ } catch (Exception ex) {
+ getLogger().error("Could not initialize ElasticSearch client.", ex);
+ throw new InitializationException(ex);
+ }
+ }
+
+ @OnDisabled
+ public void onDisabled() throws IOException {
+ this.client.close();
+ this.url = null;
+ }
+
+ private void setupClient(ConfigurationContext context) throws Exception {
+ final String hosts = context.getProperty(HTTP_HOSTS).evaluateAttributeExpressions().getValue();
+ String[] hostsSplit = hosts.split(",[\\s]*");
+ this.url = hostsSplit[0];
+ final SSLContextService sslService =
+ context.getProperty(PROP_SSL_CONTEXT_SERVICE).asControllerService(SSLContextService.class);
+ final String username = context.getProperty(USERNAME).evaluateAttributeExpressions().getValue();
+ final String password = context.getProperty(PASSWORD).evaluateAttributeExpressions().getValue();
+
+ final Integer connectTimeout = context.getProperty(CONNECT_TIMEOUT).asInteger();
+ final Integer readTimeout = context.getProperty(SOCKET_TIMEOUT).asInteger();
+ final Integer retryTimeout = context.getProperty(RETRY_TIMEOUT).asInteger();
+
+ HttpHost[] hh = new HttpHost[hostsSplit.length];
+ for (int x = 0; x < hh.length; x++) {
+ URL u = new URL(hostsSplit[x]);
+ hh[x] = new HttpHost(u.getHost(), u.getPort(), u.getProtocol());
+ }
+
+ RestClientBuilder builder = RestClient.builder(hh)
+ .setHttpClientConfigCallback(httpClientBuilder -> {
+ if (sslService != null) {
+ try {
+ KeyStore keyStore = KeyStore.getInstance(sslService.getKeyStoreType());
+ KeyStore trustStore = KeyStore.getInstance("JKS");
+
+ try (final InputStream is = new FileInputStream(sslService.getKeyStoreFile())) {
+ keyStore.load(is, sslService.getKeyStorePassword().toCharArray());
+ }
+
+ try (final InputStream is = new FileInputStream(sslService.getTrustStoreFile())) {
+ trustStore.load(is, sslService.getTrustStorePassword().toCharArray());
+ }
+
+ final KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory
+ .getDefaultAlgorithm());
+ kmf.init(keyStore, sslService.getKeyStorePassword().toCharArray());
+ final TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory
+ .getDefaultAlgorithm());
+ tmf.init(keyStore);
+ SSLContext context1 = SSLContext.getInstance(sslService.getSslAlgorithm());
+ context1.init(kmf.getKeyManagers(), tmf.getTrustManagers(), new SecureRandom());
+ httpClientBuilder = httpClientBuilder.setSSLContext(context1);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ if (username != null && password != null) {
+ final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
+ credentialsProvider.setCredentials(AuthScope.ANY,
+ new UsernamePasswordCredentials(username, password));
+ httpClientBuilder = httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
+ }
+
+ return httpClientBuilder;
+ })
+ .setRequestConfigCallback(requestConfigBuilder -> {
+ try {
+ requestConfigBuilder.setConnectTimeout(connectTimeout);
+ requestConfigBuilder.setSocketTimeout(readTimeout);
+ return requestConfigBuilder;
+ } catch (Exception ex) {
+ try {
+ PrintWriter writer = new PrintWriter(new FileWriter("/tmp/out.log"));
--- End diff --
Something I wrote to make it easier to dump the raw query. Shouldn't be in there.
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
I set up a profile called integration-tests and renamed the file to JsonQueryElasticsearch_IT. It runs really well from the project folder, but for some reason running `mvn clean install -Pintegration-tests -pl :nifi-elasticsearch-5-processors` from the root folder causes surefire to be run twice (the first time passing, the second time failing because it stops ElasticSearch in between the two runs). When run from nifi-elasticsearch-5-processors it executes as expected.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r143295816
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,459 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.impl.nio.client.HttpAsyncClientBuilder;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.codehaus.jackson.map.ObjectMapper;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "elasticsearch 5", "fetch", "read", "get"})
+@CapabilityDescription("Retrieves a document from Elasticsearch using the specified connection properties and the "
+ + "identifier of the document to retrieve. If the cluster has been configured for authorization and/or secure "
+ + "transport (SSL/TLS), and the X-Pack plugin is available, secure connections can be made. This processor "
+ + "supports Elasticsearch 5.x clusters.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+
+ private RestClient client;
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el5-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el5-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .defaultValue("")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(Validator.VALID)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el5-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax.")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor HTTP_HOSTS = new PropertyDescriptor.Builder()
+ .name("el5-http-hosts")
+ .displayName("HTTP Hosts")
+ .description("A comma-separated list of HTTP hosts that host ElasticSearch query nodes.")
+ .required(true)
+ .expressionLanguageSupported(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PROP_SSL_CONTEXT_SERVICE = new PropertyDescriptor.Builder()
+ .name("el5-ssl-context-service")
+ .displayName("SSL Context Service")
+ .description("The SSL Context Service used to provide client certificate information for TLS/SSL "
+ + "connections. This service only applies if the Elasticsearch endpoint(s) have been secured with TLS/SSL.")
+ .required(false)
+ .identifiesControllerService(SSLContextService.class)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .name("el5-username")
+ .displayName("Username")
+ .description("The username to use with XPack security.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .name("el5-password")
+ .displayName("Password")
+ .description("The password to use with XPack security.")
+ .required(false)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor CONNECT_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("el5-connect-timeout")
+ .displayName("Connect timeout")
+ .description("Controls the amount of time, in milliseconds, before a timeout occurs when trying to connect.")
+ .required(true)
+ .defaultValue("5000")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor SOCKET_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("el5-socket-timeout")
+ .displayName("Read timeout")
+ .description("Controls the amount of time, in milliseconds, before a timeout occurs when waiting for a response.")
+ .required(true)
+ .defaultValue("60000")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor RETRY_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("el5-retry-timeout")
+ .displayName("Retry timeout")
+ .description("Controls the amount of time, in milliseconds, before a timeout occurs when retrying the operation.")
+ .required(true)
+ .defaultValue("60000")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final AllowableValue BREAK_UP_YES = new AllowableValue(
+ "breakup-yes",
+ "Yes",
+ "Break up results."
+ );
+ public static final AllowableValue BREAK_UP_HITS_NO = new AllowableValue(
+ "breakup-no",
+ "No",
+ "Don't break up results."
+ );
+
+ public static final PropertyDescriptor BREAK_UP_HITS = new PropertyDescriptor.Builder()
+ .name("el5-break-up-hits")
+ .displayName("Break up search results")
+ .description("Break up search results into one flowfile per result.")
+ .allowableValues(BREAK_UP_HITS_NO, BREAK_UP_YES)
+ .defaultValue(BREAK_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+ public static final PropertyDescriptor BREAK_UP_AGGREGATIONS = new PropertyDescriptor.Builder()
+ .name("el5-break-up-aggregations")
+ .displayName("Break up aggregation results")
+ .description("Break up aggregation results into one flowfile per result.")
+ .allowableValues(BREAK_UP_HITS_NO, BREAK_UP_YES)
+ .defaultValue(BREAK_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+
+ private static final Set<Relationship> relationships;
+ private static final List<PropertyDescriptor> propertyDescriptors;
+
+ static {
+ final Set<Relationship> _rels = new HashSet<>();
+ _rels.add(REL_SUCCESS);
+ _rels.add(REL_FAILURE);
+ _rels.add(REL_HITS);
+ _rels.add(REL_AGGREGATIONS);
+ relationships = Collections.unmodifiableSet(_rels);
+
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(PROP_SSL_CONTEXT_SERVICE);
+ descriptors.add(CONNECT_TIMEOUT);
+ descriptors.add(SOCKET_TIMEOUT);
+ descriptors.add(RETRY_TIMEOUT);
+ descriptors.add(USERNAME);
+ descriptors.add(PASSWORD);
+ descriptors.add(HTTP_HOSTS);
+ descriptors.add(QUERY);
+ descriptors.add(INDEX);
+ descriptors.add(TYPE);
+ descriptors.add(BREAK_UP_HITS);
+ descriptors.add(BREAK_UP_AGGREGATIONS);
+
+ propertyDescriptors = Collections.unmodifiableList(descriptors);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return propertyDescriptors;
+ }
+
+
+ @OnUnscheduled
+ public void onStartup() {
--- End diff --
Just a style note, but this might be a misnamed/confusing method for an @OnUnscheduled annotation
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on the issue:
https://github.com/apache/nifi/pull/2113
Hey @MikeThomsen, just reviewed the changes. They look good and I think we're almost there. One thing that is missing is the comment regarding pulling the results into memory.
Also, I tried to build NiFi and test the processor an instance but hit the following error when I attempted to start up NiFi (./bin/nifi.sh start). I tried rebasing it to the latest master but to no avail.
> 2018-03-24 02:06:53,171 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.elasticsearch.JsonQueryElasticsearch could not be instantiated
java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.elasticsearch.JsonQueryElasticsearch could not be instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:232)
at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at org.apache.nifi.nar.ExtensionManager.loadExtensions(ExtensionManager.java:148)
at org.apache.nifi.nar.ExtensionManager.discoverExtensions(ExtensionManager.java:123)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:771)
at org.apache.nifi.NiFi.<init>(NiFi.java:157)
at org.apache.nifi.NiFi.<init>(NiFi.java:71)
at org.apache.nifi.NiFi.main(NiFi.java:292)
Caused by: java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper
at org.apache.nifi.processors.elasticsearch.JsonQueryElasticsearch.<init>(JsonQueryElasticsearch.java:206)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
... 8 common frames omitted
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.ObjectMapper
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 15 common frames omitted
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r143294482
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,459 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.impl.nio.client.HttpAsyncClientBuilder;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.codehaus.jackson.map.ObjectMapper;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "elasticsearch 5", "fetch", "read", "get"})
--- End diff --
The tags should probably include "query" instead of "fetch" as does QueryElasticsearchHttp. Not sure whether they should include "json" or not, I'm fine with that either way :)
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
@mattyb149 I think everything except the proxy issue is now in there.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r148787152
--- Diff: nifi-assembly/pom.xml ---
@@ -338,6 +338,16 @@
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-elasticsearch-client-service-nar</artifactId>
+ <type>nar</type>
+ </dependency>
+<!-- <dependency>
--- End diff --
If not needed please remove
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
@JPercivall FYI, part of the reason I want to get this done is I'm planning on doing a whole new set of ES processors that are based on bring the total CRUD functionality over to the official Elastic-provided APIs. The current set of processors use the transport API (deprecated) and make manual REST calls that AFAIK don't do master detection and stuff like that which comes baked-in with the Elastic APIs.
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
@mattyb149 I think I got your changes done now. I renamed a few classes to remove obvious references to ElasticSearch 5.X because the goal is to make it an obvious fit with 6.X too now that that is out.
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
@JPercivall I fixed the dependency issue and tried a simple flow. Everything seemed to work.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r157903183
--- Diff: nifi-nar-bundles/nifi-standard-services/nifi-elasticsearch-client-service-api/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchResponse.java ---
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import java.util.List;
+import java.util.Map;
+
+public class ElasticSearchResponse {
+ private List<Map<String, Object>> hits;
+ private Map<String, Object> aggregations;
+ private long numberOfHits;
+ private int took;
+ private boolean timedOut;
+
+ public ElasticSearchResponse(List<Map<String, Object>> hits, Map<String, Object> aggregations,
+ int numberOfHits, int took, boolean timedOut) {
+ this.hits = hits;
+ this.aggregations = aggregations;
+ this.numberOfHits = numberOfHits;
+ this.took = took;
+ this.timedOut = timedOut;
+ }
+
+ public Map<String, Object> getAggregations() {
+ return aggregations;
+ }
+
+ public List<Map<String, Object>> getHits() {
+ return hits;
+ }
+
+ public long getNumberOfHits() {
+ return numberOfHits;
+ }
+
+ public boolean isTimedOut() {
+ return timedOut;
+ }
+
+ public int getTook() {
+ return took;
+ }
+
+ @Override
+ public String toString() {
+ return "ElasticSearchResponse{" +
--- End diff --
Nice!
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r152056531
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+ @WritesAttribute(attribute = "mime.type"),
+ @WritesAttribute(attribute = "aggregation.name", description = "The name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder().name("original")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship. " +
+ "This applies even if you select the \"split up hits\" option to send individual hits to the " +
+ "\"hits\" relationship.").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el5-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el5-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el5-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax. Ex: " +
+ "{\n" +
--- End diff --
Nope.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r143464047
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,451 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.impl.nio.client.HttpAsyncClientBuilder;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.codehaus.jackson.map.ObjectMapper;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
--- End diff --
Yeah, that's worth considering.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175271865
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java ---
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.PrintWriter;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends AbstractControllerService implements ElasticSearchClientService {
+ private ObjectMapper mapper = new ObjectMapper();
+
+ private List<PropertyDescriptor> properties;
+
+ private RestClient client;
+
+ private String url;
+
+ @Override
+ protected void init(ControllerServiceInitializationContext config) {
+ properties = new ArrayList<>();
+ properties.add(ElasticSearchClientService.HTTP_HOSTS);
+ properties.add(ElasticSearchClientService.USERNAME);
+ properties.add(ElasticSearchClientService.PASSWORD);
+ properties.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+ properties.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+ properties.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+ properties.add(ElasticSearchClientService.RETRY_TIMEOUT);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return properties;
+ }
+
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws InitializationException {
+ try {
+ setupClient(context);
+ } catch (Exception ex) {
+ getLogger().error("Could not initialize ElasticSearch client.", ex);
+ throw new InitializationException(ex);
+ }
+ }
+
+ @OnDisabled
+ public void onDisabled() throws IOException {
+ this.client.close();
+ this.url = null;
+ }
+
+ private void setupClient(ConfigurationContext context) throws Exception {
+ final String hosts = context.getProperty(HTTP_HOSTS).evaluateAttributeExpressions().getValue();
+ String[] hostsSplit = hosts.split(",[\\s]*");
+ this.url = hostsSplit[0];
+ final SSLContextService sslService =
+ context.getProperty(PROP_SSL_CONTEXT_SERVICE).asControllerService(SSLContextService.class);
+ final String username = context.getProperty(USERNAME).evaluateAttributeExpressions().getValue();
+ final String password = context.getProperty(PASSWORD).evaluateAttributeExpressions().getValue();
+
+ final Integer connectTimeout = context.getProperty(CONNECT_TIMEOUT).asInteger();
+ final Integer readTimeout = context.getProperty(SOCKET_TIMEOUT).asInteger();
+ final Integer retryTimeout = context.getProperty(RETRY_TIMEOUT).asInteger();
+
+ HttpHost[] hh = new HttpHost[hostsSplit.length];
+ for (int x = 0; x < hh.length; x++) {
+ URL u = new URL(hostsSplit[x]);
+ hh[x] = new HttpHost(u.getHost(), u.getPort(), u.getProtocol());
+ }
+
+ RestClientBuilder builder = RestClient.builder(hh)
+ .setHttpClientConfigCallback(httpClientBuilder -> {
+ if (sslService != null) {
+ try {
+ KeyStore keyStore = KeyStore.getInstance(sslService.getKeyStoreType());
+ KeyStore trustStore = KeyStore.getInstance("JKS");
+
+ try (final InputStream is = new FileInputStream(sslService.getKeyStoreFile())) {
+ keyStore.load(is, sslService.getKeyStorePassword().toCharArray());
+ }
+
+ try (final InputStream is = new FileInputStream(sslService.getTrustStoreFile())) {
+ trustStore.load(is, sslService.getTrustStorePassword().toCharArray());
+ }
+
+ final KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory
+ .getDefaultAlgorithm());
+ kmf.init(keyStore, sslService.getKeyStorePassword().toCharArray());
+ final TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory
+ .getDefaultAlgorithm());
+ tmf.init(keyStore);
+ SSLContext context1 = SSLContext.getInstance(sslService.getSslAlgorithm());
+ context1.init(kmf.getKeyManagers(), tmf.getTrustManagers(), new SecureRandom());
+ httpClientBuilder = httpClientBuilder.setSSLContext(context1);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
--- End diff --
Removed. I think I have a better solution for that now.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175271547
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java ---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+ @WritesAttribute(attribute = "mime.type", description = "application/json"),
+ @WritesAttribute(attribute = "aggregation.name", description = "The name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder().name("original")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship. " +
+ "This applies even if you select the \"split up hits\" option to send individual hits to the " +
+ "\"hits\" relationship.").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el-rest-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el-rest-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el-rest-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax. Ex: " +
+ "{\n" +
+ "\t\"query\": {\n" +
+ "\t\t\"match\": {\n" +
+ "\t\t\t\"name\": \"John Smith\"\n" +
+ "\t\t}\n" +
+ "\t}\n" +
+ "}")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUERY_ATTRIBUTE = new PropertyDescriptor.Builder()
+ .name("el-query-attribute")
+ .displayName("Query Attribute")
+ .description("If set, the executed query will be set on each result flowfile in the specified attribute.")
+ .expressionLanguageSupported(true)
+ .addValidator(Validator.VALID)
+ .required(false)
+ .build();
+
+ public static final AllowableValue SPLIT_UP_YES = new AllowableValue(
+ "splitUp-yes",
+ "Yes",
+ "Split up results."
+ );
+ public static final AllowableValue SPLIT_UP_HITS_NO = new AllowableValue(
+ "splitUp-no",
+ "No",
+ "Don't split up results."
+ );
+
+ public static final PropertyDescriptor SPLIT_UP_HITS = new PropertyDescriptor.Builder()
+ .name("el-rest-split-up-hits")
+ .displayName("Split up search results")
+ .description("Split up search results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+ public static final PropertyDescriptor SPLIT_UP_AGGREGATIONS = new PropertyDescriptor.Builder()
+ .name("el-rest-split-up-aggregations")
+ .displayName("Split up aggregation results")
+ .description("Split up aggregation results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+
+ public static final PropertyDescriptor CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name("el-rest-client-service")
+ .displayName("Client Service")
+ .description("An ElasticSearch client service to use for running queries.")
+ .identifiesControllerService(ElasticSearchClientService.class)
+ .required(true)
+ .build();
+
+ private static final Set<Relationship> relationships;
+ private static final List<PropertyDescriptor> propertyDescriptors;
+
+ private ElasticSearchClientService clientService;
+
+ static {
+ final Set<Relationship> _rels = new HashSet<>();
+ _rels.add(REL_ORIGINAL);
+ _rels.add(REL_FAILURE);
+ _rels.add(REL_HITS);
+ _rels.add(REL_AGGREGATIONS);
+ relationships = Collections.unmodifiableSet(_rels);
+
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(QUERY);
+ descriptors.add(QUERY_ATTRIBUTE);
+ descriptors.add(INDEX);
+ descriptors.add(TYPE);
+ descriptors.add(CLIENT_SERVICE);
+ descriptors.add(SPLIT_UP_HITS);
+ descriptors.add(SPLIT_UP_AGGREGATIONS);
+
+ propertyDescriptors = Collections.unmodifiableList(descriptors);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return propertyDescriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ clientService = context.getProperty(CLIENT_SERVICE).asControllerService(ElasticSearchClientService.class);
+ }
+
+ @OnStopped
+ public void onStopped() {
+ this.clientService = null;
+ }
+
+
+ private final ObjectMapper mapper = new ObjectMapper();
+
+ private String getQuery(FlowFile input, ProcessContext context, ProcessSession session) throws IOException {
+ String retVal = null;
+ if (context.getProperty(QUERY).isSet()) {
+ retVal = context.getProperty(QUERY).evaluateAttributeExpressions(input).getValue();
+ } else if (input != null) {
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+ session.exportTo(input, out);
+ out.close();
+
+ retVal = new String(out.toByteArray());
+ }
+
+ return retVal;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile input = null;
+ if (context.hasIncomingConnection()) {
+ input = session.get();
+
+ if (input == null && context.hasNonLoopConnection()) {
+ return;
+ }
+ }
+
+ try {
+
+ final String query = getQuery(input, context, session);
+ final String index = context.getProperty(INDEX).evaluateAttributeExpressions(input).getValue();
+ final String type = context.getProperty(TYPE).evaluateAttributeExpressions(input).getValue();
+
+ Optional<SearchResponse> resp = clientService.search(query, index, type);
+ if (resp.isPresent()) {
+ SearchResponse response = resp.get();
+
+ List<FlowFile> hitsFlowFiles = handleHits(response.getHits(), context, session, input);
+ List<FlowFile> aggsFlowFiles = handleAggregations(response.getAggregations(), context, session, input);
+
+ if (hitsFlowFiles.size() > 0) {
+ for (FlowFile ff : hitsFlowFiles) {
+ session.transfer(ff, REL_HITS);
--- End diff --
Done.
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on the issue:
https://github.com/apache/nifi/pull/2113
Reviewing...
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
@JPercivall @mattyb149 That exception only happens for me when I stop the processor. It happens here:
``
if (hitsFlowFiles.size() > 0) {
session.transfer(hitsFlowFiles, REL_HITS);
for (FlowFile ff : hitsFlowFiles) {
session.getProvenanceReporter().send(ff, clientService.getTransitUrl(index, type));
}
}
``
On the provenance reporting line. Any ideas?
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r143334704
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,459 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.impl.nio.client.HttpAsyncClientBuilder;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.codehaus.jackson.map.ObjectMapper;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "elasticsearch 5", "fetch", "read", "get"})
+@CapabilityDescription("Retrieves a document from Elasticsearch using the specified connection properties and the "
+ + "identifier of the document to retrieve. If the cluster has been configured for authorization and/or secure "
+ + "transport (SSL/TLS), and the X-Pack plugin is available, secure connections can be made. This processor "
+ + "supports Elasticsearch 5.x clusters.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+
+ private RestClient client;
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el5-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el5-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .defaultValue("")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(Validator.VALID)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el5-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax.")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor HTTP_HOSTS = new PropertyDescriptor.Builder()
+ .name("el5-http-hosts")
+ .displayName("HTTP Hosts")
+ .description("A comma-separated list of HTTP hosts that host ElasticSearch query nodes.")
+ .required(true)
+ .expressionLanguageSupported(false)
--- End diff --
I think other processors have allowed EL in the password, it's not very secure without encrypted properties/registry entries, but that might be acceptable for some use cases
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175305604
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java ---
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends AbstractControllerService implements ElasticSearchClientService {
+ private ObjectMapper mapper = new ObjectMapper();
+
+ static final private List<PropertyDescriptor> properties;
+
+ private RestClient client;
+
+ private String url;
+
+ static {
+ List _props = new ArrayList();
+ _props.add(ElasticSearchClientService.HTTP_HOSTS);
+ _props.add(ElasticSearchClientService.USERNAME);
+ _props.add(ElasticSearchClientService.PASSWORD);
+ _props.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+ _props.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+ _props.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+ _props.add(ElasticSearchClientService.RETRY_TIMEOUT);
+
+ properties = Collections.unmodifiableList(_props);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return properties;
+ }
+
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws InitializationException {
+ try {
+ setupClient(context);
+ } catch (Exception ex) {
+ getLogger().error("Could not initialize ElasticSearch client.", ex);
+ throw new InitializationException(ex);
+ }
+ }
+
+ @OnDisabled
+ public void onDisabled() throws IOException {
+ this.client.close();
+ this.url = null;
+ }
+
+ private void setupClient(ConfigurationContext context) throws MalformedURLException {
+ final String hosts = context.getProperty(HTTP_HOSTS).evaluateAttributeExpressions().getValue();
+ String[] hostsSplit = hosts.split(",[\\s]*");
+ this.url = hostsSplit[0];
+ final SSLContextService sslService =
+ context.getProperty(PROP_SSL_CONTEXT_SERVICE).asControllerService(SSLContextService.class);
+ final String username = context.getProperty(USERNAME).evaluateAttributeExpressions().getValue();
+ final String password = context.getProperty(PASSWORD).evaluateAttributeExpressions().getValue();
+
+ final Integer connectTimeout = context.getProperty(CONNECT_TIMEOUT).asInteger();
+ final Integer readTimeout = context.getProperty(SOCKET_TIMEOUT).asInteger();
+ final Integer retryTimeout = context.getProperty(RETRY_TIMEOUT).asInteger();
+
+ HttpHost[] hh = new HttpHost[hostsSplit.length];
+ for (int x = 0; x < hh.length; x++) {
+ URL u = new URL(hostsSplit[x]);
+ hh[x] = new HttpHost(u.getHost(), u.getPort(), u.getProtocol());
+ }
+
+ RestClientBuilder builder = RestClient.builder(hh)
+ .setHttpClientConfigCallback(httpClientBuilder -> {
+ if (sslService != null && sslService.isKeyStoreConfigured() && sslService.isTrustStoreConfigured()) {
+ try {
+ KeyStore keyStore = KeyStore.getInstance(sslService.getKeyStoreType());
+ KeyStore trustStore = KeyStore.getInstance("JKS");
+
+ try (final InputStream is = new FileInputStream(sslService.getKeyStoreFile())) {
+ keyStore.load(is, sslService.getKeyStorePassword().toCharArray());
+ }
+
+ try (final InputStream is = new FileInputStream(sslService.getTrustStoreFile())) {
+ trustStore.load(is, sslService.getTrustStorePassword().toCharArray());
+ }
+
+ final KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory
+ .getDefaultAlgorithm());
+ kmf.init(keyStore, sslService.getKeyStorePassword().toCharArray());
+ final TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory
+ .getDefaultAlgorithm());
+ tmf.init(keyStore);
+ SSLContext context1 = SSLContext.getInstance(sslService.getSslAlgorithm());
+ context1.init(kmf.getKeyManagers(), tmf.getTrustManagers(), new SecureRandom());
+ httpClientBuilder = httpClientBuilder.setSSLContext(context1);
+ } catch (Exception e) {
+ getLogger().error("Error setting up SSL.", e);
+ }
+ }
+
+ if (username != null && password != null) {
+ final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
+ credentialsProvider.setCredentials(AuthScope.ANY,
+ new UsernamePasswordCredentials(username, password));
+ httpClientBuilder = httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
+ }
+
+ return httpClientBuilder;
+ })
+ .setRequestConfigCallback(requestConfigBuilder -> {
+ requestConfigBuilder.setConnectTimeout(connectTimeout);
+ requestConfigBuilder.setSocketTimeout(readTimeout);
+ return requestConfigBuilder;
+ })
+ .setMaxRetryTimeoutMillis(retryTimeout);
+
+ this.client = builder.build();
+ }
+
+ private Response runQuery(String query, String index, String type) throws IOException {
+ StringBuilder sb = new StringBuilder()
+ .append("/")
+ .append(index);
+ if (type != null && !type.equals("")) {
+ sb.append("/")
+ .append(type);
+ }
+
+ sb.append("/_search");
+
+ HttpEntity queryEntity = new NStringEntity(query, ContentType.APPLICATION_JSON);
+
+ return client.performRequest("POST", sb.toString(), Collections.emptyMap(), queryEntity);
+ }
+
+ @Override
+ public Optional<SearchResponse> search(String query, String index, String type) throws IOException {
+ Response response = runQuery(query, index, type);
+ Map<String, Object> parsed = mapper.readValue(IOUtils.toString(response.getEntity().getContent()), Map.class);
+
+ int took = (Integer)parsed.get("took");
+ boolean timedOut = (Boolean)parsed.get("timed_out");
+ Map<String, Object> aggregations = parsed.get("aggregations") != null
+ ? (Map<String, Object>)parsed.get("aggregations") : new HashMap<>();
+ Map<String, Object> hitsParent = (Map<String, Object>)parsed.get("hits");
+ int count = (Integer)hitsParent.get("total");
+ List<Map<String, Object>> hits = (List<Map<String, Object>>)hitsParent.get("hits");
+
+ SearchResponse esr = new SearchResponse(hits, aggregations, count, took, timedOut);
+
+ if (getLogger().isDebugEnabled()) {
+ StringBuilder sb = new StringBuilder();
+ sb.append("******************");
+ sb.append(String.format("Took: %d", took));
+ sb.append(String.format("Timed out: %s", timedOut));
+ sb.append(String.format("Aggregation count: %d", aggregations.size()));
+ sb.append(String.format("Hit count: %d", hits.size()));
+ sb.append(String.format("Total found: %d", count));
+ sb.append("******************");
+
+ getLogger().debug(sb.toString());
+ }
+
+ return Optional.of(esr);
--- End diff --
Does this still need to be an Optional?
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175264961
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java ---
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.PrintWriter;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends AbstractControllerService implements ElasticSearchClientService {
+ private ObjectMapper mapper = new ObjectMapper();
+
+ private List<PropertyDescriptor> properties;
--- End diff --
Can this be declared similar to the properties in the processor and marked final?
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175306340
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java ---
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends AbstractControllerService implements ElasticSearchClientService {
+ private ObjectMapper mapper = new ObjectMapper();
+
+ static final private List<PropertyDescriptor> properties;
+
+ private RestClient client;
+
+ private String url;
+
+ static {
+ List _props = new ArrayList();
+ _props.add(ElasticSearchClientService.HTTP_HOSTS);
+ _props.add(ElasticSearchClientService.USERNAME);
+ _props.add(ElasticSearchClientService.PASSWORD);
+ _props.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+ _props.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+ _props.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+ _props.add(ElasticSearchClientService.RETRY_TIMEOUT);
+
+ properties = Collections.unmodifiableList(_props);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return properties;
+ }
+
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws InitializationException {
+ try {
+ setupClient(context);
+ } catch (Exception ex) {
+ getLogger().error("Could not initialize ElasticSearch client.", ex);
+ throw new InitializationException(ex);
+ }
+ }
+
+ @OnDisabled
+ public void onDisabled() throws IOException {
+ this.client.close();
+ this.url = null;
+ }
+
+ private void setupClient(ConfigurationContext context) throws MalformedURLException {
+ final String hosts = context.getProperty(HTTP_HOSTS).evaluateAttributeExpressions().getValue();
+ String[] hostsSplit = hosts.split(",[\\s]*");
+ this.url = hostsSplit[0];
+ final SSLContextService sslService =
+ context.getProperty(PROP_SSL_CONTEXT_SERVICE).asControllerService(SSLContextService.class);
+ final String username = context.getProperty(USERNAME).evaluateAttributeExpressions().getValue();
+ final String password = context.getProperty(PASSWORD).evaluateAttributeExpressions().getValue();
+
+ final Integer connectTimeout = context.getProperty(CONNECT_TIMEOUT).asInteger();
+ final Integer readTimeout = context.getProperty(SOCKET_TIMEOUT).asInteger();
+ final Integer retryTimeout = context.getProperty(RETRY_TIMEOUT).asInteger();
+
+ HttpHost[] hh = new HttpHost[hostsSplit.length];
+ for (int x = 0; x < hh.length; x++) {
+ URL u = new URL(hostsSplit[x]);
+ hh[x] = new HttpHost(u.getHost(), u.getPort(), u.getProtocol());
+ }
+
+ RestClientBuilder builder = RestClient.builder(hh)
+ .setHttpClientConfigCallback(httpClientBuilder -> {
+ if (sslService != null && sslService.isKeyStoreConfigured() && sslService.isTrustStoreConfigured()) {
+ try {
+ KeyStore keyStore = KeyStore.getInstance(sslService.getKeyStoreType());
+ KeyStore trustStore = KeyStore.getInstance("JKS");
+
+ try (final InputStream is = new FileInputStream(sslService.getKeyStoreFile())) {
+ keyStore.load(is, sslService.getKeyStorePassword().toCharArray());
+ }
+
+ try (final InputStream is = new FileInputStream(sslService.getTrustStoreFile())) {
+ trustStore.load(is, sslService.getTrustStorePassword().toCharArray());
+ }
+
+ final KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory
+ .getDefaultAlgorithm());
+ kmf.init(keyStore, sslService.getKeyStorePassword().toCharArray());
+ final TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory
+ .getDefaultAlgorithm());
+ tmf.init(keyStore);
+ SSLContext context1 = SSLContext.getInstance(sslService.getSslAlgorithm());
+ context1.init(kmf.getKeyManagers(), tmf.getTrustManagers(), new SecureRandom());
+ httpClientBuilder = httpClientBuilder.setSSLContext(context1);
+ } catch (Exception e) {
+ getLogger().error("Error setting up SSL.", e);
+ }
+ }
+
+ if (username != null && password != null) {
+ final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
+ credentialsProvider.setCredentials(AuthScope.ANY,
+ new UsernamePasswordCredentials(username, password));
+ httpClientBuilder = httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
+ }
+
+ return httpClientBuilder;
+ })
+ .setRequestConfigCallback(requestConfigBuilder -> {
+ requestConfigBuilder.setConnectTimeout(connectTimeout);
+ requestConfigBuilder.setSocketTimeout(readTimeout);
+ return requestConfigBuilder;
+ })
+ .setMaxRetryTimeoutMillis(retryTimeout);
+
+ this.client = builder.build();
+ }
+
+ private Response runQuery(String query, String index, String type) throws IOException {
+ StringBuilder sb = new StringBuilder()
+ .append("/")
+ .append(index);
+ if (type != null && !type.equals("")) {
+ sb.append("/")
+ .append(type);
+ }
+
+ sb.append("/_search");
+
+ HttpEntity queryEntity = new NStringEntity(query, ContentType.APPLICATION_JSON);
+
+ return client.performRequest("POST", sb.toString(), Collections.emptyMap(), queryEntity);
+ }
+
+ @Override
+ public Optional<SearchResponse> search(String query, String index, String type) throws IOException {
+ Response response = runQuery(query, index, type);
+ Map<String, Object> parsed = mapper.readValue(IOUtils.toString(response.getEntity().getContent()), Map.class);
--- End diff --
Also, prior to interpreting this as a map, the status line of the response should be checked to see if it actually succeeded. That way we can log the ES error message in the processor as a bulletin for easy diagnosing of problems by the user. You can see handling of the status message in the FetchElasticsearcHttp processor: https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearchHttp.java#L262
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175271497
--- Diff: nifi-nar-bundles/nifi-standard-services/pom.xml ---
@@ -15,6 +15,18 @@
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
+ <dependencies>
--- End diff --
Done.
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on the issue:
https://github.com/apache/nifi/pull/2113
Hey @mattyb149, what's your status for this review? From a cursory look, it appears like it just needs an updated pom from @MikeThomsen and the final approval. With talks of 1.6.0 happening it would be nice to get this in so those using ES 6 aren't limited to the HTTP processor.
If help is needed to finalize things just let me know where I can help.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175264989
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java ---
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.PrintWriter;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends AbstractControllerService implements ElasticSearchClientService {
+ private ObjectMapper mapper = new ObjectMapper();
+
+ private List<PropertyDescriptor> properties;
+
+ private RestClient client;
+
+ private String url;
+
+ @Override
+ protected void init(ControllerServiceInitializationContext config) {
+ properties = new ArrayList<>();
+ properties.add(ElasticSearchClientService.HTTP_HOSTS);
+ properties.add(ElasticSearchClientService.USERNAME);
+ properties.add(ElasticSearchClientService.PASSWORD);
+ properties.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+ properties.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+ properties.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+ properties.add(ElasticSearchClientService.RETRY_TIMEOUT);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return properties;
+ }
+
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws InitializationException {
+ try {
+ setupClient(context);
+ } catch (Exception ex) {
+ getLogger().error("Could not initialize ElasticSearch client.", ex);
+ throw new InitializationException(ex);
+ }
+ }
+
+ @OnDisabled
+ public void onDisabled() throws IOException {
+ this.client.close();
+ this.url = null;
+ }
+
+ private void setupClient(ConfigurationContext context) throws Exception {
--- End diff --
Try to avoid "throws Exception". Looks like the only one thrown is "MalformedURLException".
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on the issue:
https://github.com/apache/nifi/pull/2113
The nifi-elasticsearch-client-service-nar is missing a license and notice.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r150726813
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+ @WritesAttribute(attribute = "mime.type"),
+ @WritesAttribute(attribute = "aggregation.name", description = "The name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder().name("original")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship. " +
+ "This applies even if you select the \"split up hits\" option to send individual hits to the " +
+ "\"hits\" relationship.").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el5-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el5-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el5-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax. Ex: " +
+ "{\n" +
--- End diff --
I haven't yet tried this out, do newlines and tabs show up in the tooltips ok?
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r150726318
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,451 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.impl.nio.client.HttpAsyncClientBuilder;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.codehaus.jackson.map.ObjectMapper;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
--- End diff --
Having the 5 and Http can be confusing as well, so I'm good with the name as-is :)
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
@JPercivall Done.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r149120851
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+ public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship. " +
+ "This applies even if you select the \"break up hits\" option to send individual hits to the " +
+ "\"hits\" relationship.").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el5-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el5-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .defaultValue("")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(Validator.VALID)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el5-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax.")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final AllowableValue BREAK_UP_YES = new AllowableValue(
+ "breakup-yes",
+ "Yes",
+ "Break up results."
+ );
+ public static final AllowableValue BREAK_UP_HITS_NO = new AllowableValue(
+ "breakup-no",
+ "No",
+ "Don't break up results."
+ );
+
+ public static final PropertyDescriptor BREAK_UP_HITS = new PropertyDescriptor.Builder()
+ .name("el5-break-up-hits")
+ .displayName("Break up search results")
--- End diff --
This is just a style comment (so not a requirement just a suggestion), other processors use the term "split" rather than "break up".
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r157902719
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+ @WritesAttribute(attribute = "mime.type", description = "application/json"),
+ @WritesAttribute(attribute = "aggregation.name", description = "The name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder().name("original")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship. " +
+ "This applies even if you select the \"split up hits\" option to send individual hits to the " +
+ "\"hits\" relationship.").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el5-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el5-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el5-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax. Ex: " +
+ "{\n" +
+ "\t\"query\": {\n" +
+ "\t\t\"match\": {\n" +
+ "\t\t\t\"name\": \"John Smith\"\n" +
+ "\t\t}\n" +
+ "\t}\n" +
+ "}")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final AllowableValue SPLIT_UP_YES = new AllowableValue(
+ "splitUp-yes",
+ "Yes",
+ "Split up results."
+ );
+ public static final AllowableValue SPLIT_UP_HITS_NO = new AllowableValue(
+ "splitUp-no",
+ "No",
+ "Don't split up results."
+ );
+
+ public static final PropertyDescriptor SPLIT_UP_HITS = new PropertyDescriptor.Builder()
+ .name("el5-split-up-hits")
+ .displayName("Split up search results")
+ .description("Split up search results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+ public static final PropertyDescriptor SPLIT_UP_AGGREGATIONS = new PropertyDescriptor.Builder()
+ .name("el5-split-up-aggregations")
+ .displayName("Split up aggregation results")
+ .description("Split up aggregation results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+
+ public static final PropertyDescriptor CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name("el5-client-service")
+ .displayName("Client Service")
+ .description("An ElasticSearch client service to use for running queries.")
+ .identifiesControllerService(ElasticSearchClientService.class)
+ .required(true)
+ .build();
+
+ private static final Set<Relationship> relationships;
+ private static final List<PropertyDescriptor> propertyDescriptors;
+
+ private ElasticSearchClientService clientService;
+
+ static {
+ final Set<Relationship> _rels = new HashSet<>();
+ _rels.add(REL_ORIGINAL);
+ _rels.add(REL_FAILURE);
+ _rels.add(REL_HITS);
+ _rels.add(REL_AGGREGATIONS);
+ relationships = Collections.unmodifiableSet(_rels);
+
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(QUERY);
+ descriptors.add(INDEX);
+ descriptors.add(TYPE);
+ descriptors.add(CLIENT_SERVICE);
+ descriptors.add(SPLIT_UP_HITS);
+ descriptors.add(SPLIT_UP_AGGREGATIONS);
+
+ propertyDescriptors = Collections.unmodifiableList(descriptors);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return propertyDescriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ clientService = context.getProperty(CLIENT_SERVICE).asControllerService(ElasticSearchClientService.class);
+ }
+
+ @OnUnscheduled
+ public void onUnscheduled() {
+ this.clientService = null;
+ }
+
+
+ private ObjectMapper mapper = new ObjectMapper();
--- End diff --
Nitpick, could be final?
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
Rebased and building. @mattyb149 I changed the functionality to match the GetMongo functionality you reviewed recently. Had to spend a while getting the build to work again.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/2113
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
@mattyb149 I've been thinking about what to do with this PR, and I think it needs to be put into a new NAR. I'm thinking "nifi-elasticsearch-restapi.nar" or something like that. What I'd ideally like to do is position this to be a new package that does full CRUD using only the official REST API from Elastic Co.
It might also help with the issue that @joewitt raised about the sizing because the current mix of ElasticSearch processors use different toolkits which needlessly raises the size of the NARs.
Thoughts?
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r149119932
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+ public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship. " +
--- End diff --
Should this be called "original" if the incoming flow files (that don't cause an error) are always routed here?
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r143295442
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,459 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.impl.nio.client.HttpAsyncClientBuilder;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.codehaus.jackson.map.ObjectMapper;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "elasticsearch 5", "fetch", "read", "get"})
+@CapabilityDescription("Retrieves a document from Elasticsearch using the specified connection properties and the "
+ + "identifier of the document to retrieve. If the cluster has been configured for authorization and/or secure "
+ + "transport (SSL/TLS), and the X-Pack plugin is available, secure connections can be made. This processor "
+ + "supports Elasticsearch 5.x clusters.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+
+ private RestClient client;
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el5-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el5-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .defaultValue("")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(Validator.VALID)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el5-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax.")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor HTTP_HOSTS = new PropertyDescriptor.Builder()
+ .name("el5-http-hosts")
+ .displayName("HTTP Hosts")
+ .description("A comma-separated list of HTTP hosts that host ElasticSearch query nodes.")
+ .required(true)
+ .expressionLanguageSupported(false)
--- End diff --
Whether or not Expression Language has been historically supported in other ES processors, this seems like a good place to support them :) Pretty much goes for all these properties, except maybe sensitive properties (for now) like password. I realize a parameterized username might be useless without a parameterized password, but no harm no foul!
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on the issue:
https://github.com/apache/nifi/pull/2113
Thanks @MikeThomsen, looks like that fixed the NPE. Still needs the License and Notice in the client service nar.
Also, it's a bit of an anti-pattern to have the logging and error handling in the Controller Service for methods called by the processor. This is because any time there is a problem with the search, the bulletin is logged at the Controller Service. This means that not only will a user not know which processor is the cause of the error (since it could be shared) but also when looking at the map the processor appears to be working without issue. Lastly, since the error isn't bubbled back to the processor, it never yields so if there is something wrong with the query it will continually error and log without pause.
Related, there's also some weirdness with error handling when there is an incoming connection. Currently, input is only sent to Failure instead of Original when an error is caught at the top level. This means that if a query is messed up and is caught as a JsonProcessingException in the Controller service, the input will be routed to original and no other outputs. As a user, I would expect that flowfile to be sent to Failure.
So in summary, the CS should let the calling processor decide how to handle any exceptions encountered during processing. This allows the processor to more granularlly decide what to do under different situations. For example, was it a JSON parsing exception and it's acting as a source processor? well we better yield the whole processor since it's probably just going to continually fail. Is there an incoming connection but some network error happened when searching? again, later FlowFiles will probably fail too, so we should yield. If you have any questions on when to yield the processor vs penalizing the FlowFile, feel free to ask.
Lastly, it's preferred if you don't squash your commits every time. If you don't, that allows the reviewer to more easily see exactly what changed since they last reviewed it. Also allows reviewers to see how the PR evolved over time in response to different comments.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r150726684
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+ @WritesAttribute(attribute = "mime.type"),
+ @WritesAttribute(attribute = "aggregation.name", description = "The name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder().name("original")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship. " +
+ "This applies even if you select the \"split up hits\" option to send individual hits to the " +
+ "\"hits\" relationship.").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el5-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el5-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el5-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax. Ex: " +
+ "{\n" +
+ "\t\"query\": {\n" +
+ "\t\t\"match\": {\n" +
+ "\t\t\t\"name\": \"John Smith\"\n" +
+ "\t\t}\n" +
+ "\t}\n" +
+ "}")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final AllowableValue SPLIT_UP_YES = new AllowableValue(
+ "splitUp-yes",
+ "Yes",
+ "Split up results."
+ );
+ public static final AllowableValue SPLIT_UP_HITS_NO = new AllowableValue(
+ "splitUp-no",
+ "No",
+ "Don't split up results."
+ );
+
+ public static final PropertyDescriptor SPLIT_UP_HITS = new PropertyDescriptor.Builder()
+ .name("el5-split-up-hits")
+ .displayName("Split up search results")
+ .description("Split up search results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+ public static final PropertyDescriptor SPLIT_UP_AGGREGATIONS = new PropertyDescriptor.Builder()
+ .name("el5-split-up-aggregations")
+ .displayName("Split up aggregation results")
+ .description("Split up aggregation results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+
+ public static final PropertyDescriptor CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name("el5-client-service")
+ .displayName("Client Service")
+ .description("An ElasticSearch client service to use for running queries.")
+ .identifiesControllerService(ElasticSearchClientService.class)
+ .required(true)
+ .build();
+
+ private static final Set<Relationship> relationships;
+ private static final List<PropertyDescriptor> propertyDescriptors;
+
+ private ElasticSearchClientService clientService;
+
+ static {
+ final Set<Relationship> _rels = new HashSet<>();
+ _rels.add(REL_ORIGINAL);
+ _rels.add(REL_FAILURE);
+ _rels.add(REL_HITS);
+ _rels.add(REL_AGGREGATIONS);
+ relationships = Collections.unmodifiableSet(_rels);
+
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(QUERY);
+ descriptors.add(INDEX);
+ descriptors.add(TYPE);
+ descriptors.add(CLIENT_SERVICE);
+ descriptors.add(SPLIT_UP_HITS);
+ descriptors.add(SPLIT_UP_AGGREGATIONS);
+
+ propertyDescriptors = Collections.unmodifiableList(descriptors);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return propertyDescriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ clientService = context.getProperty(CLIENT_SERVICE).asControllerService(ElasticSearchClientService.class);
+ }
+
+ @OnUnscheduled
+ public void onUnscheduled() {
+ this.clientService = null;
+ }
+
+
+ private ObjectMapper mapper = new ObjectMapper();
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ return;
+ }
+
+ try {
+
+ String query = context.getProperty(QUERY).evaluateAttributeExpressions(flowFile).getValue();
+ String index = context.getProperty(INDEX).evaluateAttributeExpressions(flowFile).getValue();
+ String type = context.getProperty(TYPE).evaluateAttributeExpressions(flowFile).getValue();
+
+ Optional<ElasticSearchResponse> resp = clientService.search(query, index, type);
+ ElasticSearchResponse response = resp.get();
+
+ List<FlowFile> hitsFlowFiles = handleHits(response.getHits(), context, session, flowFile);
+ List<FlowFile> aggsFlowFiles = handleAggregations(response.getAggregations(), context, session, flowFile);
+
+ if (hitsFlowFiles.size() > 0) {
+ session.transfer(hitsFlowFiles, REL_HITS);
+ for (FlowFile ff : hitsFlowFiles) {
+ session.getProvenanceReporter().create(ff);
--- End diff --
Since these are coming from an external system, I believe the convention is to use receive() instead of create(). Same goes below for aggregations.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r173996495
--- Diff: nifi-assembly/pom.xml ---
@@ -445,6 +445,24 @@ language governing permissions and limitations under the License. -->
<version>1.6.0-SNAPSHOT</version>
<type>nar</type>
</dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-elasticsearch-client-service-api-nar</artifactId>
--- End diff --
My build is failing due to this. I don't see this artifact created anywhere.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r143463999
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,451 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.impl.nio.client.HttpAsyncClientBuilder;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.codehaus.jackson.map.ObjectMapper;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+
+ private RestClient client;
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el5-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el5-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .defaultValue("")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(Validator.VALID)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el5-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax.")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor HTTP_HOSTS = new PropertyDescriptor.Builder()
+ .name("el5-http-hosts")
+ .displayName("HTTP Hosts")
+ .description("A comma-separated list of HTTP hosts that host ElasticSearch query nodes.")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PROP_SSL_CONTEXT_SERVICE = new PropertyDescriptor.Builder()
+ .name("el5-ssl-context-service")
+ .displayName("SSL Context Service")
+ .description("The SSL Context Service used to provide client certificate information for TLS/SSL "
+ + "connections. This service only applies if the Elasticsearch endpoint(s) have been secured with TLS/SSL.")
+ .required(false)
+ .identifiesControllerService(SSLContextService.class)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .name("el5-username")
+ .displayName("Username")
+ .description("The username to use with XPack security.")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .name("el5-password")
+ .displayName("Password")
+ .description("The password to use with XPack security.")
+ .required(false)
+ .sensitive(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor CONNECT_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("el5-connect-timeout")
+ .displayName("Connect timeout")
+ .description("Controls the amount of time, in milliseconds, before a timeout occurs when trying to connect.")
+ .required(true)
+ .defaultValue("5000")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor SOCKET_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("el5-socket-timeout")
+ .displayName("Read timeout")
+ .description("Controls the amount of time, in milliseconds, before a timeout occurs when waiting for a response.")
+ .required(true)
+ .defaultValue("60000")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor RETRY_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("el5-retry-timeout")
+ .displayName("Retry timeout")
+ .description("Controls the amount of time, in milliseconds, before a timeout occurs when retrying the operation.")
+ .required(true)
+ .defaultValue("60000")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final AllowableValue BREAK_UP_YES = new AllowableValue(
+ "breakup-yes",
+ "Yes",
+ "Break up results."
+ );
+ public static final AllowableValue BREAK_UP_HITS_NO = new AllowableValue(
+ "breakup-no",
+ "No",
+ "Don't break up results."
+ );
+
+ public static final PropertyDescriptor BREAK_UP_HITS = new PropertyDescriptor.Builder()
+ .name("el5-break-up-hits")
+ .displayName("Break up search results")
+ .description("Break up search results into one flowfile per result.")
+ .allowableValues(BREAK_UP_HITS_NO, BREAK_UP_YES)
+ .defaultValue(BREAK_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+ public static final PropertyDescriptor BREAK_UP_AGGREGATIONS = new PropertyDescriptor.Builder()
+ .name("el5-break-up-aggregations")
+ .displayName("Break up aggregation results")
+ .description("Break up aggregation results into one flowfile per result.")
+ .allowableValues(BREAK_UP_HITS_NO, BREAK_UP_YES)
+ .defaultValue(BREAK_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+
+ private static final Set<Relationship> relationships;
+ private static final List<PropertyDescriptor> propertyDescriptors;
+
+ static {
+ final Set<Relationship> _rels = new HashSet<>();
+ _rels.add(REL_SUCCESS);
+ _rels.add(REL_FAILURE);
+ _rels.add(REL_HITS);
+ _rels.add(REL_AGGREGATIONS);
+ relationships = Collections.unmodifiableSet(_rels);
+
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(PROP_SSL_CONTEXT_SERVICE);
+ descriptors.add(CONNECT_TIMEOUT);
+ descriptors.add(SOCKET_TIMEOUT);
+ descriptors.add(RETRY_TIMEOUT);
+ descriptors.add(USERNAME);
+ descriptors.add(PASSWORD);
+ descriptors.add(HTTP_HOSTS);
+ descriptors.add(QUERY);
+ descriptors.add(INDEX);
+ descriptors.add(TYPE);
+ descriptors.add(BREAK_UP_HITS);
+ descriptors.add(BREAK_UP_AGGREGATIONS);
+
+ propertyDescriptors = Collections.unmodifiableList(descriptors);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return propertyDescriptors;
+ }
+
+
+ @OnUnscheduled
+ public void onUnscheduled() {
+ this.client = null;
+ }
+
+ private void setupClient(ProcessContext context) throws Exception {
+ final String hosts = context.getProperty(HTTP_HOSTS).evaluateAttributeExpressions().getValue();
+ String[] hostsSplit = hosts.split(",[\\s]*");
+ final SSLContextService sslService =
+ context.getProperty(PROP_SSL_CONTEXT_SERVICE).asControllerService(SSLContextService.class);
+ final String username = context.getProperty(USERNAME).evaluateAttributeExpressions().getValue();
+ final String password = context.getProperty(PASSWORD).evaluateAttributeExpressions().getValue();
+
+ final Integer connectTimeout = context.getProperty(CONNECT_TIMEOUT).asInteger();
+ final Integer readTimeout = context.getProperty(SOCKET_TIMEOUT).asInteger();
+ final Integer retryTimeout = context.getProperty(RETRY_TIMEOUT).asInteger();
+
+ HttpHost[] hh = new HttpHost[hostsSplit.length];
+ for (int x = 0; x < hh.length; x++) {
+ URL u = new URL(hostsSplit[x]);
+ hh[x] = new HttpHost(u.getHost(), u.getPort(), u.getProtocol());
+ }
+
+ RestClientBuilder builder = RestClient.builder(hh)
+ .setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
+ @Override
+ public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
+ if (sslService != null) {
+ try {
+ KeyStore keyStore = KeyStore.getInstance(sslService.getKeyStoreType());
+ KeyStore trustStore = KeyStore.getInstance("JKS");
+
+ try (final InputStream is = new FileInputStream(sslService.getKeyStoreFile())) {
+ keyStore.load(is, sslService.getKeyStorePassword().toCharArray());
+ }
+
+ try (final InputStream is = new FileInputStream(sslService.getTrustStoreFile())) {
+ trustStore.load(is, sslService.getTrustStorePassword().toCharArray());
+ }
+
+ final KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory
+ .getDefaultAlgorithm());
+ kmf.init(keyStore, sslService.getKeyStorePassword().toCharArray());
+ final TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory
+ .getDefaultAlgorithm());
+ tmf.init(keyStore);
+ SSLContext context = SSLContext.getInstance(sslService.getSslAlgorithm());
+ context.init(kmf.getKeyManagers(), tmf.getTrustManagers(), new SecureRandom());
+ httpClientBuilder = httpClientBuilder.setSSLContext(context);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ if (username != null && password != null) {
+ final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
+ credentialsProvider.setCredentials(AuthScope.ANY,
+ new UsernamePasswordCredentials(username, password));
+ httpClientBuilder = httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
+ }
+
+ return httpClientBuilder;
+ }
+ })
+ .setRequestConfigCallback(requestConfigBuilder -> requestConfigBuilder.setConnectTimeout(connectTimeout)
+ .setSocketTimeout(readTimeout))
+ .setMaxRetryTimeoutMillis(retryTimeout);
+
+ this.client = builder.build();
+ }
+
+ private Response runQuery(String query, String index, String type) throws Exception {
+ StringBuilder sb = new StringBuilder()
+ .append("/")
+ .append(index);
+ if (type != null && !type.equals("")) {
+ sb.append("/")
+ .append(type);
+ }
+
+ sb.append("/_search");
+
+ HttpEntity queryEntity = new NStringEntity(query, ContentType.APPLICATION_JSON);
+
+ return client.performRequest("POST", sb.toString(), Collections.emptyMap(), queryEntity);
+ }
+
+ private ObjectMapper mapper = new ObjectMapper();
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ return;
+ }
+
+ try {
+ if (this.client == null) {
+ setupClient(context);
+ }
+
+ String query = context.getProperty(QUERY).evaluateAttributeExpressions(flowFile).getValue();
+ String index = context.getProperty(INDEX).evaluateAttributeExpressions(flowFile).getValue();
+ String type = context.getProperty(TYPE).evaluateAttributeExpressions(flowFile).getValue();
+
+ Response resp = runQuery(query, index, type);
+
+ InputStream content = resp.getEntity().getContent();
+ String rawJson = IOUtils.toString(content, "UTF-8");
--- End diff --
Apparently Lucene requires it: https://stackoverflow.com/a/23044215
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on the issue:
https://github.com/apache/nifi/pull/2113
Taking a look now. WRT:
> Ok, I can do that from now on. Shouldn't they be squashed before a merge into master?
Yup, they will be squashed prior to merge to master but the reviewer can do that at the same time they add the comment to the commit to close the PR.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r143294684
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,459 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.impl.nio.client.HttpAsyncClientBuilder;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.codehaus.jackson.map.ObjectMapper;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "elasticsearch 5", "fetch", "read", "get"})
+@CapabilityDescription("Retrieves a document from Elasticsearch using the specified connection properties and the "
+ + "identifier of the document to retrieve. If the cluster has been configured for authorization and/or secure "
--- End diff --
This processor does more than retrieve based on the identifier, it's based on the query right? Maybe elaborate on that as well as the various results and their relationships?
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r149123022
--- Diff: nifi-nar-bundles/nifi-standard-services/nifi-elasticsearch-client-service-bundle/nifi-elasticsearch-client-service/pom.xml ---
@@ -0,0 +1,158 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-elasticsearch-client-service-bundle</artifactId>
+ <version>1.5.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-elasticsearch-client-service</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-lookup-service-api</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-utils</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-hadoop-utils</artifactId>
--- End diff --
Does this package need nifi-hadoop-utils? Please remove if not.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r150086797
--- Diff: nifi-assembly/pom.xml ---
@@ -338,6 +338,16 @@
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-elasticsearch-client-service-nar</artifactId>
+ <type>nar</type>
+ </dependency>
+<!-- <dependency>
--- End diff --
That's how I included the nar file for the client service that the new processor uses. Is there a better approach?
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on the issue:
https://github.com/apache/nifi/pull/2113
I'm not sure I'll have time to close the loop before 1.6.0, so if you'd like to finish the review/merge after Mike's rebase that would be very cool, thanks!
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r149120565
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+ public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship. " +
+ "This applies even if you select the \"break up hits\" option to send individual hits to the " +
+ "\"hits\" relationship.").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el5-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el5-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .defaultValue("")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(Validator.VALID)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el5-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax.")
--- End diff --
An example would be really helpful here, although I realize it might look messy to have a JSON doc in the description. Any super-small query that would illustrate the kind of syntax to be used? If this is not the right place, perhaps in the CapabilityDescription or you could add an Additional Details page with some examples of queries, aggregations, etc.?
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175305389
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java ---
@@ -0,0 +1,329 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+ @WritesAttribute(attribute = "mime.type", description = "application/json"),
+ @WritesAttribute(attribute = "aggregation.name", description = "The name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder().name("original")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship. " +
+ "This applies even if you select the \"split up hits\" option to send individual hits to the " +
+ "\"hits\" relationship.").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el-rest-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el-rest-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el-rest-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax. Ex: " +
+ "{\n" +
+ "\t\"query\": {\n" +
+ "\t\t\"match\": {\n" +
+ "\t\t\t\"name\": \"John Smith\"\n" +
+ "\t\t}\n" +
+ "\t}\n" +
+ "}")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUERY_ATTRIBUTE = new PropertyDescriptor.Builder()
+ .name("el-query-attribute")
+ .displayName("Query Attribute")
+ .description("If set, the executed query will be set on each result flowfile in the specified attribute.")
+ .expressionLanguageSupported(true)
+ .addValidator(Validator.VALID)
+ .required(false)
+ .build();
+
+ public static final AllowableValue SPLIT_UP_YES = new AllowableValue(
+ "splitUp-yes",
+ "Yes",
+ "Split up results."
+ );
+ public static final AllowableValue SPLIT_UP_HITS_NO = new AllowableValue(
+ "splitUp-no",
+ "No",
+ "Don't split up results."
+ );
+
+ public static final PropertyDescriptor SPLIT_UP_HITS = new PropertyDescriptor.Builder()
+ .name("el-rest-split-up-hits")
+ .displayName("Split up search results")
+ .description("Split up search results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+ public static final PropertyDescriptor SPLIT_UP_AGGREGATIONS = new PropertyDescriptor.Builder()
+ .name("el-rest-split-up-aggregations")
+ .displayName("Split up aggregation results")
+ .description("Split up aggregation results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+
+ public static final PropertyDescriptor CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name("el-rest-client-service")
+ .displayName("Client Service")
+ .description("An ElasticSearch client service to use for running queries.")
+ .identifiesControllerService(ElasticSearchClientService.class)
+ .required(true)
+ .build();
+
+ private static final Set<Relationship> relationships;
+ private static final List<PropertyDescriptor> propertyDescriptors;
+
+ private volatile ElasticSearchClientService clientService;
+
+ static {
+ final Set<Relationship> _rels = new HashSet<>();
+ _rels.add(REL_ORIGINAL);
+ _rels.add(REL_FAILURE);
+ _rels.add(REL_HITS);
+ _rels.add(REL_AGGREGATIONS);
+ relationships = Collections.unmodifiableSet(_rels);
+
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(QUERY);
+ descriptors.add(QUERY_ATTRIBUTE);
+ descriptors.add(INDEX);
+ descriptors.add(TYPE);
+ descriptors.add(CLIENT_SERVICE);
+ descriptors.add(SPLIT_UP_HITS);
+ descriptors.add(SPLIT_UP_AGGREGATIONS);
+
+ propertyDescriptors = Collections.unmodifiableList(descriptors);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return propertyDescriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ clientService = context.getProperty(CLIENT_SERVICE).asControllerService(ElasticSearchClientService.class);
+ }
+
+ @OnStopped
+ public void onStopped() {
+ this.clientService = null;
+ }
+
+
+ private final ObjectMapper mapper = new ObjectMapper();
+
+ private String getQuery(FlowFile input, ProcessContext context, ProcessSession session) throws IOException {
+ String retVal = null;
+ if (context.getProperty(QUERY).isSet()) {
+ retVal = context.getProperty(QUERY).evaluateAttributeExpressions(input).getValue();
+ } else if (input != null) {
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+ session.exportTo(input, out);
--- End diff --
Looks like if the Query property isn't set and there is an incoming FF then it will use the FF's content as the query. I don't see this documented in the Query props. This should at least be documented and also ideally, there would be some custom validation too.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175304957
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java ---
@@ -61,22 +58,23 @@
public class ElasticSearchClientServiceImpl extends AbstractControllerService implements ElasticSearchClientService {
private ObjectMapper mapper = new ObjectMapper();
- private List<PropertyDescriptor> properties;
+ static final private List<PropertyDescriptor> properties;
private RestClient client;
private String url;
- @Override
- protected void init(ControllerServiceInitializationContext config) {
- properties = new ArrayList<>();
- properties.add(ElasticSearchClientService.HTTP_HOSTS);
- properties.add(ElasticSearchClientService.USERNAME);
- properties.add(ElasticSearchClientService.PASSWORD);
- properties.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
- properties.add(ElasticSearchClientService.CONNECT_TIMEOUT);
- properties.add(ElasticSearchClientService.SOCKET_TIMEOUT);
- properties.add(ElasticSearchClientService.RETRY_TIMEOUT);
+ static {
+ List _props = new ArrayList();
--- End diff --
Bit of a nit pick but can this init be changed so it's not of raw type:
> List\<PropertyDescriptor\> _props = new ArrayList\<\>();
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175264876
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java ---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+ @WritesAttribute(attribute = "mime.type", description = "application/json"),
+ @WritesAttribute(attribute = "aggregation.name", description = "The name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder().name("original")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship. " +
+ "This applies even if you select the \"split up hits\" option to send individual hits to the " +
+ "\"hits\" relationship.").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el-rest-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el-rest-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el-rest-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax. Ex: " +
+ "{\n" +
+ "\t\"query\": {\n" +
+ "\t\t\"match\": {\n" +
+ "\t\t\t\"name\": \"John Smith\"\n" +
+ "\t\t}\n" +
+ "\t}\n" +
+ "}")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUERY_ATTRIBUTE = new PropertyDescriptor.Builder()
+ .name("el-query-attribute")
+ .displayName("Query Attribute")
+ .description("If set, the executed query will be set on each result flowfile in the specified attribute.")
+ .expressionLanguageSupported(true)
+ .addValidator(Validator.VALID)
+ .required(false)
+ .build();
+
+ public static final AllowableValue SPLIT_UP_YES = new AllowableValue(
+ "splitUp-yes",
+ "Yes",
+ "Split up results."
+ );
+ public static final AllowableValue SPLIT_UP_HITS_NO = new AllowableValue(
+ "splitUp-no",
+ "No",
+ "Don't split up results."
+ );
+
+ public static final PropertyDescriptor SPLIT_UP_HITS = new PropertyDescriptor.Builder()
+ .name("el-rest-split-up-hits")
+ .displayName("Split up search results")
+ .description("Split up search results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+ public static final PropertyDescriptor SPLIT_UP_AGGREGATIONS = new PropertyDescriptor.Builder()
+ .name("el-rest-split-up-aggregations")
+ .displayName("Split up aggregation results")
+ .description("Split up aggregation results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+
+ public static final PropertyDescriptor CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name("el-rest-client-service")
+ .displayName("Client Service")
+ .description("An ElasticSearch client service to use for running queries.")
+ .identifiesControllerService(ElasticSearchClientService.class)
+ .required(true)
+ .build();
+
+ private static final Set<Relationship> relationships;
+ private static final List<PropertyDescriptor> propertyDescriptors;
+
+ private ElasticSearchClientService clientService;
+
+ static {
+ final Set<Relationship> _rels = new HashSet<>();
+ _rels.add(REL_ORIGINAL);
+ _rels.add(REL_FAILURE);
+ _rels.add(REL_HITS);
+ _rels.add(REL_AGGREGATIONS);
+ relationships = Collections.unmodifiableSet(_rels);
+
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(QUERY);
+ descriptors.add(QUERY_ATTRIBUTE);
+ descriptors.add(INDEX);
+ descriptors.add(TYPE);
+ descriptors.add(CLIENT_SERVICE);
+ descriptors.add(SPLIT_UP_HITS);
+ descriptors.add(SPLIT_UP_AGGREGATIONS);
+
+ propertyDescriptors = Collections.unmodifiableList(descriptors);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return propertyDescriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ clientService = context.getProperty(CLIENT_SERVICE).asControllerService(ElasticSearchClientService.class);
+ }
+
+ @OnStopped
+ public void onStopped() {
+ this.clientService = null;
+ }
+
+
+ private final ObjectMapper mapper = new ObjectMapper();
+
+ private String getQuery(FlowFile input, ProcessContext context, ProcessSession session) throws IOException {
+ String retVal = null;
+ if (context.getProperty(QUERY).isSet()) {
+ retVal = context.getProperty(QUERY).evaluateAttributeExpressions(input).getValue();
+ } else if (input != null) {
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+ session.exportTo(input, out);
+ out.close();
+
+ retVal = new String(out.toByteArray());
+ }
+
+ return retVal;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile input = null;
+ if (context.hasIncomingConnection()) {
+ input = session.get();
+
+ if (input == null && context.hasNonLoopConnection()) {
+ return;
+ }
+ }
+
+ try {
+
+ final String query = getQuery(input, context, session);
+ final String index = context.getProperty(INDEX).evaluateAttributeExpressions(input).getValue();
+ final String type = context.getProperty(TYPE).evaluateAttributeExpressions(input).getValue();
+
+ Optional<SearchResponse> resp = clientService.search(query, index, type);
+ if (resp.isPresent()) {
+ SearchResponse response = resp.get();
+
+ List<FlowFile> hitsFlowFiles = handleHits(response.getHits(), context, session, input);
+ List<FlowFile> aggsFlowFiles = handleAggregations(response.getAggregations(), context, session, input);
+
+ if (hitsFlowFiles.size() > 0) {
+ for (FlowFile ff : hitsFlowFiles) {
+ session.transfer(ff, REL_HITS);
--- End diff --
This should be able to be switched by to the single transfer since the root issue was setting the clientService to null on unScheduled.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175249262
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java ---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+ @WritesAttribute(attribute = "mime.type", description = "application/json"),
+ @WritesAttribute(attribute = "aggregation.name", description = "The name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder().name("original")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship. " +
+ "This applies even if you select the \"split up hits\" option to send individual hits to the " +
+ "\"hits\" relationship.").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el-rest-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el-rest-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el-rest-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax. Ex: " +
+ "{\n" +
+ "\t\"query\": {\n" +
+ "\t\t\"match\": {\n" +
+ "\t\t\t\"name\": \"John Smith\"\n" +
+ "\t\t}\n" +
+ "\t}\n" +
+ "}")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUERY_ATTRIBUTE = new PropertyDescriptor.Builder()
+ .name("el-query-attribute")
+ .displayName("Query Attribute")
+ .description("If set, the executed query will be set on each result flowfile in the specified attribute.")
+ .expressionLanguageSupported(true)
+ .addValidator(Validator.VALID)
+ .required(false)
+ .build();
+
+ public static final AllowableValue SPLIT_UP_YES = new AllowableValue(
+ "splitUp-yes",
+ "Yes",
+ "Split up results."
+ );
+ public static final AllowableValue SPLIT_UP_HITS_NO = new AllowableValue(
+ "splitUp-no",
+ "No",
+ "Don't split up results."
+ );
+
+ public static final PropertyDescriptor SPLIT_UP_HITS = new PropertyDescriptor.Builder()
+ .name("el-rest-split-up-hits")
+ .displayName("Split up search results")
+ .description("Split up search results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+ public static final PropertyDescriptor SPLIT_UP_AGGREGATIONS = new PropertyDescriptor.Builder()
+ .name("el-rest-split-up-aggregations")
+ .displayName("Split up aggregation results")
+ .description("Split up aggregation results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+
+ public static final PropertyDescriptor CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name("el-rest-client-service")
+ .displayName("Client Service")
+ .description("An ElasticSearch client service to use for running queries.")
+ .identifiesControllerService(ElasticSearchClientService.class)
+ .required(true)
+ .build();
+
+ private static final Set<Relationship> relationships;
+ private static final List<PropertyDescriptor> propertyDescriptors;
+
+ private ElasticSearchClientService clientService;
+
+ static {
+ final Set<Relationship> _rels = new HashSet<>();
+ _rels.add(REL_ORIGINAL);
+ _rels.add(REL_FAILURE);
+ _rels.add(REL_HITS);
+ _rels.add(REL_AGGREGATIONS);
+ relationships = Collections.unmodifiableSet(_rels);
+
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(QUERY);
+ descriptors.add(QUERY_ATTRIBUTE);
+ descriptors.add(INDEX);
+ descriptors.add(TYPE);
+ descriptors.add(CLIENT_SERVICE);
+ descriptors.add(SPLIT_UP_HITS);
+ descriptors.add(SPLIT_UP_AGGREGATIONS);
+
+ propertyDescriptors = Collections.unmodifiableList(descriptors);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return propertyDescriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ clientService = context.getProperty(CLIENT_SERVICE).asControllerService(ElasticSearchClientService.class);
+ }
+
+ @OnUnscheduled
+ public void onUnscheduled() {
+ this.clientService = null;
--- End diff --
This should be changed from "OnUnscheduled"[1] to "OnStopped"[2]. "OnUnscheduled" will be run as soon as the user clicks stop. This stops the processor from being schedueld again but there could still be lingering tasks running (thus leading to null pointer exceptions when it tries to use the clientService). "OnStopped" is run once all tasks finish.
[1] https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/annotation/lifecycle/OnUnscheduled.java
[2] https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/annotation/lifecycle/OnStopped.java
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on the issue:
https://github.com/apache/nifi/pull/2113
Looks like the master build is failing with some check style issues, hence why your builds failed. I'll work on fixing them.
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on the issue:
https://github.com/apache/nifi/pull/2113
+1
Overall looks good and works as expected. I've built with contrib check and tested using a vanilla instance. I will squash and merge to master. Thanks @MikeThomsen.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175265049
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java ---
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.PrintWriter;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends AbstractControllerService implements ElasticSearchClientService {
+ private ObjectMapper mapper = new ObjectMapper();
+
+ private List<PropertyDescriptor> properties;
+
+ private RestClient client;
+
+ private String url;
+
+ @Override
+ protected void init(ControllerServiceInitializationContext config) {
+ properties = new ArrayList<>();
+ properties.add(ElasticSearchClientService.HTTP_HOSTS);
+ properties.add(ElasticSearchClientService.USERNAME);
+ properties.add(ElasticSearchClientService.PASSWORD);
+ properties.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+ properties.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+ properties.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+ properties.add(ElasticSearchClientService.RETRY_TIMEOUT);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return properties;
+ }
+
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws InitializationException {
+ try {
+ setupClient(context);
+ } catch (Exception ex) {
+ getLogger().error("Could not initialize ElasticSearch client.", ex);
+ throw new InitializationException(ex);
+ }
+ }
+
+ @OnDisabled
+ public void onDisabled() throws IOException {
+ this.client.close();
+ this.url = null;
+ }
+
+ private void setupClient(ConfigurationContext context) throws Exception {
+ final String hosts = context.getProperty(HTTP_HOSTS).evaluateAttributeExpressions().getValue();
+ String[] hostsSplit = hosts.split(",[\\s]*");
+ this.url = hostsSplit[0];
+ final SSLContextService sslService =
+ context.getProperty(PROP_SSL_CONTEXT_SERVICE).asControllerService(SSLContextService.class);
+ final String username = context.getProperty(USERNAME).evaluateAttributeExpressions().getValue();
+ final String password = context.getProperty(PASSWORD).evaluateAttributeExpressions().getValue();
+
+ final Integer connectTimeout = context.getProperty(CONNECT_TIMEOUT).asInteger();
+ final Integer readTimeout = context.getProperty(SOCKET_TIMEOUT).asInteger();
+ final Integer retryTimeout = context.getProperty(RETRY_TIMEOUT).asInteger();
+
+ HttpHost[] hh = new HttpHost[hostsSplit.length];
+ for (int x = 0; x < hh.length; x++) {
+ URL u = new URL(hostsSplit[x]);
+ hh[x] = new HttpHost(u.getHost(), u.getPort(), u.getProtocol());
+ }
+
+ RestClientBuilder builder = RestClient.builder(hh)
+ .setHttpClientConfigCallback(httpClientBuilder -> {
+ if (sslService != null) {
+ try {
+ KeyStore keyStore = KeyStore.getInstance(sslService.getKeyStoreType());
+ KeyStore trustStore = KeyStore.getInstance("JKS");
+
+ try (final InputStream is = new FileInputStream(sslService.getKeyStoreFile())) {
+ keyStore.load(is, sslService.getKeyStorePassword().toCharArray());
+ }
+
+ try (final InputStream is = new FileInputStream(sslService.getTrustStoreFile())) {
+ trustStore.load(is, sslService.getTrustStorePassword().toCharArray());
+ }
+
+ final KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory
+ .getDefaultAlgorithm());
+ kmf.init(keyStore, sslService.getKeyStorePassword().toCharArray());
+ final TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory
+ .getDefaultAlgorithm());
+ tmf.init(keyStore);
+ SSLContext context1 = SSLContext.getInstance(sslService.getSslAlgorithm());
+ context1.init(kmf.getKeyManagers(), tmf.getTrustManagers(), new SecureRandom());
+ httpClientBuilder = httpClientBuilder.setSSLContext(context1);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ if (username != null && password != null) {
+ final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
+ credentialsProvider.setCredentials(AuthScope.ANY,
+ new UsernamePasswordCredentials(username, password));
+ httpClientBuilder = httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
+ }
+
+ return httpClientBuilder;
+ })
+ .setRequestConfigCallback(requestConfigBuilder -> {
+ try {
+ requestConfigBuilder.setConnectTimeout(connectTimeout);
+ requestConfigBuilder.setSocketTimeout(readTimeout);
+ return requestConfigBuilder;
+ } catch (Exception ex) {
+ try {
+ PrintWriter writer = new PrintWriter(new FileWriter("/tmp/out.log"));
--- End diff --
Why write to "/tmp/out.log"?
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
@JPercivall It's ready for review.
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
@JPercivall @mattyb149 Once I changed the code to use `transfer(FlowFile, Relationship)` and not `transfer(List, Relationship)` the error stopped happening.
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
@mattyb149 Have you had a chance to take a look at it?
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on the issue:
https://github.com/apache/nifi/pull/2113
Makes sense to me, we had to do the same with Elasticsearch 2.x and 5.x because of the difference in transport client versions. As folks move exclusively to ES5+, it would be nice to deprecate the other NARs and go forward with versions of all the processors (Get,Put,Search, etc.) using the new Java client.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r149127323
--- Diff: nifi-nar-bundles/nifi-standard-services/nifi-elasticsearch-client-service-api/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientService.java ---
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.elasticsearch;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ControllerService;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+
+import java.util.Map;
+import java.util.Optional;
+
+@Tags({"elasticsearch", "client"})
+@CapabilityDescription("A controller service for accessing an ElasticSearch client.")
+public interface ElasticSearchClientService extends ControllerService {
--- End diff --
The current Http versions of the ES processors have the future capability of configuring a proxy, although these properties are not yet exposed to the user (see [NIFI-4198](https://issues.apache.org/jira/browse/NIFI-4198) and #2094); consider adding proxy capabilities here, or creating an improvement Jira after this PR/Jira has been merged.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r150087242
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+ public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship. " +
+ "This applies even if you select the \"break up hits\" option to send individual hits to the " +
+ "\"hits\" relationship.").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el5-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el5-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .defaultValue("")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(Validator.VALID)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el5-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax.")
--- End diff --
Yeah, good point.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175420184
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java ---
@@ -0,0 +1,329 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+ @WritesAttribute(attribute = "mime.type", description = "application/json"),
+ @WritesAttribute(attribute = "aggregation.name", description = "The name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder().name("original")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship. " +
+ "This applies even if you select the \"split up hits\" option to send individual hits to the " +
+ "\"hits\" relationship.").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el-rest-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el-rest-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el-rest-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax. Ex: " +
+ "{\n" +
+ "\t\"query\": {\n" +
+ "\t\t\"match\": {\n" +
+ "\t\t\t\"name\": \"John Smith\"\n" +
+ "\t\t}\n" +
+ "\t}\n" +
+ "}")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUERY_ATTRIBUTE = new PropertyDescriptor.Builder()
--- End diff --
Fixed.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175271885
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java ---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+ @WritesAttribute(attribute = "mime.type", description = "application/json"),
+ @WritesAttribute(attribute = "aggregation.name", description = "The name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
--- End diff --
Ok. I should also note that scroll queries are not supported.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175306495
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java ---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+ @WritesAttribute(attribute = "mime.type", description = "application/json"),
+ @WritesAttribute(attribute = "aggregation.name", description = "The name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
--- End diff --
I don't see this in the latest commit
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175264906
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java ---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+ @WritesAttribute(attribute = "mime.type", description = "application/json"),
+ @WritesAttribute(attribute = "aggregation.name", description = "The name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder().name("original")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship. " +
+ "This applies even if you select the \"split up hits\" option to send individual hits to the " +
+ "\"hits\" relationship.").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el-rest-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el-rest-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el-rest-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax. Ex: " +
+ "{\n" +
+ "\t\"query\": {\n" +
+ "\t\t\"match\": {\n" +
+ "\t\t\t\"name\": \"John Smith\"\n" +
+ "\t\t}\n" +
+ "\t}\n" +
+ "}")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUERY_ATTRIBUTE = new PropertyDescriptor.Builder()
+ .name("el-query-attribute")
+ .displayName("Query Attribute")
+ .description("If set, the executed query will be set on each result flowfile in the specified attribute.")
+ .expressionLanguageSupported(true)
+ .addValidator(Validator.VALID)
+ .required(false)
+ .build();
+
+ public static final AllowableValue SPLIT_UP_YES = new AllowableValue(
+ "splitUp-yes",
+ "Yes",
+ "Split up results."
+ );
+ public static final AllowableValue SPLIT_UP_HITS_NO = new AllowableValue(
+ "splitUp-no",
+ "No",
+ "Don't split up results."
+ );
+
+ public static final PropertyDescriptor SPLIT_UP_HITS = new PropertyDescriptor.Builder()
+ .name("el-rest-split-up-hits")
+ .displayName("Split up search results")
+ .description("Split up search results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+ public static final PropertyDescriptor SPLIT_UP_AGGREGATIONS = new PropertyDescriptor.Builder()
+ .name("el-rest-split-up-aggregations")
+ .displayName("Split up aggregation results")
+ .description("Split up aggregation results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+
+ public static final PropertyDescriptor CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name("el-rest-client-service")
+ .displayName("Client Service")
+ .description("An ElasticSearch client service to use for running queries.")
+ .identifiesControllerService(ElasticSearchClientService.class)
+ .required(true)
+ .build();
+
+ private static final Set<Relationship> relationships;
+ private static final List<PropertyDescriptor> propertyDescriptors;
+
+ private ElasticSearchClientService clientService;
--- End diff --
This should be set as volatile since it is referenced by multiple different threads.
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on the issue:
https://github.com/apache/nifi/pull/2113
Reviewing
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
@mattyb149 Ok. Changes are made. I refactored the commit to be based on a controller service. For now, that service only handles a single function: basic search. However, there is now a (hopefully) extensible base via a controller service to move forward. The controller service has integration tests only; the processor has junit tests with a mock service.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r150087559
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+ public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship. " +
+ "This applies even if you select the \"break up hits\" option to send individual hits to the " +
+ "\"hits\" relationship.").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el5-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el5-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .defaultValue("")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(Validator.VALID)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el5-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax.")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final AllowableValue BREAK_UP_YES = new AllowableValue(
+ "breakup-yes",
+ "Yes",
+ "Break up results."
+ );
+ public static final AllowableValue BREAK_UP_HITS_NO = new AllowableValue(
+ "breakup-no",
+ "No",
+ "Don't break up results."
+ );
+
+ public static final PropertyDescriptor BREAK_UP_HITS = new PropertyDescriptor.Builder()
+ .name("el5-break-up-hits")
+ .displayName("Break up search results")
--- End diff --
Fair enough.
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on the issue:
https://github.com/apache/nifi/pull/2113
I'm seeing NPE and FlowFile handling exceptions. I didn't do anything special, just using the example query. I'll attach a template. Errors from the logs below.
> 2018-03-12 23:58:11,386 ERROR [Timer-Driven Process Thread-10] o.a.n.p.e.JsonQueryElasticsearch JsonQueryElasticsearch[id=1d5d83d1-0162-1000-cdf2-92fa9e61ef42] Error processing flowfile.: java.lang.NullPointerException
java.lang.NullPointerException: null
at org.apache.nifi.processors.elasticsearch.JsonQueryElasticsearch.onTrigger(JsonQueryElasticsearch.java:248)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1123)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2018-03-12 23:58:11,389 ERROR [Timer-Driven Process Thread-10] o.a.n.p.e.JsonQueryElasticsearch JsonQueryElasticsearch[id=1d5d83d1-0162-1000-cdf2-92fa9e61ef42] JsonQueryElasticsearch[id=1d5d83d1-0162-1000-cdf2-92fa9e61ef42] failed to process due to org.apache.nifi.processor.exception.FlowFileHandlingException: StandardFlowFileRecord[uuid=77ad5fff-d4b6-4e11-8d8c-b07ac1d0a6cb,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1520913052434-1, container=default, section=1], offset=30, length=228],offset=226,name=46172028496405,size=2] transfer relationship not specified; rolling back session: {}
org.apache.nifi.processor.exception.FlowFileHandlingException: StandardFlowFileRecord[uuid=77ad5fff-d4b6-4e11-8d8c-b07ac1d0a6cb,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1520913052434-1, container=default, section=1], offset=30, length=228],offset=226,name=46172028496405,size=2] transfer relationship not specified
at org.apache.nifi.controller.repository.StandardProcessSession.checkpoint(StandardProcessSession.java:251)
at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:321)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1123)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175271747
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java ---
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.PrintWriter;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends AbstractControllerService implements ElasticSearchClientService {
+ private ObjectMapper mapper = new ObjectMapper();
+
+ private List<PropertyDescriptor> properties;
+
+ private RestClient client;
+
+ private String url;
+
+ @Override
+ protected void init(ControllerServiceInitializationContext config) {
+ properties = new ArrayList<>();
+ properties.add(ElasticSearchClientService.HTTP_HOSTS);
+ properties.add(ElasticSearchClientService.USERNAME);
+ properties.add(ElasticSearchClientService.PASSWORD);
+ properties.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+ properties.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+ properties.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+ properties.add(ElasticSearchClientService.RETRY_TIMEOUT);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return properties;
+ }
+
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws InitializationException {
+ try {
+ setupClient(context);
+ } catch (Exception ex) {
+ getLogger().error("Could not initialize ElasticSearch client.", ex);
+ throw new InitializationException(ex);
+ }
+ }
+
+ @OnDisabled
+ public void onDisabled() throws IOException {
+ this.client.close();
+ this.url = null;
+ }
+
+ private void setupClient(ConfigurationContext context) throws Exception {
--- End diff --
Done.
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
> I hope my review doesn't come off in the wrong way, there is a lot of great work here and I just want to make sure the usability is top notch.
Not at all. It's all fair and good feedback. I'll take a crack at these tomorrow.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175271557
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java ---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+ @WritesAttribute(attribute = "mime.type", description = "application/json"),
+ @WritesAttribute(attribute = "aggregation.name", description = "The name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder().name("original")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship. " +
+ "This applies even if you select the \"split up hits\" option to send individual hits to the " +
+ "\"hits\" relationship.").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el-rest-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el-rest-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el-rest-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax. Ex: " +
+ "{\n" +
+ "\t\"query\": {\n" +
+ "\t\t\"match\": {\n" +
+ "\t\t\t\"name\": \"John Smith\"\n" +
+ "\t\t}\n" +
+ "\t}\n" +
+ "}")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUERY_ATTRIBUTE = new PropertyDescriptor.Builder()
+ .name("el-query-attribute")
+ .displayName("Query Attribute")
+ .description("If set, the executed query will be set on each result flowfile in the specified attribute.")
+ .expressionLanguageSupported(true)
+ .addValidator(Validator.VALID)
+ .required(false)
+ .build();
+
+ public static final AllowableValue SPLIT_UP_YES = new AllowableValue(
+ "splitUp-yes",
+ "Yes",
+ "Split up results."
+ );
+ public static final AllowableValue SPLIT_UP_HITS_NO = new AllowableValue(
+ "splitUp-no",
+ "No",
+ "Don't split up results."
+ );
+
+ public static final PropertyDescriptor SPLIT_UP_HITS = new PropertyDescriptor.Builder()
+ .name("el-rest-split-up-hits")
+ .displayName("Split up search results")
+ .description("Split up search results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+ public static final PropertyDescriptor SPLIT_UP_AGGREGATIONS = new PropertyDescriptor.Builder()
+ .name("el-rest-split-up-aggregations")
+ .displayName("Split up aggregation results")
+ .description("Split up aggregation results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+
+ public static final PropertyDescriptor CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name("el-rest-client-service")
+ .displayName("Client Service")
+ .description("An ElasticSearch client service to use for running queries.")
+ .identifiesControllerService(ElasticSearchClientService.class)
+ .required(true)
+ .build();
+
+ private static final Set<Relationship> relationships;
+ private static final List<PropertyDescriptor> propertyDescriptors;
+
+ private ElasticSearchClientService clientService;
--- End diff --
Done.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r157902956
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+ @WritesAttribute(attribute = "mime.type", description = "application/json"),
+ @WritesAttribute(attribute = "aggregation.name", description = "The name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder().name("original")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship. " +
+ "This applies even if you select the \"split up hits\" option to send individual hits to the " +
+ "\"hits\" relationship.").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el5-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el5-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el5-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax. Ex: " +
+ "{\n" +
+ "\t\"query\": {\n" +
+ "\t\t\"match\": {\n" +
+ "\t\t\t\"name\": \"John Smith\"\n" +
+ "\t\t}\n" +
+ "\t}\n" +
+ "}")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final AllowableValue SPLIT_UP_YES = new AllowableValue(
+ "splitUp-yes",
+ "Yes",
+ "Split up results."
+ );
+ public static final AllowableValue SPLIT_UP_HITS_NO = new AllowableValue(
+ "splitUp-no",
+ "No",
+ "Don't split up results."
+ );
+
+ public static final PropertyDescriptor SPLIT_UP_HITS = new PropertyDescriptor.Builder()
+ .name("el5-split-up-hits")
+ .displayName("Split up search results")
+ .description("Split up search results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+ public static final PropertyDescriptor SPLIT_UP_AGGREGATIONS = new PropertyDescriptor.Builder()
+ .name("el5-split-up-aggregations")
+ .displayName("Split up aggregation results")
+ .description("Split up aggregation results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+
+ public static final PropertyDescriptor CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name("el5-client-service")
+ .displayName("Client Service")
+ .description("An ElasticSearch client service to use for running queries.")
+ .identifiesControllerService(ElasticSearchClientService.class)
+ .required(true)
+ .build();
+
+ private static final Set<Relationship> relationships;
+ private static final List<PropertyDescriptor> propertyDescriptors;
+
+ private ElasticSearchClientService clientService;
+
+ static {
+ final Set<Relationship> _rels = new HashSet<>();
+ _rels.add(REL_ORIGINAL);
+ _rels.add(REL_FAILURE);
+ _rels.add(REL_HITS);
+ _rels.add(REL_AGGREGATIONS);
+ relationships = Collections.unmodifiableSet(_rels);
+
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(QUERY);
+ descriptors.add(INDEX);
+ descriptors.add(TYPE);
+ descriptors.add(CLIENT_SERVICE);
+ descriptors.add(SPLIT_UP_HITS);
+ descriptors.add(SPLIT_UP_AGGREGATIONS);
+
+ propertyDescriptors = Collections.unmodifiableList(descriptors);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return propertyDescriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ clientService = context.getProperty(CLIENT_SERVICE).asControllerService(ElasticSearchClientService.class);
+ }
+
+ @OnUnscheduled
+ public void onUnscheduled() {
+ this.clientService = null;
+ }
+
+
+ private ObjectMapper mapper = new ObjectMapper();
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ return;
+ }
+
+ try {
+
+ String query = context.getProperty(QUERY).evaluateAttributeExpressions(flowFile).getValue();
+ String index = context.getProperty(INDEX).evaluateAttributeExpressions(flowFile).getValue();
+ String type = context.getProperty(TYPE).evaluateAttributeExpressions(flowFile).getValue();
+
+ Optional<ElasticSearchResponse> resp = clientService.search(query, index, type);
+ ElasticSearchResponse response = resp.get();
--- End diff --
the search() method returns an Optional, any chance the response object might not be present? Perhaps check isPresent() and send any incoming (not created) flow file to failure?
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175411128
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java ---
@@ -61,22 +58,23 @@
public class ElasticSearchClientServiceImpl extends AbstractControllerService implements ElasticSearchClientService {
private ObjectMapper mapper = new ObjectMapper();
- private List<PropertyDescriptor> properties;
+ static final private List<PropertyDescriptor> properties;
private RestClient client;
private String url;
- @Override
- protected void init(ControllerServiceInitializationContext config) {
- properties = new ArrayList<>();
- properties.add(ElasticSearchClientService.HTTP_HOSTS);
- properties.add(ElasticSearchClientService.USERNAME);
- properties.add(ElasticSearchClientService.PASSWORD);
- properties.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
- properties.add(ElasticSearchClientService.CONNECT_TIMEOUT);
- properties.add(ElasticSearchClientService.SOCKET_TIMEOUT);
- properties.add(ElasticSearchClientService.RETRY_TIMEOUT);
+ static {
+ List _props = new ArrayList();
--- End diff --
Done.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r174078307
--- Diff: nifi-assembly/pom.xml ---
@@ -445,6 +445,24 @@ language governing permissions and limitations under the License. -->
<version>1.6.0-SNAPSHOT</version>
<type>nar</type>
</dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-elasticsearch-client-service-api-nar</artifactId>
--- End diff --
Weird, I thought that I took that out.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175306133
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java ---
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends AbstractControllerService implements ElasticSearchClientService {
+ private ObjectMapper mapper = new ObjectMapper();
+
+ static final private List<PropertyDescriptor> properties;
+
+ private RestClient client;
+
+ private String url;
+
+ static {
+ List _props = new ArrayList();
+ _props.add(ElasticSearchClientService.HTTP_HOSTS);
+ _props.add(ElasticSearchClientService.USERNAME);
+ _props.add(ElasticSearchClientService.PASSWORD);
+ _props.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+ _props.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+ _props.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+ _props.add(ElasticSearchClientService.RETRY_TIMEOUT);
+
+ properties = Collections.unmodifiableList(_props);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return properties;
+ }
+
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws InitializationException {
+ try {
+ setupClient(context);
+ } catch (Exception ex) {
+ getLogger().error("Could not initialize ElasticSearch client.", ex);
+ throw new InitializationException(ex);
+ }
+ }
+
+ @OnDisabled
+ public void onDisabled() throws IOException {
+ this.client.close();
+ this.url = null;
+ }
+
+ private void setupClient(ConfigurationContext context) throws MalformedURLException {
+ final String hosts = context.getProperty(HTTP_HOSTS).evaluateAttributeExpressions().getValue();
+ String[] hostsSplit = hosts.split(",[\\s]*");
+ this.url = hostsSplit[0];
+ final SSLContextService sslService =
+ context.getProperty(PROP_SSL_CONTEXT_SERVICE).asControllerService(SSLContextService.class);
+ final String username = context.getProperty(USERNAME).evaluateAttributeExpressions().getValue();
+ final String password = context.getProperty(PASSWORD).evaluateAttributeExpressions().getValue();
+
+ final Integer connectTimeout = context.getProperty(CONNECT_TIMEOUT).asInteger();
+ final Integer readTimeout = context.getProperty(SOCKET_TIMEOUT).asInteger();
+ final Integer retryTimeout = context.getProperty(RETRY_TIMEOUT).asInteger();
+
+ HttpHost[] hh = new HttpHost[hostsSplit.length];
+ for (int x = 0; x < hh.length; x++) {
+ URL u = new URL(hostsSplit[x]);
+ hh[x] = new HttpHost(u.getHost(), u.getPort(), u.getProtocol());
+ }
+
+ RestClientBuilder builder = RestClient.builder(hh)
+ .setHttpClientConfigCallback(httpClientBuilder -> {
+ if (sslService != null && sslService.isKeyStoreConfigured() && sslService.isTrustStoreConfigured()) {
+ try {
+ KeyStore keyStore = KeyStore.getInstance(sslService.getKeyStoreType());
+ KeyStore trustStore = KeyStore.getInstance("JKS");
+
+ try (final InputStream is = new FileInputStream(sslService.getKeyStoreFile())) {
+ keyStore.load(is, sslService.getKeyStorePassword().toCharArray());
+ }
+
+ try (final InputStream is = new FileInputStream(sslService.getTrustStoreFile())) {
+ trustStore.load(is, sslService.getTrustStorePassword().toCharArray());
+ }
+
+ final KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory
+ .getDefaultAlgorithm());
+ kmf.init(keyStore, sslService.getKeyStorePassword().toCharArray());
+ final TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory
+ .getDefaultAlgorithm());
+ tmf.init(keyStore);
+ SSLContext context1 = SSLContext.getInstance(sslService.getSslAlgorithm());
+ context1.init(kmf.getKeyManagers(), tmf.getTrustManagers(), new SecureRandom());
+ httpClientBuilder = httpClientBuilder.setSSLContext(context1);
+ } catch (Exception e) {
+ getLogger().error("Error setting up SSL.", e);
+ }
+ }
+
+ if (username != null && password != null) {
+ final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
+ credentialsProvider.setCredentials(AuthScope.ANY,
+ new UsernamePasswordCredentials(username, password));
+ httpClientBuilder = httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
+ }
+
+ return httpClientBuilder;
+ })
+ .setRequestConfigCallback(requestConfigBuilder -> {
+ requestConfigBuilder.setConnectTimeout(connectTimeout);
+ requestConfigBuilder.setSocketTimeout(readTimeout);
+ return requestConfigBuilder;
+ })
+ .setMaxRetryTimeoutMillis(retryTimeout);
+
+ this.client = builder.build();
+ }
+
+ private Response runQuery(String query, String index, String type) throws IOException {
+ StringBuilder sb = new StringBuilder()
+ .append("/")
+ .append(index);
+ if (type != null && !type.equals("")) {
+ sb.append("/")
+ .append(type);
+ }
+
+ sb.append("/_search");
+
+ HttpEntity queryEntity = new NStringEntity(query, ContentType.APPLICATION_JSON);
+
+ return client.performRequest("POST", sb.toString(), Collections.emptyMap(), queryEntity);
+ }
+
+ @Override
+ public Optional<SearchResponse> search(String query, String index, String type) throws IOException {
+ Response response = runQuery(query, index, type);
+ Map<String, Object> parsed = mapper.readValue(IOUtils.toString(response.getEntity().getContent()), Map.class);
--- End diff --
2 things:
1. IOUtils.toString is deprecated, you should pass the expected charset to use.
2. In general, you should avoid doing a lot of nesting of logic into a single line. It helps a lot with readability and diagnosing errors. For example, if we hit an NPE here, it could be for any of 3 different objects. At the very least, the IOUtils call to get the response content as a string should be on another line.
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on the issue:
https://github.com/apache/nifi/pull/2113
Hey @MikeThomsen, I'm planning on reviewing this tomorrow evening
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175411050
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java ---
@@ -121,7 +119,7 @@ private void setupClient(ConfigurationContext context) throws Exception {
RestClientBuilder builder = RestClient.builder(hh)
.setHttpClientConfigCallback(httpClientBuilder -> {
- if (sslService != null) {
+ if (sslService != null && sslService.isKeyStoreConfigured() && sslService.isTrustStoreConfigured()) {
try {
--- End diff --
Done. I moved that logic out and have the it bubble it up with an InitializationException if any of those exceptions are thrown.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175407337
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java ---
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends AbstractControllerService implements ElasticSearchClientService {
+ private ObjectMapper mapper = new ObjectMapper();
+
+ static final private List<PropertyDescriptor> properties;
+
+ private RestClient client;
+
+ private String url;
+
+ static {
+ List _props = new ArrayList();
+ _props.add(ElasticSearchClientService.HTTP_HOSTS);
+ _props.add(ElasticSearchClientService.USERNAME);
+ _props.add(ElasticSearchClientService.PASSWORD);
+ _props.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+ _props.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+ _props.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+ _props.add(ElasticSearchClientService.RETRY_TIMEOUT);
+
+ properties = Collections.unmodifiableList(_props);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return properties;
+ }
+
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws InitializationException {
+ try {
+ setupClient(context);
+ } catch (Exception ex) {
+ getLogger().error("Could not initialize ElasticSearch client.", ex);
+ throw new InitializationException(ex);
+ }
+ }
+
+ @OnDisabled
+ public void onDisabled() throws IOException {
+ this.client.close();
+ this.url = null;
+ }
+
+ private void setupClient(ConfigurationContext context) throws MalformedURLException {
+ final String hosts = context.getProperty(HTTP_HOSTS).evaluateAttributeExpressions().getValue();
+ String[] hostsSplit = hosts.split(",[\\s]*");
+ this.url = hostsSplit[0];
+ final SSLContextService sslService =
+ context.getProperty(PROP_SSL_CONTEXT_SERVICE).asControllerService(SSLContextService.class);
+ final String username = context.getProperty(USERNAME).evaluateAttributeExpressions().getValue();
+ final String password = context.getProperty(PASSWORD).evaluateAttributeExpressions().getValue();
+
+ final Integer connectTimeout = context.getProperty(CONNECT_TIMEOUT).asInteger();
+ final Integer readTimeout = context.getProperty(SOCKET_TIMEOUT).asInteger();
+ final Integer retryTimeout = context.getProperty(RETRY_TIMEOUT).asInteger();
+
+ HttpHost[] hh = new HttpHost[hostsSplit.length];
+ for (int x = 0; x < hh.length; x++) {
+ URL u = new URL(hostsSplit[x]);
+ hh[x] = new HttpHost(u.getHost(), u.getPort(), u.getProtocol());
+ }
+
+ RestClientBuilder builder = RestClient.builder(hh)
+ .setHttpClientConfigCallback(httpClientBuilder -> {
+ if (sslService != null && sslService.isKeyStoreConfigured() && sslService.isTrustStoreConfigured()) {
+ try {
+ KeyStore keyStore = KeyStore.getInstance(sslService.getKeyStoreType());
+ KeyStore trustStore = KeyStore.getInstance("JKS");
+
+ try (final InputStream is = new FileInputStream(sslService.getKeyStoreFile())) {
+ keyStore.load(is, sslService.getKeyStorePassword().toCharArray());
+ }
+
+ try (final InputStream is = new FileInputStream(sslService.getTrustStoreFile())) {
+ trustStore.load(is, sslService.getTrustStorePassword().toCharArray());
+ }
+
+ final KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory
+ .getDefaultAlgorithm());
+ kmf.init(keyStore, sslService.getKeyStorePassword().toCharArray());
+ final TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory
+ .getDefaultAlgorithm());
+ tmf.init(keyStore);
+ SSLContext context1 = SSLContext.getInstance(sslService.getSslAlgorithm());
+ context1.init(kmf.getKeyManagers(), tmf.getTrustManagers(), new SecureRandom());
+ httpClientBuilder = httpClientBuilder.setSSLContext(context1);
+ } catch (Exception e) {
+ getLogger().error("Error setting up SSL.", e);
+ }
+ }
+
+ if (username != null && password != null) {
+ final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
+ credentialsProvider.setCredentials(AuthScope.ANY,
+ new UsernamePasswordCredentials(username, password));
+ httpClientBuilder = httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
+ }
+
+ return httpClientBuilder;
+ })
+ .setRequestConfigCallback(requestConfigBuilder -> {
+ requestConfigBuilder.setConnectTimeout(connectTimeout);
+ requestConfigBuilder.setSocketTimeout(readTimeout);
+ return requestConfigBuilder;
+ })
+ .setMaxRetryTimeoutMillis(retryTimeout);
+
+ this.client = builder.build();
+ }
+
+ private Response runQuery(String query, String index, String type) throws IOException {
+ StringBuilder sb = new StringBuilder()
+ .append("/")
+ .append(index);
+ if (type != null && !type.equals("")) {
+ sb.append("/")
+ .append(type);
+ }
+
+ sb.append("/_search");
+
+ HttpEntity queryEntity = new NStringEntity(query, ContentType.APPLICATION_JSON);
+
+ return client.performRequest("POST", sb.toString(), Collections.emptyMap(), queryEntity);
+ }
+
+ @Override
+ public Optional<SearchResponse> search(String query, String index, String type) throws IOException {
+ Response response = runQuery(query, index, type);
+ Map<String, Object> parsed = mapper.readValue(IOUtils.toString(response.getEntity().getContent()), Map.class);
+
+ int took = (Integer)parsed.get("took");
+ boolean timedOut = (Boolean)parsed.get("timed_out");
+ Map<String, Object> aggregations = parsed.get("aggregations") != null
+ ? (Map<String, Object>)parsed.get("aggregations") : new HashMap<>();
+ Map<String, Object> hitsParent = (Map<String, Object>)parsed.get("hits");
+ int count = (Integer)hitsParent.get("total");
+ List<Map<String, Object>> hits = (List<Map<String, Object>>)hitsParent.get("hits");
+
+ SearchResponse esr = new SearchResponse(hits, aggregations, count, took, timedOut);
+
+ if (getLogger().isDebugEnabled()) {
+ StringBuilder sb = new StringBuilder();
+ sb.append("******************");
+ sb.append(String.format("Took: %d", took));
+ sb.append(String.format("Timed out: %s", timedOut));
+ sb.append(String.format("Aggregation count: %d", aggregations.size()));
+ sb.append(String.format("Hit count: %d", hits.size()));
+ sb.append(String.format("Total found: %d", count));
+ sb.append("******************");
+
+ getLogger().debug(sb.toString());
+ }
+
+ return Optional.of(esr);
--- End diff --
Probably not.
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
@mattyb149 I'm going to leave this open, but I decided to refactor the heck out of it around a client service for ElasticSearch. The service only has one method for now, but I think it's the way to go so that in the future as services become injectable in scripts and such, it'll be more flexible.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175305459
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java ---
@@ -0,0 +1,329 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+ @WritesAttribute(attribute = "mime.type", description = "application/json"),
+ @WritesAttribute(attribute = "aggregation.name", description = "The name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder().name("original")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship. " +
+ "This applies even if you select the \"split up hits\" option to send individual hits to the " +
+ "\"hits\" relationship.").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el-rest-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el-rest-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el-rest-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax. Ex: " +
+ "{\n" +
+ "\t\"query\": {\n" +
+ "\t\t\"match\": {\n" +
+ "\t\t\t\"name\": \"John Smith\"\n" +
+ "\t\t}\n" +
+ "\t}\n" +
+ "}")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUERY_ATTRIBUTE = new PropertyDescriptor.Builder()
--- End diff --
Where is this prop used? I only see it referenced when it's added to the descriptors. I do see the name named attribute used a lot in the mongo processors though.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175411532
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java ---
@@ -0,0 +1,329 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+ @WritesAttribute(attribute = "mime.type", description = "application/json"),
+ @WritesAttribute(attribute = "aggregation.name", description = "The name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder().name("original")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship. " +
+ "This applies even if you select the \"split up hits\" option to send individual hits to the " +
+ "\"hits\" relationship.").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el-rest-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el-rest-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el-rest-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax. Ex: " +
+ "{\n" +
+ "\t\"query\": {\n" +
+ "\t\t\"match\": {\n" +
+ "\t\t\t\"name\": \"John Smith\"\n" +
+ "\t\t}\n" +
+ "\t}\n" +
+ "}")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUERY_ATTRIBUTE = new PropertyDescriptor.Builder()
+ .name("el-query-attribute")
+ .displayName("Query Attribute")
+ .description("If set, the executed query will be set on each result flowfile in the specified attribute.")
+ .expressionLanguageSupported(true)
+ .addValidator(Validator.VALID)
+ .required(false)
+ .build();
+
+ public static final AllowableValue SPLIT_UP_YES = new AllowableValue(
+ "splitUp-yes",
+ "Yes",
+ "Split up results."
+ );
+ public static final AllowableValue SPLIT_UP_HITS_NO = new AllowableValue(
+ "splitUp-no",
+ "No",
+ "Don't split up results."
+ );
+
+ public static final PropertyDescriptor SPLIT_UP_HITS = new PropertyDescriptor.Builder()
+ .name("el-rest-split-up-hits")
+ .displayName("Split up search results")
+ .description("Split up search results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+ public static final PropertyDescriptor SPLIT_UP_AGGREGATIONS = new PropertyDescriptor.Builder()
+ .name("el-rest-split-up-aggregations")
+ .displayName("Split up aggregation results")
+ .description("Split up aggregation results into one flowfile per result.")
+ .allowableValues(SPLIT_UP_HITS_NO, SPLIT_UP_YES)
+ .defaultValue(SPLIT_UP_HITS_NO.getValue())
+ .required(true)
+ .expressionLanguageSupported(false)
+ .build();
+
+ public static final PropertyDescriptor CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name("el-rest-client-service")
+ .displayName("Client Service")
+ .description("An ElasticSearch client service to use for running queries.")
+ .identifiesControllerService(ElasticSearchClientService.class)
+ .required(true)
+ .build();
+
+ private static final Set<Relationship> relationships;
+ private static final List<PropertyDescriptor> propertyDescriptors;
+
+ private volatile ElasticSearchClientService clientService;
+
+ static {
+ final Set<Relationship> _rels = new HashSet<>();
+ _rels.add(REL_ORIGINAL);
+ _rels.add(REL_FAILURE);
+ _rels.add(REL_HITS);
+ _rels.add(REL_AGGREGATIONS);
+ relationships = Collections.unmodifiableSet(_rels);
+
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(QUERY);
+ descriptors.add(QUERY_ATTRIBUTE);
+ descriptors.add(INDEX);
+ descriptors.add(TYPE);
+ descriptors.add(CLIENT_SERVICE);
+ descriptors.add(SPLIT_UP_HITS);
+ descriptors.add(SPLIT_UP_AGGREGATIONS);
+
+ propertyDescriptors = Collections.unmodifiableList(descriptors);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return propertyDescriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ clientService = context.getProperty(CLIENT_SERVICE).asControllerService(ElasticSearchClientService.class);
+ }
+
+ @OnStopped
+ public void onStopped() {
+ this.clientService = null;
+ }
+
+
+ private final ObjectMapper mapper = new ObjectMapper();
+
+ private String getQuery(FlowFile input, ProcessContext context, ProcessSession session) throws IOException {
+ String retVal = null;
+ if (context.getProperty(QUERY).isSet()) {
+ retVal = context.getProperty(QUERY).evaluateAttributeExpressions(input).getValue();
+ } else if (input != null) {
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+ session.exportTo(input, out);
--- End diff --
Yeah, it needs the documentation. That was a recent addition. I'm not sure about the validation because it's expecting a JSON payload, and if that's malformed in any respect the client service will throw an exception back to the processor now.
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
@JPercivall I just got done testing it w/ X-Pack for SSL and u/p auth based on Elastic docs. Everything was in working order there. [Steps if you want to validate](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/configuring-tls.html)
I also added the documentation there. Sorry about that.
AFAIK, that warning is going to be more of a polite warning than a real "here be dragons" sort of thing because IIRC ES limits you to 10k results unless you explicitly up the number in the index configuration. Probably a good reminder, but I doubt anyone is going to blow up their NiFi installation without having to take deliberate steps to facilitate that.
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on the issue:
https://github.com/apache/nifi/pull/2113
Template for FlowFile handling exception.
[FlowFile_Handling_Exception.txt](https://github.com/apache/nifi/files/1805420/FlowFile_Handling_Exception.txt)
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
Thanks @mattyb149 I'll try to find some time to address these today or tomorrow. I've been knee deep in some Mongo-related work for a client. I have two pull requests for Mongo and record api-related fixes/updates if you're up for reviewing two small patches (one is 3 lines of code :) )
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by joewitt <gi...@git.apache.org>.
Github user joewitt commented on the issue:
https://github.com/apache/nifi/pull/2113
@MikeThomsen It is awesome that you're contributing code/PRs. As you can see though we do have quite a large backlog of PRs that need review/feedback/cycles. Some of them, like integration with Mongo involve more effort than others to help test and review. Please do consider helping by participating in/conducting reviews of others contributions as well. That might help things like this be evaluated faster as there would be less overall backlog. Thanks
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r157902648
--- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java ---
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+ @WritesAttribute(attribute = "mime.type"),
+ @WritesAttribute(attribute = "aggregation.name", description = "The name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query (with aggregations) written with the " +
+ "ElasticSearch JSON DSL. It currently does not support pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder().name("original")
+ .description("All original flowfiles that don't cause an error to occur go to this relationship. " +
+ "This applies even if you select the \"split up hits\" option to send individual hits to the " +
+ "\"hits\" relationship.").build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure")
+ .description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build();
+
+ public static final Relationship REL_HITS = new Relationship.Builder().name("hits")
+ .description("Search hits are routed to this relationship.")
+ .build();
+
+ public static final Relationship REL_AGGREGATIONS = new Relationship.Builder().name("aggregations")
+ .description("Aggregations are routed to this relationship.")
+ .build();
+
+ public static final PropertyDescriptor INDEX = new PropertyDescriptor.Builder()
+ .name("el5-fetch-index")
+ .displayName("Index")
+ .description("The name of the index to read from")
+ .required(true)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor TYPE = new PropertyDescriptor.Builder()
+ .name("el5-type")
+ .displayName("Type")
+ .description("The type of this document (used by Elasticsearch for indexing and searching)")
+ .required(false)
+ .expressionLanguageSupported(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder()
+ .name("el5-query")
+ .displayName("Query")
+ .description("A query in JSON syntax, not Lucene syntax. Ex: " +
+ "{\n" +
--- End diff --
No worries, just having an example (whether pretty-printed or not) is helpful
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on the issue:
https://github.com/apache/nifi/pull/2113
Apologises, won't be able to finish the review this evening. I reviewed the restapi nar License and Notice files they look good to go. Just need to do a bit of functional testing and verify @mattyb149 comments were addressed and we should be ready to merge (pending the L/N for the service nar).
---