You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by bbende <gi...@git.apache.org> on 2015/03/24 01:18:59 UTC
[GitHub] incubator-nifi pull request: NIFI-461 Add processors to interact w...
GitHub user bbende opened a pull request:
https://github.com/apache/incubator-nifi/pull/43
NIFI-461 Add processors to interact with Apache Solr
This pull request provides two processors:
* GetSolr - extracts data from Solr by running a query and sending the results out in XML, the intent is to run on a timer, or cron, and only get the results since last execution, similar to GetHttp.
* PutSolrContentStream - streams the content of a FlowFile to a Solr update handler, the main uses cases would be indexing json, xml, and csv, but any update handler could be used. MergeContent could be used in front of this processor to combine documents for more efficient submission to Solr. A good overview of how Solr indexes JSON is here: http://lucidworks.com/blog/indexing-custom-json-data/ and this processor could be used to post the JSON.
Both processors support a stand-alone Solr instance, as well as a SolrCloud cluster.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bbende/incubator-nifi nifi-solr-bundle
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-nifi/pull/43.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #43
----
commit 71b6ffc95885cf648c38b0ae2c8283cd849332e4
Author: bbende <bb...@gmail.com>
Date: 2015-03-22T22:59:22Z
Adding nifi-solr-bundle with processors adding and retrieving data from Apache Solr
commit 4107a0e93492037c39333ef082da22725ce500cb
Author: bbende <bb...@gmail.com>
Date: 2015-03-23T23:46:14Z
Merge remote-tracking branch 'upstream/develop' into nifi-solr-bundle
Conflicts:
nifi/nifi-assembly/pom.xml
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-nifi pull request: NIFI-461 Add processors to interact w...
Posted by madrob <gi...@git.apache.org>.
Github user madrob commented on the pull request:
https://github.com/apache/incubator-nifi/pull/43#issuecomment-94583196
First a caveat - I'm less familiar with the mechanics of the ContentStreamUpdateRequest, but it looks like this should work fine. Does this handle all of insert/update/delete, or just insert? I'm more used to seeing separate docs added like you do in the test code for the GetSolr, which is what I expected the PutSolr to look like.
+1 from me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-nifi pull request: NIFI-461 Add processors to interact w...
Posted by bbende <gi...@git.apache.org>.
Github user bbende commented on the pull request:
https://github.com/apache/incubator-nifi/pull/43#issuecomment-94598108
Added a test case that shows a delete by query using xml, ex:
<delete>
<query>...</query>
</delete>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-nifi pull request: NIFI-461 Add processors to interact w...
Posted by madrob <gi...@git.apache.org>.
Github user madrob commented on the pull request:
https://github.com/apache/incubator-nifi/pull/43#issuecomment-94591156
> I actually hadn't thought about using it for deletes, but I just did a test and it worked.
Awesome! Can you add that test to the Unit Tests?
> I was thinking there eventually might be a use case for another type of "PutSolr" processor that reads in data in a standard format (maybe avro) and converted to a SolrInputDocument, but seemed like the ContentStream approach gave the most flexibility initially.
Agreed. Just file a follow-on issue and somebody (maybe you) will get to it when the need arises.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-nifi pull request: NIFI-461 Add processors to interact w...
Posted by madrob <gi...@git.apache.org>.
Github user madrob commented on a diff in the pull request:
https://github.com/apache/incubator-nifi/pull/43#discussion_r28737517
--- Diff: nifi/nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/pom.xml ---
@@ -0,0 +1,92 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-solr-bundle</artifactId>
+ <version>0.1.0-incubating-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-solr-processors</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.solr</groupId>
+ <artifactId>solr-solrj</artifactId>
+ <version>${solr.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-processor-utils</artifactId>
+ </dependency>
+ <!-- test dependencies -->
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-simple</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>commons-logging</groupId>
+ <artifactId>commons-logging</artifactId>
+ <version>1.1.3</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <version>4.11</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.solr</groupId>
+ <artifactId>solr-core</artifactId>
+ <version>${solr.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <!-- These Lucene deps should be brought in through solr-core, but there
+ appears to be an issue with 5.0.0 that still references some 4.10.3 poms -->
--- End diff --
I would suggest filing an issue at https://issues.apache.org/jira/browse/SOLR and see what response shakes out.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-nifi pull request: NIFI-461 Add processors to interact w...
Posted by madrob <gi...@git.apache.org>.
Github user madrob commented on a diff in the pull request:
https://github.com/apache/incubator-nifi/pull/43#discussion_r28733794
--- Diff: nifi/nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/pom.xml ---
@@ -0,0 +1,92 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-solr-bundle</artifactId>
+ <version>0.1.0-incubating-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-solr-processors</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.solr</groupId>
+ <artifactId>solr-solrj</artifactId>
+ <version>${solr.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-processor-utils</artifactId>
+ </dependency>
+ <!-- test dependencies -->
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-simple</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>commons-logging</groupId>
+ <artifactId>commons-logging</artifactId>
+ <version>1.1.3</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <version>4.11</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.solr</groupId>
+ <artifactId>solr-core</artifactId>
+ <version>${solr.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <!-- These Lucene deps should be brought in through solr-core, but there
+ appears to be an issue with 5.0.0 that still references some 4.10.3 poms -->
--- End diff --
Is this still an issue with 5.1?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-nifi pull request: NIFI-461 Add processors to interact w...
Posted by bbende <gi...@git.apache.org>.
Github user bbende commented on the pull request:
https://github.com/apache/incubator-nifi/pull/43#issuecomment-94590366
Thanks for reviewing Mike.
With regards to the ContentStreamUpdateRequest... I actually hadn't thought about using it for deletes, but I just did a test and it worked. So it should work for almost any add/update/delete example in the Solr documentation. It would just require an upstream processor that was producing the right commands, where as for the case of adding new documents it seems more likely that you already have some data (json, csv, etc) that you can just fork into Solr.
I was thinking there eventually might be a use case for another type of "PutSolr" processor that reads in data in a standard format (maybe avro) and converted to a SolrInputDocument, but seemed like the ContentStream approach gave the most flexibility initially.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-nifi pull request: NIFI-461 Add processors to interact w...
Posted by joewitt <gi...@git.apache.org>.
Github user joewitt commented on the pull request:
https://github.com/apache/incubator-nifi/pull/43#issuecomment-92592767
@madrob Can you take a look?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-nifi pull request: NIFI-461 Add processors to interact w...
Posted by madrob <gi...@git.apache.org>.
Github user madrob commented on a diff in the pull request:
https://github.com/apache/incubator-nifi/pull/43#discussion_r28554788
--- Diff: nifi/nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/main/java/org/apache/nifi/processors/solr/SolrProcessor.java ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.nifi.processors.solr;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+
+/**
+ * A base class for processors that interact with Apache Solr.
+ *
+ */
+public abstract class SolrProcessor extends AbstractProcessor {
+
+ public static final AllowableValue SOLR_TYPE_CLOUD = new AllowableValue(
+ "Cloud", "Cloud", "A SolrCloud instance.");
+
+ public static final AllowableValue SOLR_TYPE_STANDARD = new AllowableValue(
+ "Standard", "Standard", "A stand-alone Solr instance.");
+
+ public static final PropertyDescriptor SOLR_TYPE = new PropertyDescriptor
+ .Builder().name("Solr Type")
+ .description("The type of Solr instance, Cloud or Standard.")
+ .required(true)
+ .allowableValues(SOLR_TYPE_CLOUD, SOLR_TYPE_STANDARD)
+ .defaultValue(SOLR_TYPE_STANDARD.getValue())
+ .build();
+
+ public static final PropertyDescriptor SOLR_LOCATION = new PropertyDescriptor
+ .Builder().name("Solr Location")
+ .description("The Solr url for a Solr Type of Standard, " +
+ "or the ZooKeeper hosts for a Solr Type of Cloud.")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor COLLECTION = new PropertyDescriptor
+ .Builder().name("Collection")
+ .description("The Solr collection name, only used with a Solr Type of Cloud")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(true)
+ .build();
+
+ private volatile SolrClient solrClient;
+
+ @OnScheduled
+ public final void onScheduled(final ProcessContext context) throws IOException {
+ this.solrClient = createSolrClient(context);
+ }
+
+ /**
+ * Create a SolrClient based on the type of Solr specified.
+ *
+ * @param context
+ * The context
+ * @return an HttpSolrClient or CloudSolrClient
+ */
+ protected SolrClient createSolrClient(final ProcessContext context) {
+ if (SOLR_TYPE_STANDARD.equals(context.getProperty(SOLR_TYPE).getValue())) {
+ return new HttpSolrClient(context.getProperty(SOLR_LOCATION).getValue());
+ } else {
+ CloudSolrClient cloudSolrServer = new CloudSolrClient(
--- End diff --
Nit: This naming is potentially confusing. Variable name could be cloudSolrClient.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-nifi pull request: NIFI-461 Add processors to interact w...
Posted by madrob <gi...@git.apache.org>.
Github user madrob commented on a diff in the pull request:
https://github.com/apache/incubator-nifi/pull/43#discussion_r28735551
--- Diff: nifi/nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/test/java/org/apache/nifi/processors/solr/TestGetSolr.java ---
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.nifi.processors.solr;
+
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrInputDocument;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+import static org.junit.Assert.assertTrue;
+
+public class TestGetSolr {
+
+ static final String DEFAULT_SOLR_CORE = "testCollection";
+
+ private SolrClient solrClient;
+
+ @Before
+ public void setup() {
+ // create the conf dir if it doesn't exist
+ File confDir = new File("conf");
+ if (!confDir.exists()) {
+ confDir.mkdir();
+ }
+
+ try {
+ // create an EmbeddedSolrServer for the processor to use
+ String relPath = getClass().getProtectionDomain().getCodeSource()
+ .getLocation().getFile() + "../../target";
+
+ solrClient = EmbeddedSolrServerFactory.create(EmbeddedSolrServerFactory.DEFAULT_SOLR_HOME,
+ EmbeddedSolrServerFactory.DEFAULT_CORE_HOME, DEFAULT_SOLR_CORE, relPath);
+
+ // create some test documents
+ SolrInputDocument doc1 = new SolrInputDocument();
+ doc1.addField("first", "bob");
+ doc1.addField("last", "smith");
+ doc1.addField("created", new Date());
+
+ SolrInputDocument doc2 = new SolrInputDocument();
+ doc2.addField("first", "alice");
+ doc2.addField("last", "smith");
+ doc2.addField("created", new Date());
+
+ SolrInputDocument doc3 = new SolrInputDocument();
+ doc3.addField("first", "mike");
+ doc3.addField("last", "smith");
+ doc3.addField("created", new Date());
+
+ SolrInputDocument doc4 = new SolrInputDocument();
+ doc4.addField("first", "john");
+ doc4.addField("last", "smith");
+ doc4.addField("created", new Date());
+
+ SolrInputDocument doc5 = new SolrInputDocument();
+ doc5.addField("first", "joan");
+ doc5.addField("last", "smith");
+ doc5.addField("created", new Date());
+
+ // add the test data to the index
+ solrClient.add(doc1);
--- End diff --
Can use a collection here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-nifi pull request: NIFI-461 Add processors to interact w...
Posted by madrob <gi...@git.apache.org>.
Github user madrob commented on a diff in the pull request:
https://github.com/apache/incubator-nifi/pull/43#discussion_r28554451
--- Diff: nifi/nifi-nar-bundles/nifi-solr-bundle/pom.xml ---
@@ -0,0 +1,39 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-nar-bundles</artifactId>
+ <version>0.1.0-incubating-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-solr-bundle</artifactId>
+ <packaging>pom</packaging>
+
+ <description>A bundle of processors that can store and retrieve data from Apache Solr</description>
+
+ <properties>
+ <solr.version>5.0.0</solr.version>
--- End diff --
5.1.0 was released this week, no reason not to use that instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-nifi pull request: NIFI-461 Add processors to interact w...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-nifi/pull/43
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-nifi pull request: NIFI-461 Add processors to interact w...
Posted by bbende <gi...@git.apache.org>.
Github user bbende commented on the pull request:
https://github.com/apache/incubator-nifi/pull/43#issuecomment-94598161
<delete>
<query>...</query>
</delete>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-nifi pull request: NIFI-461 Add processors to interact w...
Posted by madrob <gi...@git.apache.org>.
Github user madrob commented on a diff in the pull request:
https://github.com/apache/incubator-nifi/pull/43#discussion_r28554900
--- Diff: nifi/nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/main/java/org/apache/nifi/processors/solr/SolrProcessor.java ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.nifi.processors.solr;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+
+/**
+ * A base class for processors that interact with Apache Solr.
+ *
+ */
+public abstract class SolrProcessor extends AbstractProcessor {
+
+ public static final AllowableValue SOLR_TYPE_CLOUD = new AllowableValue(
+ "Cloud", "Cloud", "A SolrCloud instance.");
+
+ public static final AllowableValue SOLR_TYPE_STANDARD = new AllowableValue(
+ "Standard", "Standard", "A stand-alone Solr instance.");
+
+ public static final PropertyDescriptor SOLR_TYPE = new PropertyDescriptor
+ .Builder().name("Solr Type")
+ .description("The type of Solr instance, Cloud or Standard.")
+ .required(true)
+ .allowableValues(SOLR_TYPE_CLOUD, SOLR_TYPE_STANDARD)
+ .defaultValue(SOLR_TYPE_STANDARD.getValue())
+ .build();
+
+ public static final PropertyDescriptor SOLR_LOCATION = new PropertyDescriptor
+ .Builder().name("Solr Location")
+ .description("The Solr url for a Solr Type of Standard, " +
+ "or the ZooKeeper hosts for a Solr Type of Cloud.")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor COLLECTION = new PropertyDescriptor
+ .Builder().name("Collection")
+ .description("The Solr collection name, only used with a Solr Type of Cloud")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(true)
+ .build();
+
+ private volatile SolrClient solrClient;
+
+ @OnScheduled
+ public final void onScheduled(final ProcessContext context) throws IOException {
+ this.solrClient = createSolrClient(context);
+ }
+
+ /**
+ * Create a SolrClient based on the type of Solr specified.
+ *
+ * @param context
+ * The context
+ * @return an HttpSolrClient or CloudSolrClient
+ */
+ protected SolrClient createSolrClient(final ProcessContext context) {
+ if (SOLR_TYPE_STANDARD.equals(context.getProperty(SOLR_TYPE).getValue())) {
+ return new HttpSolrClient(context.getProperty(SOLR_LOCATION).getValue());
+ } else {
+ CloudSolrClient cloudSolrServer = new CloudSolrClient(
+ context.getProperty(SOLR_LOCATION).getValue());
+ cloudSolrServer.setDefaultCollection(
+ context.getProperty(COLLECTION).evaluateAttributeExpressions().getValue());
+ return cloudSolrServer;
+ }
+ }
+
+ /**
+ * Returns the {@link org.apache.solr.client.solrj.SolrClient} that was created by the
+ * {@link #createSolrClient(org.apache.nifi.processor.ProcessContext)} method
+ *
+ * @return
--- End diff --
Nit: empty javadoc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-nifi pull request: NIFI-461 Add processors to interact w...
Posted by bbende <gi...@git.apache.org>.
Github user bbende commented on a diff in the pull request:
https://github.com/apache/incubator-nifi/pull/43#discussion_r28736751
--- Diff: nifi/nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/pom.xml ---
@@ -0,0 +1,92 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-solr-bundle</artifactId>
+ <version>0.1.0-incubating-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-solr-processors</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.solr</groupId>
+ <artifactId>solr-solrj</artifactId>
+ <version>${solr.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-processor-utils</artifactId>
+ </dependency>
+ <!-- test dependencies -->
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-simple</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>commons-logging</groupId>
+ <artifactId>commons-logging</artifactId>
+ <version>1.1.3</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <version>4.11</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.solr</groupId>
+ <artifactId>solr-core</artifactId>
+ <version>${solr.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <!-- These Lucene deps should be brought in through solr-core, but there
+ appears to be an issue with 5.0.0 that still references some 4.10.3 poms -->
--- End diff --
It looks like it still is, here are the dependencies brought in through solr-core 5.1.0:
+- org.apache.solr:solr-core:jar:5.1.0:test
[INFO] | +- org.apache.lucene:lucene-analyzers-common:jar:4.10.3:test
[INFO] | +- org.apache.lucene:lucene-analyzers-kuromoji:jar:5.1.0:test
[INFO] | +- org.apache.lucene:lucene-analyzers-phonetic:jar:5.1.0:test
[INFO] | +- org.apache.lucene:lucene-backward-codecs:jar:5.1.0:test
[INFO] | +- org.apache.lucene:lucene-codecs:jar:5.1.0:test
[INFO] | +- org.apache.lucene:lucene-core:jar:4.10.3:test
[INFO] | +- org.apache.lucene:lucene-expressions:jar:5.1.0:test
[INFO] | +- org.apache.lucene:lucene-grouping:jar:5.1.0:test
[INFO] | +- org.apache.lucene:lucene-highlighter:jar:5.1.0:test
[INFO] | +- org.apache.lucene:lucene-join:jar:5.1.0:test
[INFO] | +- org.apache.lucene:lucene-memory:jar:5.1.0:test
[INFO] | +- org.apache.lucene:lucene-misc:jar:5.1.0:test
[INFO] | +- org.apache.lucene:lucene-queries:jar:5.1.0:test
[INFO] | +- org.apache.lucene:lucene-queryparser:jar:4.10.3:test
I looked at the solr-core 5.1.0 pom, but I don't see why that is happening, and it looks like it got pulled from Maven central:
#Thu Apr 16 20:46:02 EDT 2015
solr-core-5.1.0.jar>central=
solr-core-5.1.0.pom>central=
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---