You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by dlukyanov <gi...@git.apache.org> on 2017/04/10 21:28:24 UTC
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
GitHub user dlukyanov opened a pull request:
https://github.com/apache/nifi/pull/1662
NIFI-3688 Extended Groovy Nifi Processor
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [YES] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
in last one only
- [YES] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [YES] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [YES?] Is your initial contribution a single, squashed commit?
### For code changes:
- [YES] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
- [NO] Have you written or updated unit tests to verify your changes?
no changes done to existing code. but no tests to groovyx processor.
- [YES] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [YES?] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
- [YES?] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
- [NO] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?
not applicable?
### For documentation related changes:
- [YES] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dlukyanov/nifi master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/1662.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1662
----
commit 279e0502deef66d2a2934a211430aac52d3a5b28
Author: dlukyanov <dl...@ukr.net>
Date: 2017-04-02T07:55:22Z
groovyx initial version
commit 0bb6f9f970b186b6ec018d70e5afa68c9345b7c1
Author: dlukyanov <dl...@ukr.net>
Date: 2017-04-10T21:09:54Z
https://issues.apache.org/jira/browse/NIFI-3688
commit 8d142dd629fee4f42434f754c16c897edfa08447
Author: dlukyanov <dl...@ukr.net>
Date: 2017-04-10T21:22:38Z
NIFI-3688 license update to ASF
commit cd26cb1f3f2ebcbdc2ebda8904c424c62f2e5298
Author: dlukyanov <dl...@ukr.net>
Date: 2017-04-10T21:24:30Z
NIFI-3688 add groovyx dependency
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by dlukyanov <gi...@git.apache.org>.
Github user dlukyanov commented on the issue:
https://github.com/apache/nifi/pull/1662
In this case i believe it's ready for review.
I've reverted last commits and rebased all commits into one
Tried to build locally - and it's fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by dlukyanov <gi...@git.apache.org>.
Github user dlukyanov commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r110773203
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-nar/pom.xml ---
@@ -0,0 +1,44 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-groovyx-bundle</artifactId>
+ <version>1.2.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-groovyx-nar</artifactId>
+ <packaging>nar</packaging>
+ <properties>
+ <maven.javadoc.skip>true</maven.javadoc.skip>
--- End diff --
no reason. i'll fix it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on the issue:
https://github.com/apache/nifi/pull/1662
Don't worry about test failures in other parts of the project, they can be handled with other Jira cases and fixes. When I review your PR I build and test the relevant parts, and ensure a successful full build (whether all the tests pass or not). Feel free to revert those commits and rebase the whole thing into one, will make it easier to review/merge, please and thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by dlukyanov <gi...@git.apache.org>.
Github user dlukyanov commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r111661969
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-nar/src/main/resources/META-INF/NOTICE ---
@@ -0,0 +1,123 @@
+nifi-groovyx-nar
+Copyright 2014-2017 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+******************
+Apache Software License v2
+******************
+
+The following binary components are provided under the Apache Software License v2
+
+ (ASLv2) Apache Commons IO
+ The following NOTICE information applies:
+ Apache Commons IO
+ Copyright 2002-2016 The Apache Software Foundation
+
+ (ASLv2) Apache Commons Email
+ The following NOTICE information applies:
+ Apache Commons Email
+ Copyright 2002-2016 The Apache Software Foundation
+
+ (ASLv2) Apache Commons Lang
+ The following NOTICE information applies:
+ Apache Commons Lang
+ Copyright 2001-2015 The Apache Software Foundation
+
+ This product includes software from the Spring Framework,
+ under the Apache License 2.0 (see: StringUtils.containsWhitespace())
+
+ (ASLv2) Apache Commons Logging
+ The following NOTICE information applies:
+ Apache Commons Logging
+ Copyright 2003-2016 The Apache Software Foundation
+
+ (ASLv2) Apache HttpComponents
+ The following NOTICE information applies:
+ Apache HttpClient
+ Copyright 1999-2015 The Apache Software Foundation
+
+ Apache HttpCore
+ Copyright 2005-2015 The Apache Software Foundation
+
+ Apache HttpMime
+ Copyright 1999-2013 The Apache Software Foundation
+
+ This project contains annotations derived from JCIP-ANNOTATIONS
+ Copyright (c) 2005 Brian Goetz and Tim Peierls. See http://www.jcip.net
+
+ (ASLv2) Spring Framework
+ The following NOTICE information applies:
+ Spring Framework
+ Copyright 2002-2016
+
+ (ASLv2) SubEthaSMTP - A SMTP mail server
+ The following NOTICE information applies:
+ Spring Framework
+ Copyright 2006-2007
+
+ (ASLv2) Apache POI
+ The following NOTICE information applies:
+
+ This product contains parts that were originally based on software from BEA.
+ Copyright (c) 2000-2003, BEA Systems, <http://www.bea.com/>.
+
+ This product contains W3C XML Schema documents. Copyright 2001-2003 (c)
+ World Wide Web Consortium (Massachusetts Institute of Technology, European
+ Research Consortium for Informatics and Mathematics, Keio University)
+
+ This product contains the Piccolo XML Parser for Java
+ (http://piccolo.sourceforge.net/). Copyright 2002 Yuval Oren.
+
+ This product contains the chunks_parse_cmds.tbl file from the vsdump program.
+ Copyright (C) 2006-2007 Valek Filippov (frob@df.ru)
+
+ This product contains parts of the eID Applet project
+ (http://eid-applet.googlecode.com). Copyright (c) 2009-2014
+ FedICT (federal ICT department of Belgium), e-Contract.be BVBA (https://www.e-contract.be),
+ Bart Hanssens from FedICT
+
+ CurvesAIP is BSD-licensed software ( https://github.com/virtuald/curvesapi/)
+ Copyright (c) 2005, Graph Builder
+
+ (ASLv2) Joda Time
+ The following NOTICE information applies:
+ This product includes software developed by
+ Joda.org (http://www.joda.org/).
+
+
+************************
+Common Development and Distribution License 1.1
+************************
+
+The following binary components are provided under the Common Development and Distribution License 1.1. See project link for details.
+
+ (CDDL 1.1) (GPL2 w/ CPE) JavaMail API (compat) (javax.mail:mail:jar:1.5.6 - https://java.net/projects/javamail/pages/Home)
+
+************************
+Common Development and Distribution License 1.0
+************************
+
+The following binary components are provided under the Common Development and Distribution License 1.0. See project link for details.
+
+ (CDDL 1.0) JavaBeans Activation Framework (JAF) (javax.activation:activation:jar:1.1 - http://java.sun.com/products/javabeans/jaf/index.jsp)
+
+************************
+The MIT License
+************************
+
+The following binary components are provided under the MIT License. See project link for details.
+
+ (MIT License) EWS Java API
--- End diff --
This just copy-paste from another processor. Could you point to the right one license header?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by dlukyanov <gi...@git.apache.org>.
Github user dlukyanov commented on the issue:
https://github.com/apache/nifi/pull/1662
@mattyb149, I fixed everything except:
- Failure strategy
this property does not limit anything in error handling. it provides default algorithms for non-handled exceptions. the user still can do try-catch-transfer in code. i beleive i've added enough description of this property. ready for discussion about this point.
- rightShift
i really like it, but the main reason why i did not do it - that in similar cases in groovy only `leftShift` used.
-- http://docs.groovy-lang.org/latest/html/groovy-jdk/java/io/OutputStream.html
-- http://docs.groovy-lang.org/latest/html/groovy-jdk/java/io/Writer.html
-- http://docs.groovy-lang.org/latest/html/groovy-jdk/java/util/Collection.html
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on the issue:
https://github.com/apache/nifi/pull/1662
Sorry for the delay, I've been meaning to get back to this review. Unfortunately in the meantime the master branch has moved to version 1.4.0-SNAPSHOT, can you change your references from 1.3.0-SNAPSHOT to 1.4.0-SNAPSHOT and rebase against the latest master? Please and thank you!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by dlukyanov <gi...@git.apache.org>.
Github user dlukyanov commented on the issue:
https://github.com/apache/nifi/pull/1662
seems I fixed all checkstyle issues) but appveyor failed.. what should i do with it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r144080861
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-nar/pom.xml ---
@@ -0,0 +1,44 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-groovyx-bundle</artifactId>
+ <version>1.4.0-SNAPSHOT</version>
--- End diff --
Sorry I keep losing track of this PR :( In the meantime we have released 1.4.0, do you mind changing these references to 1.5.0-SNAPSHOT and rebasing against the latest master? Please and thanks!
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r148413690
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/java/org/apache/nifi/processors/groovyx/ExecuteGroovyScript.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.groovyx;
+
+import java.io.File;
+import java.lang.reflect.Method;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ControllerService;
+import org.apache.nifi.dbcp.DBCPService;
+import org.apache.nifi.flowfile.FlowFile;
--- End diff --
Checkstyle (via the -Pcontrib-check Maven profile) says this is unused, so it should be removed
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r124116427
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/pom.xml ---
@@ -0,0 +1,76 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-groovyx-bundle</artifactId>
+ <version>1.4.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-groovyx-processors</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-processor-utils</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.codehaus.groovy</groupId>
+ <artifactId>groovy-all</artifactId>
+ <version>2.4.11</version>
+ <!--scope>compile</scope-->
--- End diff --
This unused line can be removed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by dlukyanov <gi...@git.apache.org>.
Github user dlukyanov commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r148956693
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/resources/docs/org.apache.nifi.processors.groovyx.ExecuteGroovyScript/additionalDetails.html ---
@@ -0,0 +1,202 @@
+<!DOCTYPE html>
+<html lang="en">
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<head>
+ <meta charset="utf-8" />
+ <title>Groovy</title>
+ <!--link rel="stylesheet" href="../../css/component-usage.css" type="text/css" /-->
+ <link rel="stylesheet" href="/nifi-docs/css/component-usage.css" type="text/css" />
+</head>
+
+<body>
+<h2>Summary</h2>
+<p>This is a grooviest groovy script :)</p>
+<h2>Script Bindings:</h2>
+<table>
+<tr><th>variable</th><th>type</th><th>description</th></tr>
+<tr>
+ <td>session</td>
+ <td>org.apache.nifi.processor.ProcessSession</td>
+ <td>the session that is used to get, change, and transfer input files</td>
+</tr>
+<tr>
+ <td>context</td>
+ <td>org.apache.nifi.processor.ProcessContext</td>
+ <td>the context (almost unusefull)</td>
+</tr>
+<tr>
+ <td>log</td>
+ <td>org.apache.nifi.logging.ComponentLog</td>
+ <td>the logger for this processor instance</td>
+</tr>
+<tr>
+ <td>REL_SUCCESS</td>
+ <td>org.apache.nifi.processor.Relationship</td>
+ <td>the success relationship</td>
+</tr>
+<tr>
+ <td>REL_FAILURE</td>
+ <td>org.apache.nifi.processor.Relationship</td>
+ <td>the failure relationship</td>
+</tr>
+<tr>
+ <td>flowFile</td>
+ <td>org.apache.nifi.flowfile.FlowFile</td>
+ <td>Binded only if the property `Require flow file`=true for the processor</td>
+</tr>
+<tr>
+ <td>CTL</td>
+ <td>java.util.HashMap</td>
+ <td>Map populated with controller services binded through `CTL.*` processor properties</td>
+</tr>
+<tr>
+ <td>Dynamic processor properties</td>
+ <td>org.apache.nifi.components.PropertyDescriptor</td>
+ <td>All processor properties not started with `CTL.` are binded to script variables</td>
+</tr>
+</table>
+
+<h2>CTL map</h2>
+<p>
+CTL.* objects accessible if corresponding processor property defined.<br/>
+<b>Example:</b> if you defined property <code>`CTL.cache`</code> to DistributedMapCacheClientService, then you can access it from code <code>CTL.cache</code><br/>
+If CTL property references to Database connection pool, then corresponding CTL entry will contain groovy.sql.Sql object connected to database with autocommit=false.</br>
--- End diff --
Here I call `setAutoCommit(false)`: [ExecuteGroovyScript.java#L312](https://github.com/dlukyanov/nifi/blob/master/nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/java/org/apache/nifi/processors/groovyx/ExecuteGroovyScript.java#L312)
with hive that's interesting question. according to code [HiveConnection.setAutoCommit()](https://github.com/apache/hive/blob/master/jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java#L1338) will log warning. but the [HiveConnection.commit()](https://github.com/apache/hive/blob/master/jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java#L873) will not work in any case. What do you think? Allow `groovy.sql.Sql` to be created only for `DBCPService` ?
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r114324386
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/java/org/apache/nifi/processors/groovyx/flow/GroovySessionFile.java ---
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.groovyx.flow;
+
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.io.InputStreamCallback;
+
+import groovy.lang.Writable;
+import groovy.lang.Closure;
+import groovy.lang.MetaClass;
+import groovy.lang.GroovyObject;
+import org.codehaus.groovy.runtime.InvokerHelper;
+
+
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.Writer;
+
+/**
+ * SessionFile with groovy specific methods.
+ */
+public class GroovySessionFile extends SessionFile implements GroovyObject {
+ private transient MetaClass metaClass;
+
+ protected GroovySessionFile(ProcessSessionWrap session, FlowFile f) {
+ super(session, f);
+ setMetaClass(null); //set defult metaclass
--- End diff --
Minor typo here (default vs defult)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r124123922
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/java/org/apache/nifi/processors/groovyx/ExecuteGroovyScript.java ---
@@ -0,0 +1,468 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.groovyx;
+
+import java.io.File;
+import java.lang.reflect.Method;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ControllerService;
+import org.apache.nifi.dbcp.DBCPService;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.codehaus.groovy.control.CompilerConfiguration;
+import org.codehaus.groovy.runtime.ResourceGroovyMethods;
+import org.codehaus.groovy.runtime.StackTraceUtils;
+
+import org.apache.nifi.processors.groovyx.sql.OSql;
+import org.apache.nifi.processors.groovyx.util.Files;
+import org.apache.nifi.processors.groovyx.util.Validators;
+import org.apache.nifi.processors.groovyx.flow.GroovyProcessSessionWrap;
+
+import groovy.lang.GroovyShell;
+import groovy.lang.Script;
+
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.ValidationContext;
+
+@EventDriven
+@Tags({"script", "groovy", "groovyx"})
+@CapabilityDescription(
+ "Experimental Extended Groovy script processor. The script is responsible for "
+ + "handling the incoming flow file (transfer to SUCCESS or remove, e.g.) as well as any flow files created by "
+ + "the script. If the handling is incomplete or incorrect, the session will be rolled back.")
+@Restricted("Provides operator the ability to execute arbitrary code assuming all permissions that NiFi has.")
+@SeeAlso({})
+@DynamicProperty(name = "A script engine property to update",
+ value = "The value to set it to",
+ supportsExpressionLanguage = true,
+ description = "Updates a script engine property specified by the Dynamic Property's key with the value "
+ + "specified by the Dynamic Property's value. Use `CTL.` to access any controller services.")
+public class ExecuteGroovyScript extends AbstractProcessor {
+ public static final String GROOVY_CLASSPATH = "${groovy.classes.path}";
+
+ private static final String PRELOADS = "import org.apache.nifi.components.*;" + "import org.apache.nifi.flowfile.FlowFile;" + "import org.apache.nifi.processor.*;"
+ + "import org.apache.nifi.processor.FlowFileFilter.FlowFileFilterResult;" + "import org.apache.nifi.processor.exception.*;" + "import org.apache.nifi.processor.io.*;"
+ + "import org.apache.nifi.processor.util.*;" + "import org.apache.nifi.processors.script.*;" + "import org.apache.nifi.logging.ComponentLog;";
+
+ public static final PropertyDescriptor SCRIPT_FILE = new PropertyDescriptor.Builder().name("Script File").required(false)
+ .description("Path to script file to execute. Only one of Script File or Script Body may be used").addValidator(Validators.createFileExistsAndReadableValidator())
+ .expressionLanguageSupported(true).build();
+
+ public static final PropertyDescriptor SCRIPT_BODY = new PropertyDescriptor.Builder().name("Script Body").required(false)
+ .description("Body of script to execute. Only one of Script File or Script Body may be used").addValidator(StandardValidators.NON_EMPTY_VALIDATOR).expressionLanguageSupported(false)
+ .build();
+
+ public static String[] VALID_BOOLEANS = {"true", "false"};
+ public static final PropertyDescriptor REQUIRE_FLOW = new PropertyDescriptor.Builder().name("Requires flow file")
--- End diff --
I see the benefit of saving the user the trouble of the following boilerplate:
```
def flowFile = session.get()
if(!flowFile) return
```
However there are other methods on ProcessSession that might be helpful to the user, for example getting a batch of flow files (if available):
```
def flowFiles = session.get(100)
if(!flowFiles) return
flowFiles.each { flowFile ->
// do something
}
```
or applying a filter (I see GroovyProcessSessionWrap has a handy get(Closure) override but it is unused). For the small amount of boilerplate code, I'm not sure such a property is necessary. Thoughts?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r110773031
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-nar/pom.xml ---
@@ -0,0 +1,44 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-groovyx-bundle</artifactId>
+ <version>1.2.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-groovyx-nar</artifactId>
+ <packaging>nar</packaging>
+ <properties>
+ <maven.javadoc.skip>true</maven.javadoc.skip>
--- End diff --
I don't think this is compatible with our coding style guide. Is there a reason Javadoc needs to be skipped?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r148411913
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/resources/docs/org.apache.nifi.processors.groovyx.ExecuteGroovyScript/additionalDetails.html ---
@@ -0,0 +1,202 @@
+<!DOCTYPE html>
+<html lang="en">
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<head>
+ <meta charset="utf-8" />
+ <title>Groovy</title>
+ <!--link rel="stylesheet" href="../../css/component-usage.css" type="text/css" /-->
+ <link rel="stylesheet" href="/nifi-docs/css/component-usage.css" type="text/css" />
+</head>
+
+<body>
+<h2>Summary</h2>
+<p>This is a grooviest groovy script :)</p>
+<h2>Script Bindings:</h2>
+<table>
+<tr><th>variable</th><th>type</th><th>description</th></tr>
+<tr>
+ <td>session</td>
+ <td>org.apache.nifi.processor.ProcessSession</td>
+ <td>the session that is used to get, change, and transfer input files</td>
+</tr>
+<tr>
+ <td>context</td>
+ <td>org.apache.nifi.processor.ProcessContext</td>
+ <td>the context (almost unusefull)</td>
+</tr>
+<tr>
+ <td>log</td>
+ <td>org.apache.nifi.logging.ComponentLog</td>
+ <td>the logger for this processor instance</td>
+</tr>
+<tr>
+ <td>REL_SUCCESS</td>
+ <td>org.apache.nifi.processor.Relationship</td>
+ <td>the success relationship</td>
+</tr>
+<tr>
+ <td>REL_FAILURE</td>
+ <td>org.apache.nifi.processor.Relationship</td>
+ <td>the failure relationship</td>
+</tr>
+<tr>
+ <td>flowFile</td>
+ <td>org.apache.nifi.flowfile.FlowFile</td>
+ <td>Binded only if the property `Require flow file`=true for the processor</td>
+</tr>
+<tr>
+ <td>CTL</td>
+ <td>java.util.HashMap</td>
+ <td>Map populated with controller services binded through `CTL.*` processor properties</td>
+</tr>
+<tr>
+ <td>Dynamic processor properties</td>
+ <td>org.apache.nifi.components.PropertyDescriptor</td>
+ <td>All processor properties not started with `CTL.` are binded to script variables</td>
+</tr>
+</table>
+
+<h2>CTL map</h2>
+<p>
+CTL.* objects accessible if corresponding processor property defined.<br/>
+<b>Example:</b> if you defined property <code>`CTL.cache`</code> to DistributedMapCacheClientService, then you can access it from code <code>CTL.cache</code><br/>
+If CTL property references to Database connection pool, then corresponding CTL entry will contain groovy.sql.Sql object connected to database with autocommit=false.</br>
--- End diff --
How is it guaranteed that autocommit will be false? If setAutoCommit is called somewhere (either by your code -- which I couldn't find a reference to -- or by groovy's Sql class), it can cause problems in a couple of scenarios, one being Oracle if the DBA has disallowed changing auto-commit, and another is Hive (since HiveConnectionPool extends DBCPService, it should be available via CTL right?)
---
[GitHub] nifi issue #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by dlukyanov <gi...@git.apache.org>.
Github user dlukyanov commented on the issue:
https://github.com/apache/nifi/pull/1662
@joewitt , about naming: CallGroovy would be fine?
Actually it's another implementation of groovy script.
And the Script processor is just a Script - and not CallScript.
I wanted to point that it's extended groovy script.. What do you think about the name?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by dlukyanov <gi...@git.apache.org>.
Github user dlukyanov commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r124206087
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/java/org/apache/nifi/processors/groovyx/GroovyMethods.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.groovyx;
+
+import groovy.lang.DelegatingMetaClass;
+import groovy.lang.GroovySystem;
+
+import org.apache.nifi.processors.groovyx.flow.ProcessSessionWrap;
+import org.apache.nifi.processors.groovyx.flow.SessionFile;
+
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.flowfile.FlowFile;
+
+import java.util.Collection;
+import java.util.List;
+
+/**
+ * Class to initialize additional groovy methods to work with SessionFile, Relationship, and Sessions easier
+ */
+class GroovyMethods {
+ private static boolean initialized = false;
+
+ static void init() {
+ if (!initialized) {
+ synchronized (GroovyMethods.class) {
+ if (!initialized) {
+ initialized = metaRelationship();
+ }
+ }
+ }
+ }
+
+ private static boolean metaRelationship() {
+ GroovySystem.getMetaClassRegistry().setMetaClass(Relationship.class, new DelegatingMetaClass(Relationship.class) {
+ @Override
+ public Object invokeMethod(Object object, String methodName, Object[] args) {
+ if (object instanceof Relationship) {
+ if ("leftShift".equals(methodName) && args.length == 1) {
+ if (args[0] instanceof SessionFile) {
+ return this.leftShift((Relationship) object, (SessionFile) args[0]);
+ } else if (args[0] instanceof Collection) {
+ return this.leftShift((Relationship) object, (Collection) args[0]);
+ }
+ }
+ }
+ return super.invokeMethod(object, methodName, args);
+ }
+
+ /** to support: REL_SUCCESS << sessionFile */
+ private Relationship leftShift(Relationship r, SessionFile f) {
--- End diff --
btw. leftshift allows syntax:
`REL_SUCCESS << a << b << c`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by dlukyanov <gi...@git.apache.org>.
Github user dlukyanov commented on the issue:
https://github.com/apache/nifi/pull/1662
mattyb149, rebased against the latest master 1.5.0-SNAPSHOT
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r114323960
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/java/org/apache/nifi/processors/groovyx/ExecuteGroovyScript.java ---
@@ -0,0 +1,461 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.groovyx;
+
+import java.io.File;
+import java.lang.reflect.Method;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ControllerService;
+import org.apache.nifi.dbcp.DBCPService;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.codehaus.groovy.control.CompilerConfiguration;
+import org.codehaus.groovy.runtime.ResourceGroovyMethods;
+import org.codehaus.groovy.runtime.StackTraceUtils;
+
+import org.apache.nifi.processors.groovyx.sql.OSql;
+import org.apache.nifi.processors.groovyx.util.Files;
+import org.apache.nifi.processors.groovyx.util.Validators;
+import org.apache.nifi.processors.groovyx.flow.GroovyProcessSessionWrap;
+
+import groovy.lang.GroovyShell;
+import groovy.lang.Script;
+import groovy.sql.Sql;
+
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.ValidationContext;
+
+@EventDriven
+@Tags({"script", "groovy", "groovyx", "extended"})
--- End diff --
I'm not sure that users will search "extended" looking for this processor, perhaps this tag is not needed?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by dlukyanov <gi...@git.apache.org>.
Github user dlukyanov commented on the issue:
https://github.com/apache/nifi/pull/1662
Can somebody help me?
Avro tests fails all the time and I did not touch it.
Failed tests:
TestAvroRecordReader.testLogicalTypes:109 expected:<2017-04-0[4]> but was:<2017-04-0[5]>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on the issue:
https://github.com/apache/nifi/pull/1662
+1 LGTM, thanks for sticking with this, it's a great addition! I made one final commit for some CheckStyle and typo stuff (and added displayName() to the properties), then rebased into one commit with you as the author. Thanks again for this feature! Merging to master
---
[GitHub] nifi issue #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on the issue:
https://github.com/apache/nifi/pull/1662
Also I ran a full build with unit tests, and tried various scripts and features to exercise the processor.
---
[GitHub] nifi issue #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on the issue:
https://github.com/apache/nifi/pull/1662
Appveyor always fails, no worries. I will take a look at your latest
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on the issue:
https://github.com/apache/nifi/pull/1662
Also there are still 22 checkstyle violations (mostly lines too long), you can see which files are affected by running "mvn clean install -Pcontrib-check" from your nifi-groovyx-bundle directory.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on the issue:
https://github.com/apache/nifi/pull/1662
This NAR won't be added to the NiFi distribution unless you add an entry for it in nifi-assembly/pom.xml, and set its version in the root level pom.xml (see those files for many examples).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by joewitt <gi...@git.apache.org>.
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r110994314
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-nar/src/main/resources/META-INF/NOTICE ---
@@ -0,0 +1,123 @@
+nifi-groovyx-nar
+Copyright 2014-2017 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+******************
+Apache Software License v2
+******************
+
+The following binary components are provided under the Apache Software License v2
+
+ (ASLv2) Apache Commons IO
+ The following NOTICE information applies:
+ Apache Commons IO
+ Copyright 2002-2016 The Apache Software Foundation
+
+ (ASLv2) Apache Commons Email
+ The following NOTICE information applies:
+ Apache Commons Email
+ Copyright 2002-2016 The Apache Software Foundation
+
+ (ASLv2) Apache Commons Lang
+ The following NOTICE information applies:
+ Apache Commons Lang
+ Copyright 2001-2015 The Apache Software Foundation
+
+ This product includes software from the Spring Framework,
+ under the Apache License 2.0 (see: StringUtils.containsWhitespace())
+
+ (ASLv2) Apache Commons Logging
+ The following NOTICE information applies:
+ Apache Commons Logging
+ Copyright 2003-2016 The Apache Software Foundation
+
+ (ASLv2) Apache HttpComponents
+ The following NOTICE information applies:
+ Apache HttpClient
+ Copyright 1999-2015 The Apache Software Foundation
+
+ Apache HttpCore
+ Copyright 2005-2015 The Apache Software Foundation
+
+ Apache HttpMime
+ Copyright 1999-2013 The Apache Software Foundation
+
+ This project contains annotations derived from JCIP-ANNOTATIONS
+ Copyright (c) 2005 Brian Goetz and Tim Peierls. See http://www.jcip.net
+
+ (ASLv2) Spring Framework
+ The following NOTICE information applies:
+ Spring Framework
+ Copyright 2002-2016
+
+ (ASLv2) SubEthaSMTP - A SMTP mail server
+ The following NOTICE information applies:
+ Spring Framework
+ Copyright 2006-2007
+
+ (ASLv2) Apache POI
+ The following NOTICE information applies:
+
+ This product contains parts that were originally based on software from BEA.
+ Copyright (c) 2000-2003, BEA Systems, <http://www.bea.com/>.
+
+ This product contains W3C XML Schema documents. Copyright 2001-2003 (c)
+ World Wide Web Consortium (Massachusetts Institute of Technology, European
+ Research Consortium for Informatics and Mathematics, Keio University)
+
+ This product contains the Piccolo XML Parser for Java
+ (http://piccolo.sourceforge.net/). Copyright 2002 Yuval Oren.
+
+ This product contains the chunks_parse_cmds.tbl file from the vsdump program.
+ Copyright (C) 2006-2007 Valek Filippov (frob@df.ru)
+
+ This product contains parts of the eID Applet project
+ (http://eid-applet.googlecode.com). Copyright (c) 2009-2014
+ FedICT (federal ICT department of Belgium), e-Contract.be BVBA (https://www.e-contract.be),
+ Bart Hanssens from FedICT
+
+ CurvesAIP is BSD-licensed software ( https://github.com/virtuald/curvesapi/)
+ Copyright (c) 2005, Graph Builder
+
+ (ASLv2) Joda Time
+ The following NOTICE information applies:
+ This product includes software developed by
+ Joda.org (http://www.joda.org/).
+
+
+************************
+Common Development and Distribution License 1.1
+************************
+
+The following binary components are provided under the Common Development and Distribution License 1.1. See project link for details.
+
+ (CDDL 1.1) (GPL2 w/ CPE) JavaMail API (compat) (javax.mail:mail:jar:1.5.6 - https://java.net/projects/javamail/pages/Home)
+
+************************
+Common Development and Distribution License 1.0
+************************
+
+The following binary components are provided under the Common Development and Distribution License 1.0. See project link for details.
+
+ (CDDL 1.0) JavaBeans Activation Framework (JAF) (javax.activation:activation:jar:1.1 - http://java.sun.com/products/javabeans/jaf/index.jsp)
+
+************************
+The MIT License
+************************
+
+The following binary components are provided under the MIT License. See project link for details.
+
+ (MIT License) EWS Java API
--- End diff --
You don't need to reference CategoryA deps in the NOTICE which are already covered in the LICENSE. This MIT dep and its copyright appears in the LICENSE already it seems so you can remove this section.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r124124963
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/java/org/apache/nifi/processors/groovyx/ExecuteGroovyScript.java ---
@@ -0,0 +1,468 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.groovyx;
+
+import java.io.File;
+import java.lang.reflect.Method;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ControllerService;
+import org.apache.nifi.dbcp.DBCPService;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.codehaus.groovy.control.CompilerConfiguration;
+import org.codehaus.groovy.runtime.ResourceGroovyMethods;
+import org.codehaus.groovy.runtime.StackTraceUtils;
+
+import org.apache.nifi.processors.groovyx.sql.OSql;
+import org.apache.nifi.processors.groovyx.util.Files;
+import org.apache.nifi.processors.groovyx.util.Validators;
+import org.apache.nifi.processors.groovyx.flow.GroovyProcessSessionWrap;
+
+import groovy.lang.GroovyShell;
+import groovy.lang.Script;
+
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.ValidationContext;
+
+@EventDriven
+@Tags({"script", "groovy", "groovyx"})
+@CapabilityDescription(
+ "Experimental Extended Groovy script processor. The script is responsible for "
+ + "handling the incoming flow file (transfer to SUCCESS or remove, e.g.) as well as any flow files created by "
+ + "the script. If the handling is incomplete or incorrect, the session will be rolled back.")
+@Restricted("Provides operator the ability to execute arbitrary code assuming all permissions that NiFi has.")
+@SeeAlso({})
+@DynamicProperty(name = "A script engine property to update",
+ value = "The value to set it to",
+ supportsExpressionLanguage = true,
+ description = "Updates a script engine property specified by the Dynamic Property's key with the value "
+ + "specified by the Dynamic Property's value. Use `CTL.` to access any controller services.")
+public class ExecuteGroovyScript extends AbstractProcessor {
+ public static final String GROOVY_CLASSPATH = "${groovy.classes.path}";
+
+ private static final String PRELOADS = "import org.apache.nifi.components.*;" + "import org.apache.nifi.flowfile.FlowFile;" + "import org.apache.nifi.processor.*;"
+ + "import org.apache.nifi.processor.FlowFileFilter.FlowFileFilterResult;" + "import org.apache.nifi.processor.exception.*;" + "import org.apache.nifi.processor.io.*;"
+ + "import org.apache.nifi.processor.util.*;" + "import org.apache.nifi.processors.script.*;" + "import org.apache.nifi.logging.ComponentLog;";
+
+ public static final PropertyDescriptor SCRIPT_FILE = new PropertyDescriptor.Builder().name("Script File").required(false)
+ .description("Path to script file to execute. Only one of Script File or Script Body may be used").addValidator(Validators.createFileExistsAndReadableValidator())
+ .expressionLanguageSupported(true).build();
+
+ public static final PropertyDescriptor SCRIPT_BODY = new PropertyDescriptor.Builder().name("Script Body").required(false)
+ .description("Body of script to execute. Only one of Script File or Script Body may be used").addValidator(StandardValidators.NON_EMPTY_VALIDATOR).expressionLanguageSupported(false)
+ .build();
+
+ public static String[] VALID_BOOLEANS = {"true", "false"};
+ public static final PropertyDescriptor REQUIRE_FLOW = new PropertyDescriptor.Builder().name("Requires flow file")
+ .description("If `true` then flowFile variable initialized and validated. So developer don't need to do flowFile = session.get(). If `false` the flowFile variable not initialized.")
+ .required(true).expressionLanguageSupported(false).allowableValues(VALID_BOOLEANS).defaultValue("false").build();
+
+ public static String[] VALID_FAIL_STRATEGY = {"rollback", "transfer to failure"};
+ public static final PropertyDescriptor FAIL_STRATEGY = new PropertyDescriptor.Builder()
--- End diff --
This also seems to try to automate "normal" handling of flow files, but removes a little flexibility. Inside the script, the user has the ability to roll back (via a thrown exception) or fail when desired. If the idea is to allow the flow files to be transferred to failure in the event of an exception in the script itself (rather than one caused by bad input, for example), then it might be confusing for the user to know why the flow file was transferred to failure (i.e. is the data bad or is the script bad). Maybe I'm just misunderstanding the intent, could you clarify for me?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/1662
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r148412458
--- Diff: nifi-nar-bundles/pom.xml ---
@@ -94,7 +95,7 @@
<build>
<plugins>
<!-- Makes buildRevision and buildBranch available to the NAR Plugin so they can be populated in the MANIFEST -->
- <plugin>
+ <!--plugin>
--- End diff --
If this is not needed it should be removed
---
[GitHub] nifi issue #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by dlukyanov <gi...@git.apache.org>.
Github user dlukyanov commented on the issue:
https://github.com/apache/nifi/pull/1662
@mattyb149 , @joewitt , @alopresto ,
I know that 1.2.0. released
This pull request was not ready (
I need to know your opinion about this pull request - to continue or not.
In any case I fixed documentation and nar bundle content.
You can try all the cases from pull request description.
I built groovyx nar bundle with nifi 1.2.0
https://github.com/dlukyanov/nifi/releases/tag/nifi-groovyx-nar-1.2.0
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r116502829
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-nar/pom.xml ---
@@ -0,0 +1,44 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-groovyx-bundle</artifactId>
+ <version>1.2.0-SNAPSHOT</version>
--- End diff --
Now that NiFi 1.2.0 has been released, the version of the current master branch is 1.3.0-SNAPSHOT. Do you mind rebasing your PR against master, and updating the version of this NAR to 1.3.0-SNAPSHOT wherever necessary? Please and thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1662: NIFI-3688 Extended Groovy Nifi Processor [WIP]
Posted by dlukyanov <gi...@git.apache.org>.
Github user dlukyanov commented on the issue:
https://github.com/apache/nifi/pull/1662
@mattyb149, I renamed the processor, created test cases for all mentioned in pull request description.
The case [test_sql_01_select.groovy](https://github.com/dlukyanov/nifi/blob/master/nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/test/resources/groovy/test_sql_01_select.groovy) dedicated to a new way of controller services usage from groovy script.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r114324101
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/java/org/apache/nifi/processors/groovyx/ExecuteGroovyScript.java ---
@@ -0,0 +1,461 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.groovyx;
+
+import java.io.File;
+import java.lang.reflect.Method;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ControllerService;
+import org.apache.nifi.dbcp.DBCPService;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.codehaus.groovy.control.CompilerConfiguration;
+import org.codehaus.groovy.runtime.ResourceGroovyMethods;
+import org.codehaus.groovy.runtime.StackTraceUtils;
+
+import org.apache.nifi.processors.groovyx.sql.OSql;
+import org.apache.nifi.processors.groovyx.util.Files;
+import org.apache.nifi.processors.groovyx.util.Validators;
+import org.apache.nifi.processors.groovyx.flow.GroovyProcessSessionWrap;
+
+import groovy.lang.GroovyShell;
+import groovy.lang.Script;
+import groovy.sql.Sql;
+
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.ValidationContext;
+
+@EventDriven
+@Tags({"script", "groovy", "groovyx", "extended"})
+@CapabilityDescription(
+ "Experimental Extended Groovy script processor. The script is responsible for " + "handling the incoming flow file (transfer to SUCCESS or remove, e.g.) as well as any flow files created by "
+ + "the script. If the handling is incomplete or incorrect, the session will be rolled back.")
+@SeeAlso({})
--- End diff --
This is not needed, recommend removal
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r124113688
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-nar/pom.xml ---
@@ -0,0 +1,44 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-groovyx-bundle</artifactId>
+ <version>1.2.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-groovyx-nar</artifactId>
+ <packaging>nar</packaging>
+ <properties>
+ <maven.javadoc.skip>true</maven.javadoc.skip>
--- End diff --
I think for the NAR POM this is fine, there should be no Javadoc generated for the NAR itself. This is a pretty standard pattern for NARs, I'm guessing you got it from the scripting NAR's POM :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r124126084
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/java/org/apache/nifi/processors/groovyx/GroovyMethods.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.groovyx;
+
+import groovy.lang.DelegatingMetaClass;
+import groovy.lang.GroovySystem;
+
+import org.apache.nifi.processors.groovyx.flow.ProcessSessionWrap;
+import org.apache.nifi.processors.groovyx.flow.SessionFile;
+
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.flowfile.FlowFile;
+
+import java.util.Collection;
+import java.util.List;
+
+/**
+ * Class to initialize additional groovy methods to work with SessionFile, Relationship, and Sessions easier
+ */
+class GroovyMethods {
+ private static boolean initialized = false;
+
+ static void init() {
+ if (!initialized) {
+ synchronized (GroovyMethods.class) {
+ if (!initialized) {
+ initialized = metaRelationship();
+ }
+ }
+ }
+ }
+
+ private static boolean metaRelationship() {
+ GroovySystem.getMetaClassRegistry().setMetaClass(Relationship.class, new DelegatingMetaClass(Relationship.class) {
+ @Override
+ public Object invokeMethod(Object object, String methodName, Object[] args) {
+ if (object instanceof Relationship) {
+ if ("leftShift".equals(methodName) && args.length == 1) {
+ if (args[0] instanceof SessionFile) {
+ return this.leftShift((Relationship) object, (SessionFile) args[0]);
+ } else if (args[0] instanceof Collection) {
+ return this.leftShift((Relationship) object, (Collection) args[0]);
+ }
+ }
+ }
+ return super.invokeMethod(object, methodName, args);
+ }
+
+ /** to support: REL_SUCCESS << sessionFile */
+ private Relationship leftShift(Relationship r, SessionFile f) {
--- End diff --
These are great! Should we support right-shift too?
`sessionFile >> REL_SUCCESS`
If having both is too confusing, then this one will work just fine!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/1662
@dlukyanov That is not your fault -- @mattyb149 is fixing it in [NIFI-3718](https://issues.apache.org/jira/browse/NIFI-3718).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by dlukyanov <gi...@git.apache.org>.
Github user dlukyanov commented on the issue:
https://github.com/apache/nifi/pull/1662
mattyb149, i just refactored the the `CTL` properties:
- 'CTL' provides direct access to controller services without any additional logic
- a new map named `SQL` provides fast to transactional DBCP service
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by joewitt <gi...@git.apache.org>.
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r110994393
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/java/org/apache/nifi/processors/groovyx/GroovyMethods.java ---
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.groovyx;
+
+import groovy.lang.Closure;
+import groovy.lang.DelegatingMetaClass;
+import groovy.lang.GroovySystem;
+import groovy.lang.MetaClass;
+import groovy.lang.Writable;
+
+import org.apache.nifi.processors.groovyx.flow.ProcessSessionWrap;
+import org.apache.nifi.processors.groovyx.flow.SessionFile;
+
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.FlowFileFilter;
+import org.apache.nifi.processor.FlowFileFilter.FlowFileFilterResult;
+//import org.apache.nifi.processor.ProcessSession;
--- End diff --
please remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by dlukyanov <gi...@git.apache.org>.
Github user dlukyanov commented on the issue:
https://github.com/apache/nifi/pull/1662
@mattyb149, do you have any comments to my last commit?
---
[GitHub] nifi issue #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by dlukyanov <gi...@git.apache.org>.
Github user dlukyanov commented on the issue:
https://github.com/apache/nifi/pull/1662
I've made last several commits in StandardFlowSerializerTest.java
trying to detect the problem java.nio.channels.OverlappingFileLockException in org.apache.nifi.controller.scheduling.TestProcessorLifecycle
https://issues.apache.org/jira/browse/NIFI-3853?focusedCommentId=16015655&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16015655
and succeed with build.
normally should revert this change in my fork...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by dlukyanov <gi...@git.apache.org>.
Github user dlukyanov commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r124203242
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/java/org/apache/nifi/processors/groovyx/ExecuteGroovyScript.java ---
@@ -0,0 +1,468 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.groovyx;
+
+import java.io.File;
+import java.lang.reflect.Method;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ControllerService;
+import org.apache.nifi.dbcp.DBCPService;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.codehaus.groovy.control.CompilerConfiguration;
+import org.codehaus.groovy.runtime.ResourceGroovyMethods;
+import org.codehaus.groovy.runtime.StackTraceUtils;
+
+import org.apache.nifi.processors.groovyx.sql.OSql;
+import org.apache.nifi.processors.groovyx.util.Files;
+import org.apache.nifi.processors.groovyx.util.Validators;
+import org.apache.nifi.processors.groovyx.flow.GroovyProcessSessionWrap;
+
+import groovy.lang.GroovyShell;
+import groovy.lang.Script;
+
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.ValidationContext;
+
+@EventDriven
+@Tags({"script", "groovy", "groovyx"})
+@CapabilityDescription(
+ "Experimental Extended Groovy script processor. The script is responsible for "
+ + "handling the incoming flow file (transfer to SUCCESS or remove, e.g.) as well as any flow files created by "
+ + "the script. If the handling is incomplete or incorrect, the session will be rolled back.")
+@Restricted("Provides operator the ability to execute arbitrary code assuming all permissions that NiFi has.")
+@SeeAlso({})
+@DynamicProperty(name = "A script engine property to update",
+ value = "The value to set it to",
+ supportsExpressionLanguage = true,
+ description = "Updates a script engine property specified by the Dynamic Property's key with the value "
+ + "specified by the Dynamic Property's value. Use `CTL.` to access any controller services.")
+public class ExecuteGroovyScript extends AbstractProcessor {
+ public static final String GROOVY_CLASSPATH = "${groovy.classes.path}";
+
+ private static final String PRELOADS = "import org.apache.nifi.components.*;" + "import org.apache.nifi.flowfile.FlowFile;" + "import org.apache.nifi.processor.*;"
+ + "import org.apache.nifi.processor.FlowFileFilter.FlowFileFilterResult;" + "import org.apache.nifi.processor.exception.*;" + "import org.apache.nifi.processor.io.*;"
+ + "import org.apache.nifi.processor.util.*;" + "import org.apache.nifi.processors.script.*;" + "import org.apache.nifi.logging.ComponentLog;";
+
+ public static final PropertyDescriptor SCRIPT_FILE = new PropertyDescriptor.Builder().name("Script File").required(false)
+ .description("Path to script file to execute. Only one of Script File or Script Body may be used").addValidator(Validators.createFileExistsAndReadableValidator())
+ .expressionLanguageSupported(true).build();
+
+ public static final PropertyDescriptor SCRIPT_BODY = new PropertyDescriptor.Builder().name("Script Body").required(false)
+ .description("Body of script to execute. Only one of Script File or Script Body may be used").addValidator(StandardValidators.NON_EMPTY_VALIDATOR).expressionLanguageSupported(false)
+ .build();
+
+ public static String[] VALID_BOOLEANS = {"true", "false"};
+ public static final PropertyDescriptor REQUIRE_FLOW = new PropertyDescriptor.Builder().name("Requires flow file")
--- End diff --
with `require flow file = false` you can do any session.get(...) in code
i agree it's not a big win to use `require flow file = true`.
i thought to add more values like `batch` and use `batch_size` custom property.
a short entry about `require flow file` and `fail strategy`:
we were migrating an 10years old j2ee project to nifi. 100k+ producers and 100k consumers of the files with different formats (xml,csv,json,pdf,tiff,...). you can imagine a huge amount of specific code, libraries, ... to cover all specific things we used groovy script. and in 95% we used just `session.get(0)` (require flow file)
and in 80% `rollback` strategy on exception is ok (btw it's a standard behavior of existing execute script processor). without try-catch the code just simpler. in case when using `rollback` you can do any custom try-catch in your code. and we used custom try-catch in 1% of cases.
transfer to `failure` in code usually becomes a nightmare in script especially if you want to transfer the original file - and in our case it was always original.
so, both features are not removing customization ability.
about the `require flowfile` - i agree that it could be removed.
but about `fail strategy` - i believe it's very useful. maybe the items in select should be renamed and more clarified... but i'd suggest to keep it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r114323810
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/pom.xml ---
@@ -0,0 +1,74 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-groovyx-bundle</artifactId>
+ <version>1.2.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-groovyx-processors</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-processor-utils</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.codehaus.groovy</groupId>
+ <artifactId>groovy-all</artifactId>
+ <version>2.4.7</version>
--- End diff --
I believe 2.4.11 has been released, it is not required that you upgrade it here, just mentioning it :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by dlukyanov <gi...@git.apache.org>.
Github user dlukyanov commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r124205062
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/java/org/apache/nifi/processors/groovyx/ExecuteGroovyScript.java ---
@@ -0,0 +1,468 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.groovyx;
+
+import java.io.File;
+import java.lang.reflect.Method;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ControllerService;
+import org.apache.nifi.dbcp.DBCPService;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.codehaus.groovy.control.CompilerConfiguration;
+import org.codehaus.groovy.runtime.ResourceGroovyMethods;
+import org.codehaus.groovy.runtime.StackTraceUtils;
+
+import org.apache.nifi.processors.groovyx.sql.OSql;
+import org.apache.nifi.processors.groovyx.util.Files;
+import org.apache.nifi.processors.groovyx.util.Validators;
+import org.apache.nifi.processors.groovyx.flow.GroovyProcessSessionWrap;
+
+import groovy.lang.GroovyShell;
+import groovy.lang.Script;
+
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.ValidationContext;
+
+@EventDriven
+@Tags({"script", "groovy", "groovyx"})
+@CapabilityDescription(
+ "Experimental Extended Groovy script processor. The script is responsible for "
+ + "handling the incoming flow file (transfer to SUCCESS or remove, e.g.) as well as any flow files created by "
+ + "the script. If the handling is incomplete or incorrect, the session will be rolled back.")
+@Restricted("Provides operator the ability to execute arbitrary code assuming all permissions that NiFi has.")
+@SeeAlso({})
--- End diff --
ok. it's just a copy-paste from somewhere)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by dlukyanov <gi...@git.apache.org>.
Github user dlukyanov commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r116540624
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-nar/pom.xml ---
@@ -0,0 +1,44 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-groovyx-bundle</artifactId>
+ <version>1.2.0-SNAPSHOT</version>
--- End diff --
Great! I'll do it this weekend.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r124116266
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-nar/src/main/resources/META-INF/LICENSE ---
@@ -0,0 +1,279 @@
+
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+
+
+
+ nifi-email-bundle includes subcomponents with separate copyright notices and
--- End diff --
The bundles referred to here and below are not included with this NAR and should be removed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by dlukyanov <gi...@git.apache.org>.
Github user dlukyanov commented on the issue:
https://github.com/apache/nifi/pull/1662
@mattyb149 , done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r124117117
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/java/org/apache/nifi/processors/groovyx/ExecuteGroovyScript.java ---
@@ -0,0 +1,468 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.groovyx;
+
+import java.io.File;
+import java.lang.reflect.Method;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ControllerService;
+import org.apache.nifi.dbcp.DBCPService;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.codehaus.groovy.control.CompilerConfiguration;
+import org.codehaus.groovy.runtime.ResourceGroovyMethods;
+import org.codehaus.groovy.runtime.StackTraceUtils;
+
+import org.apache.nifi.processors.groovyx.sql.OSql;
+import org.apache.nifi.processors.groovyx.util.Files;
+import org.apache.nifi.processors.groovyx.util.Validators;
+import org.apache.nifi.processors.groovyx.flow.GroovyProcessSessionWrap;
+
+import groovy.lang.GroovyShell;
+import groovy.lang.Script;
+
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.ValidationContext;
+
+@EventDriven
+@Tags({"script", "groovy", "groovyx"})
+@CapabilityDescription(
+ "Experimental Extended Groovy script processor. The script is responsible for "
+ + "handling the incoming flow file (transfer to SUCCESS or remove, e.g.) as well as any flow files created by "
+ + "the script. If the handling is incomplete or incorrect, the session will be rolled back.")
+@Restricted("Provides operator the ability to execute arbitrary code assuming all permissions that NiFi has.")
+@SeeAlso({})
--- End diff --
If you are not referring to other processors/classes, you can remove the SeeAlso annotation. Alternatively if you want to refer to a processor in another NAR (such as ExecuteScript), you could do the following:
`@SeeAlso({classNames="org.apache.nifi.processors.script.ExecuteScript"})`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r149756860
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/resources/docs/org.apache.nifi.processors.groovyx.ExecuteGroovyScript/additionalDetails.html ---
@@ -0,0 +1,202 @@
+<!DOCTYPE html>
+<html lang="en">
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<head>
+ <meta charset="utf-8" />
+ <title>Groovy</title>
+ <!--link rel="stylesheet" href="../../css/component-usage.css" type="text/css" /-->
+ <link rel="stylesheet" href="/nifi-docs/css/component-usage.css" type="text/css" />
+</head>
+
+<body>
+<h2>Summary</h2>
+<p>This is a grooviest groovy script :)</p>
+<h2>Script Bindings:</h2>
+<table>
+<tr><th>variable</th><th>type</th><th>description</th></tr>
+<tr>
+ <td>session</td>
+ <td>org.apache.nifi.processor.ProcessSession</td>
+ <td>the session that is used to get, change, and transfer input files</td>
+</tr>
+<tr>
+ <td>context</td>
+ <td>org.apache.nifi.processor.ProcessContext</td>
+ <td>the context (almost unusefull)</td>
+</tr>
+<tr>
+ <td>log</td>
+ <td>org.apache.nifi.logging.ComponentLog</td>
+ <td>the logger for this processor instance</td>
+</tr>
+<tr>
+ <td>REL_SUCCESS</td>
+ <td>org.apache.nifi.processor.Relationship</td>
+ <td>the success relationship</td>
+</tr>
+<tr>
+ <td>REL_FAILURE</td>
+ <td>org.apache.nifi.processor.Relationship</td>
+ <td>the failure relationship</td>
+</tr>
+<tr>
+ <td>flowFile</td>
+ <td>org.apache.nifi.flowfile.FlowFile</td>
+ <td>Binded only if the property `Require flow file`=true for the processor</td>
+</tr>
+<tr>
+ <td>CTL</td>
+ <td>java.util.HashMap</td>
+ <td>Map populated with controller services binded through `CTL.*` processor properties</td>
+</tr>
+<tr>
+ <td>Dynamic processor properties</td>
+ <td>org.apache.nifi.components.PropertyDescriptor</td>
+ <td>All processor properties not started with `CTL.` are binded to script variables</td>
+</tr>
+</table>
+
+<h2>CTL map</h2>
+<p>
+CTL.* objects accessible if corresponding processor property defined.<br/>
+<b>Example:</b> if you defined property <code>`CTL.cache`</code> to DistributedMapCacheClientService, then you can access it from code <code>CTL.cache</code><br/>
+If CTL property references to Database connection pool, then corresponding CTL entry will contain groovy.sql.Sql object connected to database with autocommit=false.</br>
--- End diff --
The version of setAutoCommit() you link to is their master branch. The version that ships with NiFi's Hive NAR is [here](https://github.com/apache/hive/blob/release-1.2.1/jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java#L1197). Strangely, they're handling it backwards from how they do in master.
Just to be safe, it might be better to not explicitly call setAutoCommit(false) and maybe instead allow the script writer to call it on CTL or the appropriate object?
This is your processor and contribution, so you can choose whether to support JDBC connections that don't behave with the calls you're making. Hive is just one example, but there might be others that don't implement certain methods well or at all. It is totally fine to just say "those aren't supported", I'm just bringing it up in case you wanted to address such things. Let me know either way and I'll continue the review/merge, thanks!
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by dlukyanov <gi...@git.apache.org>.
Github user dlukyanov commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r148955987
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/resources/docs/org.apache.nifi.processors.groovyx.ExecuteGroovyScript/additionalDetails.html ---
@@ -0,0 +1,202 @@
+<!DOCTYPE html>
+<html lang="en">
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<head>
+ <meta charset="utf-8" />
+ <title>Groovy</title>
+ <!--link rel="stylesheet" href="../../css/component-usage.css" type="text/css" /-->
+ <link rel="stylesheet" href="/nifi-docs/css/component-usage.css" type="text/css" />
+</head>
+
+<body>
+<h2>Summary</h2>
+<p>This is a grooviest groovy script :)</p>
+<h2>Script Bindings:</h2>
+<table>
+<tr><th>variable</th><th>type</th><th>description</th></tr>
+<tr>
+ <td>session</td>
+ <td>org.apache.nifi.processor.ProcessSession</td>
+ <td>the session that is used to get, change, and transfer input files</td>
+</tr>
+<tr>
+ <td>context</td>
+ <td>org.apache.nifi.processor.ProcessContext</td>
+ <td>the context (almost unusefull)</td>
+</tr>
+<tr>
+ <td>log</td>
+ <td>org.apache.nifi.logging.ComponentLog</td>
+ <td>the logger for this processor instance</td>
+</tr>
+<tr>
+ <td>REL_SUCCESS</td>
+ <td>org.apache.nifi.processor.Relationship</td>
+ <td>the success relationship</td>
+</tr>
+<tr>
+ <td>REL_FAILURE</td>
+ <td>org.apache.nifi.processor.Relationship</td>
+ <td>the failure relationship</td>
+</tr>
+<tr>
+ <td>flowFile</td>
+ <td>org.apache.nifi.flowfile.FlowFile</td>
+ <td>Binded only if the property `Require flow file`=true for the processor</td>
+</tr>
+<tr>
+ <td>CTL</td>
+ <td>java.util.HashMap</td>
+ <td>Map populated with controller services binded through `CTL.*` processor properties</td>
+</tr>
+<tr>
+ <td>Dynamic processor properties</td>
+ <td>org.apache.nifi.components.PropertyDescriptor</td>
+ <td>All processor properties not started with `CTL.` are binded to script variables</td>
+</tr>
+</table>
+
+<h2>CTL map</h2>
+<p>
+CTL.* objects accessible if corresponding processor property defined.<br/>
+<b>Example:</b> if you defined property <code>`CTL.cache`</code> to DistributedMapCacheClientService, then you can access it from code <code>CTL.cache</code><br/>
--- End diff --
Yes, this adds kind of limit. All custom properties with name beginning with `CTL.` automatically supposed to be linked to controller services. It means you can't define custom `CTL.*` property with plain string value. After `CTL.`there could be any chars. On the level of groovy script the additional predefined variable binded : `CTL` - a hashmap with substring after `CTL.` as a key and linked controller service as a value. If you defined `CTL.aaa` and linked it to distributed cache client service, then on level of groovy it'slike `def CTL=[:]` and `CTL['aaa']=LINK_TO_THE_SERVICE`. So, in groovy you can access this service `CTL.aaa.someServiceMethod(...)`
The only conflict expected when you'll try to define custom script variable with name `CTL`. Otherwise-no conflicts.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r124125472
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/java/org/apache/nifi/processors/groovyx/ExecuteGroovyScript.java ---
@@ -0,0 +1,468 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.groovyx;
+
+import java.io.File;
+import java.lang.reflect.Method;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ControllerService;
+import org.apache.nifi.dbcp.DBCPService;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.codehaus.groovy.control.CompilerConfiguration;
+import org.codehaus.groovy.runtime.ResourceGroovyMethods;
+import org.codehaus.groovy.runtime.StackTraceUtils;
+
+import org.apache.nifi.processors.groovyx.sql.OSql;
+import org.apache.nifi.processors.groovyx.util.Files;
+import org.apache.nifi.processors.groovyx.util.Validators;
+import org.apache.nifi.processors.groovyx.flow.GroovyProcessSessionWrap;
+
+import groovy.lang.GroovyShell;
+import groovy.lang.Script;
+
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.ValidationContext;
+
+@EventDriven
--- End diff --
You might want an InputRequirement annotation here, just for clarity. The processor will default to INPUT_ALLOWED which I think is what you want, but it would probably help here, especially in the context of your REQUIRE_FLOW property below
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r148412300
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/resources/docs/org.apache.nifi.processors.groovyx.ExecuteGroovyScript/additionalDetails.html ---
@@ -0,0 +1,202 @@
+<!DOCTYPE html>
+<html lang="en">
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<head>
+ <meta charset="utf-8" />
+ <title>Groovy</title>
+ <!--link rel="stylesheet" href="../../css/component-usage.css" type="text/css" /-->
+ <link rel="stylesheet" href="/nifi-docs/css/component-usage.css" type="text/css" />
+</head>
+
+<body>
+<h2>Summary</h2>
+<p>This is a grooviest groovy script :)</p>
+<h2>Script Bindings:</h2>
+<table>
+<tr><th>variable</th><th>type</th><th>description</th></tr>
+<tr>
+ <td>session</td>
+ <td>org.apache.nifi.processor.ProcessSession</td>
+ <td>the session that is used to get, change, and transfer input files</td>
+</tr>
+<tr>
+ <td>context</td>
+ <td>org.apache.nifi.processor.ProcessContext</td>
+ <td>the context (almost unusefull)</td>
+</tr>
+<tr>
+ <td>log</td>
+ <td>org.apache.nifi.logging.ComponentLog</td>
+ <td>the logger for this processor instance</td>
+</tr>
+<tr>
+ <td>REL_SUCCESS</td>
+ <td>org.apache.nifi.processor.Relationship</td>
+ <td>the success relationship</td>
+</tr>
+<tr>
+ <td>REL_FAILURE</td>
+ <td>org.apache.nifi.processor.Relationship</td>
+ <td>the failure relationship</td>
+</tr>
+<tr>
+ <td>flowFile</td>
+ <td>org.apache.nifi.flowfile.FlowFile</td>
+ <td>Binded only if the property `Require flow file`=true for the processor</td>
+</tr>
+<tr>
+ <td>CTL</td>
+ <td>java.util.HashMap</td>
+ <td>Map populated with controller services binded through `CTL.*` processor properties</td>
+</tr>
+<tr>
+ <td>Dynamic processor properties</td>
+ <td>org.apache.nifi.components.PropertyDescriptor</td>
+ <td>All processor properties not started with `CTL.` are binded to script variables</td>
+</tr>
+</table>
+
+<h2>CTL map</h2>
+<p>
+CTL.* objects accessible if corresponding processor property defined.<br/>
+<b>Example:</b> if you defined property <code>`CTL.cache`</code> to DistributedMapCacheClientService, then you can access it from code <code>CTL.cache</code><br/>
--- End diff --
Can you explain (at least to me here) more about linking with CTL? Does the name of the service have to be "cache" in this case? Or does there need to be a user-defined property called "CTL.cache" added to the service? If the latter, what if the service itself uses the user-defined properties? Seems like there might be conflict in processing?
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by dlukyanov <gi...@git.apache.org>.
Github user dlukyanov commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r124194297
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/java/org/apache/nifi/processors/groovyx/GroovyMethods.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.groovyx;
+
+import groovy.lang.DelegatingMetaClass;
+import groovy.lang.GroovySystem;
+
+import org.apache.nifi.processors.groovyx.flow.ProcessSessionWrap;
+import org.apache.nifi.processors.groovyx.flow.SessionFile;
+
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.flowfile.FlowFile;
+
+import java.util.Collection;
+import java.util.List;
+
+/**
+ * Class to initialize additional groovy methods to work with SessionFile, Relationship, and Sessions easier
+ */
+class GroovyMethods {
+ private static boolean initialized = false;
+
+ static void init() {
+ if (!initialized) {
+ synchronized (GroovyMethods.class) {
+ if (!initialized) {
+ initialized = metaRelationship();
+ }
+ }
+ }
+ }
+
+ private static boolean metaRelationship() {
+ GroovySystem.getMetaClassRegistry().setMetaClass(Relationship.class, new DelegatingMetaClass(Relationship.class) {
+ @Override
+ public Object invokeMethod(Object object, String methodName, Object[] args) {
+ if (object instanceof Relationship) {
+ if ("leftShift".equals(methodName) && args.length == 1) {
+ if (args[0] instanceof SessionFile) {
+ return this.leftShift((Relationship) object, (SessionFile) args[0]);
+ } else if (args[0] instanceof Collection) {
+ return this.leftShift((Relationship) object, (Collection) args[0]);
+ }
+ }
+ }
+ return super.invokeMethod(object, methodName, args);
+ }
+
+ /** to support: REL_SUCCESS << sessionFile */
+ private Relationship leftShift(Relationship r, SessionFile f) {
--- End diff --
we can do both. at the beginning i did rightShift (more logical when coding). then i looked at http://docs.groovy-lang.org/latest/html/groovy-jdk/java/io/OutputStream.html and it implements only leftShift, so i decided to use leftShift. but i like rightShift also)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by dlukyanov <gi...@git.apache.org>.
Github user dlukyanov commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r148957134
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/java/org/apache/nifi/processors/groovyx/ExecuteGroovyScript.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.groovyx;
+
+import java.io.File;
+import java.lang.reflect.Method;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ControllerService;
+import org.apache.nifi.dbcp.DBCPService;
+import org.apache.nifi.flowfile.FlowFile;
--- End diff --
fixed
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by dlukyanov <gi...@git.apache.org>.
Github user dlukyanov commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r148957155
--- Diff: nifi-nar-bundles/pom.xml ---
@@ -94,7 +95,7 @@
<build>
<plugins>
<!-- Makes buildRevision and buildBranch available to the NAR Plugin so they can be populated in the MANIFEST -->
- <plugin>
+ <!--plugin>
--- End diff --
fixed (uncommented)
---
[GitHub] nifi issue #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on the issue:
https://github.com/apache/nifi/pull/1662
I recommend ExecuteGroovyScript as the name, it seems to support the same use case as ExecuteScript but with Groovy-specific bells and whistles. I haven't had a time to review in depth, but will soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1662#discussion_r148412024
--- Diff: nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/resources/docs/org.apache.nifi.processors.groovyx.ExecuteGroovyScript/additionalDetails.html ---
@@ -0,0 +1,202 @@
+<!DOCTYPE html>
+<html lang="en">
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<head>
+ <meta charset="utf-8" />
+ <title>Groovy</title>
+ <!--link rel="stylesheet" href="../../css/component-usage.css" type="text/css" /-->
+ <link rel="stylesheet" href="/nifi-docs/css/component-usage.css" type="text/css" />
+</head>
+
+<body>
+<h2>Summary</h2>
+<p>This is a grooviest groovy script :)</p>
+<h2>Script Bindings:</h2>
+<table>
+<tr><th>variable</th><th>type</th><th>description</th></tr>
+<tr>
+ <td>session</td>
+ <td>org.apache.nifi.processor.ProcessSession</td>
+ <td>the session that is used to get, change, and transfer input files</td>
+</tr>
+<tr>
+ <td>context</td>
+ <td>org.apache.nifi.processor.ProcessContext</td>
+ <td>the context (almost unusefull)</td>
+</tr>
+<tr>
+ <td>log</td>
+ <td>org.apache.nifi.logging.ComponentLog</td>
+ <td>the logger for this processor instance</td>
+</tr>
+<tr>
+ <td>REL_SUCCESS</td>
+ <td>org.apache.nifi.processor.Relationship</td>
+ <td>the success relationship</td>
+</tr>
+<tr>
+ <td>REL_FAILURE</td>
+ <td>org.apache.nifi.processor.Relationship</td>
+ <td>the failure relationship</td>
+</tr>
+<tr>
+ <td>flowFile</td>
+ <td>org.apache.nifi.flowfile.FlowFile</td>
+ <td>Binded only if the property `Require flow file`=true for the processor</td>
+</tr>
+<tr>
+ <td>CTL</td>
+ <td>java.util.HashMap</td>
+ <td>Map populated with controller services binded through `CTL.*` processor properties</td>
+</tr>
+<tr>
+ <td>Dynamic processor properties</td>
+ <td>org.apache.nifi.components.PropertyDescriptor</td>
+ <td>All processor properties not started with `CTL.` are binded to script variables</td>
+</tr>
+</table>
+
+<h2>CTL map</h2>
+<p>
+CTL.* objects accessible if corresponding processor property defined.<br/>
+<b>Example:</b> if you defined property <code>`CTL.cache`</code> to DistributedMapCacheClientService, then you can access it from code <code>CTL.cache</code><br/>
+If CTL property references to Database connection pool, then corresponding CTL entry will contain groovy.sql.Sql object connected to database with autocommit=false.</br>
+CTL - Database transactions automatically rolled back on script exception and committed on success. Script must not disconnect connection.<br/>
+<img src="CTL.gif"/><br/>
+<img src="CTL2.gif"/>
+</p>
+
+<h2>SessionFile - flow file extension</h2>
+<p>
+ The (org.apache.nifi.processors.groovyx.flow.SessionFile) is an actual object returned by session in Extended Groovy processor.<br/>
+ This flow file is a container that references session and the real flow file.<br/>
+ This allows to use simplified syntax to work with file attributes and content:
+</p>
+<p><i>set new attribute value</i></p>
+<pre>
+ flowFile.ATTRIBUTE_NAME = ATTRIBUTE_VALUE
--- End diff --
These features are awesome!
---
[GitHub] nifi issue #1662: NIFI-3688 Extended Groovy Nifi Processor
Posted by joewitt <gi...@git.apache.org>.
Github user joewitt commented on the issue:
https://github.com/apache/nifi/pull/1662
The commit which removed the properties/skip javadoc/src does need to be in the nar bundle itself. We dont want src/javadocs for the nar bundle. Also, the processor needs a name that follows the pattern of VerbSubject instead of GroovyX.
We will definitely need unit tests for this. There are references to Oracle things in there or at least in the screenshot. The LICENSE and NOTICE appears to have a lot of copy/paste from other nars. Are all those dependencies really there?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---