You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by phrocker <gi...@git.apache.org> on 2017/04/22 01:47:48 UTC
[GitHub] nifi-minifi-cpp pull request #80: MINIFI-239: Add InvokeHTTP Processor and c...
GitHub user phrocker opened a pull request:
https://github.com/apache/nifi-minifi-cpp/pull/80
MINIFI-239: Add InvokeHTTP Processor and corresponding tests
Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with MINIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFI-239
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi-minifi-cpp/pull/80.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #80
----
commit 0bd35383f095b07f53fd9608f3643417da80f4fc
Author: Marc Parisi <ph...@apache.org>
Date: 2017-04-21T23:55:56Z
MINIFI-239: Add InvokeHTTP Processor and corresponding tests
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi-minifi-cpp pull request #80: MINIFI-239: Add InvokeHTTP Processor and c...
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/80#discussion_r112986394
--- Diff: libminifi/test/HttpGetIntegrationTest.cpp ---
@@ -0,0 +1,118 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <cassert>
+#include <chrono>
+#include <fstream>
+#include <memory>
+#include <string>
+#include <thread>
+#include <type_traits>
+#include <vector>
+#include <sys/stat.h>
+#include "utils/StringUtils.h"
+#include "../include/core/Core.h"
+#include "../include/core/logging/LogAppenders.h"
+#include "../include/core/logging/BaseLogger.h"
+#include "../include/core/logging/Logger.h"
+#include "../include/core/ProcessGroup.h"
+#include "../include/core/yaml/YamlConfiguration.h"
+#include "../include/FlowController.h"
+#include "../include/properties/Configure.h"
+#include "unit/ProvenanceTestHelper.h"
+
+std::string test_file_location;
+
+void waitToVerifyProcessor() {
+ std::this_thread::sleep_for(std::chrono::seconds(2));
+}
+
+int main(int argc, char **argv) {
+
+ if (argc > 1) {
+ test_file_location = argv[1];
+ }
+ mkdir("content_repository", S_IRWXU | S_IRWXG | S_IROTH | S_IXOTH);
+ std::ostringstream oss;
+ std::unique_ptr<logging::BaseLogger> outputLogger = std::unique_ptr<
+ logging::BaseLogger>(
+ new org::apache::nifi::minifi::core::logging::OutputStreamAppender(oss,
+ 0));
+ std::shared_ptr<logging::Logger> logger = logging::Logger::getLogger();
+ logger->updateLogger(std::move(outputLogger));
+ logger->setLogLevel("debug");
+
+ minifi::Configure *configuration = minifi::Configure::getConfigure();
+
+ std::shared_ptr<core::Repository> test_repo =
+ std::make_shared<TestRepository>();
+ std::shared_ptr<core::Repository> test_flow_repo = std::make_shared<
+ TestFlowRepository>();
+
+ configuration->set(minifi::Configure::nifi_flow_configuration_file,
+ test_file_location);
+
+ std::unique_ptr<core::FlowConfiguration> yaml_ptr = std::unique_ptr<
+ core::YamlConfiguration>(
+ new core::YamlConfiguration(test_repo, test_repo, test_file_location));
+ std::shared_ptr<TestRepository> repo =
+ std::static_pointer_cast<TestRepository>(test_repo);
+
+ std::shared_ptr<minifi::FlowController> controller = std::make_shared<
+ minifi::FlowController>(test_repo, test_flow_repo, std::move(yaml_ptr),
+ DEFAULT_ROOT_GROUP_NAME,
+ true);
+
+ core::YamlConfiguration yaml_config(test_repo, test_repo, test_file_location);
+
+ std::unique_ptr<core::ProcessGroup> ptr = yaml_config.getRoot(
+ test_file_location);
+ std::shared_ptr<core::ProcessGroup> pg = std::shared_ptr<core::ProcessGroup>(
+ ptr.get());
+ ptr.release();
+
+ controller->load();
+ controller->start();
+ waitToVerifyProcessor();
+
+ controller->waitUnload(60000);
+ std::string logs = oss.str();
+ assert(logs.find("key:filename value:") != std::string::npos);
+ assert(
+ logs.find(
+ "key:invokehttp.requerst.url value:https://curl.haxx.se/libcurl/c/httpput.html")
--- End diff --
typo
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi-minifi-cpp pull request #80: MINIFI-239: Add InvokeHTTP Processor and c...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi-minifi-cpp/pull/80
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi-minifi-cpp issue #80: MINIFI-239: Add InvokeHTTP Processor and correspo...
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on the issue:
https://github.com/apache/nifi-minifi-cpp/pull/80
verifying updates
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi-minifi-cpp issue #80: MINIFI-239: Add InvokeHTTP Processor and correspo...
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on the issue:
https://github.com/apache/nifi-minifi-cpp/pull/80
got caught up with some other things, but this looks good. will get merged, thanks for the fixes
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi-minifi-cpp issue #80: MINIFI-239: Add InvokeHTTP Processor and correspo...
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on the issue:
https://github.com/apache/nifi-minifi-cpp/pull/80
@apiri and @randerzander does not encompass all features for this ticket, but provides some basic functionality.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi-minifi-cpp pull request #80: MINIFI-239: Add InvokeHTTP Processor and c...
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/80#discussion_r112986328
--- Diff: libminifi/src/processors/InvokeHTTP.cpp ---
@@ -0,0 +1,537 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "processors/InvokeHTTP.h"
+
+#include <curl/curlbuild.h>
+#include <curl/easy.h>
+#include <sys/_types/_size_t.h>
+#include <sys/_types/_uuid_t.h>
+#include <uuid/uuid.h>
+#include <memory>
+#include <algorithm>
+#include <cctype>
+#include <cstdint>
+#include <cstring>
+#include <iostream>
+#include <iterator>
+#include <map>
+#include <set>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "core/FlowFile.h"
+#include "core/logging/Logger.h"
+#include "core/ProcessContext.h"
+#include "core/Relationship.h"
+#include "io/DataStream.h"
+#include "io/StreamFactory.h"
+#include "ResourceClaim.h"
+#include "utils/StringUtils.h"
+
+#if (__GNUC__ >= 4)
+#if (__GNUC_MINOR__ < 9)
+#include <regex.h>
+#endif
+#endif
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+const char *InvokeHTTP::ProcessorName = "InvokeHTTP";
+
+core::Property InvokeHTTP::Method(
+ "HTTP Method",
+ "HTTP request method (GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS). "
+ "Arbitrary methods are also supported. Methods other than POST, PUT and PATCH will be sent without a message body.",
+ "GET");
+core::Property InvokeHTTP::URL(
+ "Remote URL",
+ "Remote URL which will be connected to, including scheme, host, port, path.",
+ "");
+core::Property InvokeHTTP::ConnectTimeout(
+ "Connection Timeout", "Max wait time for connection to remote service.",
+ "5 secs");
+core::Property InvokeHTTP::ReadTimeout(
+ "Read Timeout", "Max wait time for response from remote service.",
+ "15 secs");
+core::Property InvokeHTTP::DateHeader(
+ "Include Date Header", "Include an RFC-2616 Date header in the request.",
+ "True");
+core::Property InvokeHTTP::FollowRedirects(
+ "Follow Redirects", "Follow HTTP redirects issued by remote server.",
+ "True");
+core::Property InvokeHTTP::AttributesToSend(
+ "Attributes to Send",
+ "Regular expression that defines which attributes to send as HTTP"
+ " headers in the request. If not defined, no attributes are sent as headers.",
+ "");
+core::Property InvokeHTTP::SSLContext(
+ "SSL Context Service",
+ "The SSL Context Service used to provide client certificate information for TLS/SSL (https) connections.",
+ "");
+core::Property InvokeHTTP::ProxyHost(
+ "Proxy Host",
+ "The fully qualified hostname or IP address of the proxy server", "");
+core::Property InvokeHTTP::ProxyPort("Proxy Port",
+ "The port of the proxy server", "");
+core::Property InvokeHTTP::ProxyUser(
+ "invokehttp-proxy-user",
+ "Username to set when authenticating against proxy", "");
+core::Property InvokeHTTP::ProxyPassword(
+ "invokehttp-proxy-password",
+ "Password to set when authenticating against proxy", "");
+core::Property InvokeHTTP::ContentType(
+ "Content-type",
+ "The Content-Type to specify for when content is being transmitted through a PUT, "
+ "POST or PATCH. In the case of an empty value after evaluating an expression language expression, "
+ "Content-Type defaults to",
+ "application/octet-stream");
+core::Property InvokeHTTP::SendBody(
+ "send-message-body",
+ "If true, sends the HTTP message body on POST/PUT/PATCH requests (default). "
+ "If false, suppresses the message body and content-type header for these requests.",
+ "true");
+
+core::Property InvokeHTTP::PropPutOutputAttributes(
+ "Put Response Body in Attribute",
+ "If set, the response body received back will be put into an attribute of the original "
+ "FlowFile instead of a separate FlowFile. The attribute key to put to is determined by evaluating value of this property. ",
+ "");
+core::Property InvokeHTTP::AlwaysOutputResponse(
+ "Always Output Response",
+ "Will force a response FlowFile to be generated and routed to the 'Response' relationship "
+ "regardless of what the server status code received is ",
+ "false");
+core::Property InvokeHTTP::PenalizeOnNoRetry(
+ "Penalize on \"No Retry\"",
+ "Enabling this property will penalize FlowFiles that are routed to the \"No Retry\" relationship.",
+ "false");
+
+const char* InvokeHTTP::STATUS_CODE = "invokehttp.status.code";
+const char* InvokeHTTP::STATUS_MESSAGE = "invokehttp.status.message";
+const char* InvokeHTTP::RESPONSE_BODY = "invokehttp.response.body";
+const char* InvokeHTTP::REQUEST_URL = "invokehttp.requerst.url";
--- End diff --
typo here and in the referencing IT
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi-minifi-cpp pull request #80: MINIFI-239: Add InvokeHTTP Processor and c...
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/80#discussion_r112988725
--- Diff: libminifi/include/processors/InvokeHTTP.h ---
@@ -0,0 +1,228 @@
+/**
+ * InvokeHTTP class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef __INVOKE_HTTP_H__
+#define __INVOKE_HTTP_H__
+
+#include <memory>
+#include <regex>
+
+#include <curl/curl.h>
+#include "FlowFileRecord.h"
+#include "core/Processor.h"
+#include "core/ProcessSession.h"
+#include "core/Core.h"
+#include "core/Property.h"
+#include "utils/ByteInputCallBack.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+struct CallBackPosition {
+ utils::ByteInputCallBack *ptr;
+ size_t pos;
+};
+
+/**
+ * HTTP Response object
+ */
+struct HTTPRequestResponse {
+ std::vector<char> data;
+
+ /**
+ * Receive HTTP Response.
+ */
+ static size_t recieve_write(char * data, size_t size, size_t nmemb,
+ void * p) {
+ return static_cast<HTTPRequestResponse*>(p)->write_content(data, size,
+ nmemb);
+ }
+ /**
+ * Callback for post, put, and patch operations
--- End diff --
comments are not aligned with current impl
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi-minifi-cpp issue #80: MINIFI-239: Add InvokeHTTP Processor and correspo...
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on the issue:
https://github.com/apache/nifi-minifi-cpp/pull/80
reviewing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---