You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by calebj <gi...@git.apache.org> on 2017/10/27 20:13:52 UTC
[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...
GitHub user calebj opened a pull request:
https://github.com/apache/nifi-minifi-cpp/pull/156
MINIFICPP-268 Implement ManipulateArchive processor
Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with MINIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [x] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/NiFiLocal/nifi-minifi-cpp ManipulateArchive
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi-minifi-cpp/pull/156.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #156
----
----
---
[GitHub] nifi-minifi-cpp issue #156: MINIFICPP-268 Implement ManipulateArchive proces...
Posted by calebj <gi...@git.apache.org>.
Github user calebj commented on the issue:
https://github.com/apache/nifi-minifi-cpp/pull/156
Should be ready now. Check to see if the exceptions and error output on misconfigurations fits what you want.
---
[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...
Posted by calebj <gi...@git.apache.org>.
Github user calebj commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/156#discussion_r147534442
--- Diff: extensions/libarchive/ManipulateArchive.cpp ---
@@ -0,0 +1,309 @@
+/**
+ * @file ManipulateArchive.cpp
+ * ManipulateArchive class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <string.h>
+#include <iostream>
+#include <fstream>
+#include <memory>
+#include <string>
+#include <set>
+
+#include <boost/filesystem.hpp>
+
+#include <archive.h>
+#include <archive_entry.h>
+
+#include "ManipulateArchive.h"
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+#include "core/FlowFile.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property ManipulateArchive::Operation("Operation", "Operation to perform on the archive", "");
+core::Property ManipulateArchive::Target("Target", "The path within the archive to perform the operation on", "");
+core::Property ManipulateArchive::Destination("Destination", "Destination for operations (move or copy) which result in new entries", "");
+core::Property ManipulateArchive::Before("Before", "For operations which result in new entries, places the new entry before the entry specified by this property", "");
+core::Property ManipulateArchive::After("After", "For operations which result in new entries, places the new entry after the entry specified by this property", "");
+core::Relationship ManipulateArchive::Success("success", "success operational on the flow record");
+
+char const* ManipulateArchive::OPERATION_REMOVE = "remove";
+char const* ManipulateArchive::OPERATION_COPY = "copy";
+char const* ManipulateArchive::OPERATION_MOVE = "move";
+char const* ManipulateArchive::OPERATION_TOUCH = "touch";
+
+void ManipulateArchive::initialize() {
+ //! Set the supported properties
+ std::set<core::Property> properties;
+ properties.insert(Operation);
+ properties.insert(Target);
+ properties.insert(Destination);
+ properties.insert(Before);
+ properties.insert(After);
+ setSupportedProperties(properties);
+
+ //! Set the supported relationships
+ std::set<core::Relationship> relationships;
+ relationships.insert(Success);
+ setSupportedRelationships(relationships);
+}
+
+void ManipulateArchive::onTrigger(core::ProcessContext *context, core::ProcessSession *session) {
+ std::shared_ptr<core::FlowFile> flowFile = session->get();
+
+ if (!flowFile) {
+ return;
+ }
+
+ std::string operation;
+ context->getProperty(Operation.getName(), operation);
+
+ std::string targetEntry;
+ context->getProperty(Target.getName(), targetEntry);
+
+ std::string destination;
+ context->getProperty(Destination.getName(), destination);
+
+ std::string before;
+ context->getProperty(Before.getName(), before);
+
+ std::string after;
+ context->getProperty(After.getName(), after);
+
+ // TODO(calebj) Validate properties
+
+ FocusArchiveEntry::ArchiveMetadata archiveMetadata;
+ FocusArchiveEntry::ReadCallback readCallback(this, &archiveMetadata);
+ session->read(flowFile, &readCallback);
+
+ logger_->log_info("ManipulateArchive performing operation %s on %s", operation.c_str(), targetEntry.c_str());
+
+ // Perform operation: REMOVE
+ if (operation == OPERATION_REMOVE) {
+ for (auto it = archiveMetadata.entryMetadata.begin(); it != archiveMetadata.entryMetadata.end();) {
+ if ((*it).entryName == targetEntry) {
+ logger_->log_info("ManipulateArchive found entry %s for removal", targetEntry.c_str());
+ std::remove((*it).tmpFileName.c_str());
+ it = archiveMetadata.entryMetadata.erase(it);
+ break;
+ } else {
+ it++;
+ }
+ }
+ }
+
+ // Perform operation: COPY
+ if (operation == OPERATION_COPY) {
+ bool found = false;
+ FocusArchiveEntry::ArchiveEntryMetadata copy;
+
+ // Find item to copy
+ for (auto it = archiveMetadata.entryMetadata.begin(); it != archiveMetadata.entryMetadata.end();) {
+ if ((*it).entryName == targetEntry) {
+ logger_->log_info("ManipulateArchive found entry %s to copy", targetEntry.c_str());
+ copy = *it;
+ found = true;
+ break;
+ } else {
+ it++;
+ }
+ }
+
+ if (found) {
+ // Copy tmp file
+ const auto origTmpFileName = copy.tmpFileName;
+ const auto newTmpFileName = boost::filesystem::unique_path().native();
+ copy.tmpFileName = newTmpFileName;
+
+ {
+ std::ifstream src(origTmpFileName, std::ios::binary);
+ std::ofstream dst(newTmpFileName, std::ios::binary);
+ dst << src.rdbuf();
+ }
+
+ copy.entryName = destination;
+
+ // Update metadata
+ if (after != "") {
+ for (auto it = archiveMetadata.entryMetadata.begin();;) {
+ if (it == archiveMetadata.entryMetadata.end()) {
+ logger_->log_info("ManipulateArchive could not find entry %s to insert copy after, so copy will be appended to end of archive", after.c_str());
+ archiveMetadata.entryMetadata.insert(it, copy);
+ break;
+ }
+
+ if ((*it).entryName == after) {
+ logger_->log_info("ManipulateArchive found entry %s to insert copy after", after.c_str());
+ it++;
+ archiveMetadata.entryMetadata.insert(it, copy);
+ break;
+ } else {
+ it++;
+ }
+ }
+ } else {
+ for (auto it = archiveMetadata.entryMetadata.begin();;) {
+ if (it == archiveMetadata.entryMetadata.end()) {
+ logger_->log_info("ManipulateArchive could not find entry %s to insert copy before, so copy will be appended to end of archive", before.c_str());
+ archiveMetadata.entryMetadata.insert(it, copy);
+ break;
+ }
+
+ if ((*it).entryName == before) {
+ logger_->log_info("ManipulateArchive found entry %s to insert copy before", before.c_str());
+ archiveMetadata.entryMetadata.insert(it, copy);
+
+ break;
+ } else {
+ it++;
+ }
+ }
+ }
+ } else {
+ logger_->log_info("ManipulateArchive could not find entry %s to copy", targetEntry.c_str());
+ }
+ }
+
+ // Perform operation: MOVE
+ if (operation == OPERATION_MOVE) {
+ bool found = false;
+ FocusArchiveEntry::ArchiveEntryMetadata moveEntry;
+
+ // Find item to move
+ for (auto it = archiveMetadata.entryMetadata.begin(); it != archiveMetadata.entryMetadata.end();) {
+ if ((*it).entryName == targetEntry) {
+ logger_->log_info("ManipulateArchive found entry %s to move", targetEntry.c_str());
+ moveEntry = *it;
+ it = archiveMetadata.entryMetadata.erase(it);
+ found = true;
+ break;
+ } else {
+ it++;
+ }
+ }
+
+ if (found) {
+ moveEntry.entryName = destination;
+
+ // Update metadata
+ if (after != "") {
+ for (auto it = archiveMetadata.entryMetadata.begin();;) {
+ if (it == archiveMetadata.entryMetadata.end()) {
+ logger_->log_info("ManipulateArchive could not find entry %s to move entry after, so entry will be appended to end of archive", after.c_str());
+ archiveMetadata.entryMetadata.insert(it, moveEntry);
+ break;
+ }
+
+ if ((*it).entryName == after) {
+ logger_->log_info("ManipulateArchive found entry %s to move entry after", after.c_str());
+ it++;
+ archiveMetadata.entryMetadata.insert(it, moveEntry);
+ break;
+ } else {
+ it++;
+ }
+ }
+ } else {
+ for (auto it = archiveMetadata.entryMetadata.begin();;) {
+ if (it == archiveMetadata.entryMetadata.end()) {
+ logger_->log_info("ManipulateArchive could not find entry %s to move entry before, so entry will be appended to end of archive", before.c_str());
+ archiveMetadata.entryMetadata.insert(it, moveEntry);
+ break;
+ }
+
+ if ((*it).entryName == before) {
+ logger_->log_info("ManipulateArchive found entry %s to move entry before", before.c_str());
+ archiveMetadata.entryMetadata.insert(it, moveEntry);
+
+ break;
+ } else {
+ it++;
+ }
+ }
+ }
+ } else {
+ logger_->log_info("ManipulateArchive could not find entry %s to move", targetEntry.c_str());
+ }
+ }
+
+ // Perform operation: TOUCH
+ if (operation == OPERATION_TOUCH) {
+ FocusArchiveEntry::ArchiveEntryMetadata touchEntry;
+ touchEntry.entryName = targetEntry;
+ touchEntry.entryType = AE_IFREG;
+ touchEntry.entrySize = 0;
+ touchEntry.entryMTime = time(nullptr);
+ touchEntry.entryMTimeNsec = 0;
+ touchEntry.entryUID = 0;
+ touchEntry.entryGID = 0;
+ touchEntry.entryPerm = 0777;
+
+ // Update metadata
+ if (after != "") {
--- End diff --
Indeed. That's the next step; the first was getting the other processor merged, and then get this building on the ever-so-picky Xcode.
---
[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...
Posted by calebj <gi...@git.apache.org>.
Github user calebj commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/156#discussion_r147534705
--- Diff: extensions/libarchive/ManipulateArchive.cpp ---
@@ -0,0 +1,309 @@
+/**
+ * @file ManipulateArchive.cpp
+ * ManipulateArchive class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <string.h>
+#include <iostream>
+#include <fstream>
+#include <memory>
+#include <string>
+#include <set>
+
+#include <boost/filesystem.hpp>
+
+#include <archive.h>
+#include <archive_entry.h>
+
+#include "ManipulateArchive.h"
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+#include "core/FlowFile.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property ManipulateArchive::Operation("Operation", "Operation to perform on the archive", "");
+core::Property ManipulateArchive::Target("Target", "The path within the archive to perform the operation on", "");
+core::Property ManipulateArchive::Destination("Destination", "Destination for operations (move or copy) which result in new entries", "");
+core::Property ManipulateArchive::Before("Before", "For operations which result in new entries, places the new entry before the entry specified by this property", "");
+core::Property ManipulateArchive::After("After", "For operations which result in new entries, places the new entry after the entry specified by this property", "");
+core::Relationship ManipulateArchive::Success("success", "success operational on the flow record");
+
+char const* ManipulateArchive::OPERATION_REMOVE = "remove";
+char const* ManipulateArchive::OPERATION_COPY = "copy";
+char const* ManipulateArchive::OPERATION_MOVE = "move";
+char const* ManipulateArchive::OPERATION_TOUCH = "touch";
+
+void ManipulateArchive::initialize() {
+ //! Set the supported properties
+ std::set<core::Property> properties;
+ properties.insert(Operation);
+ properties.insert(Target);
+ properties.insert(Destination);
+ properties.insert(Before);
+ properties.insert(After);
+ setSupportedProperties(properties);
+
+ //! Set the supported relationships
+ std::set<core::Relationship> relationships;
+ relationships.insert(Success);
+ setSupportedRelationships(relationships);
+}
+
+void ManipulateArchive::onTrigger(core::ProcessContext *context, core::ProcessSession *session) {
+ std::shared_ptr<core::FlowFile> flowFile = session->get();
+
+ if (!flowFile) {
+ return;
+ }
+
+ std::string operation;
+ context->getProperty(Operation.getName(), operation);
+
+ std::string targetEntry;
+ context->getProperty(Target.getName(), targetEntry);
+
+ std::string destination;
+ context->getProperty(Destination.getName(), destination);
+
+ std::string before;
+ context->getProperty(Before.getName(), before);
+
+ std::string after;
+ context->getProperty(After.getName(), after);
+
+ // TODO(calebj) Validate properties
+
+ FocusArchiveEntry::ArchiveMetadata archiveMetadata;
+ FocusArchiveEntry::ReadCallback readCallback(this, &archiveMetadata);
+ session->read(flowFile, &readCallback);
+
+ logger_->log_info("ManipulateArchive performing operation %s on %s", operation.c_str(), targetEntry.c_str());
+
+ // Perform operation: REMOVE
+ if (operation == OPERATION_REMOVE) {
+ for (auto it = archiveMetadata.entryMetadata.begin(); it != archiveMetadata.entryMetadata.end();) {
+ if ((*it).entryName == targetEntry) {
+ logger_->log_info("ManipulateArchive found entry %s for removal", targetEntry.c_str());
+ std::remove((*it).tmpFileName.c_str());
--- End diff --
I'm leaning towards yes. There's no surefire way to determine the uncompressed size of input files.
---
[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi-minifi-cpp/pull/156
---
[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/156#discussion_r147532903
--- Diff: extensions/libarchive/ManipulateArchive.cpp ---
@@ -0,0 +1,309 @@
+/**
+ * @file ManipulateArchive.cpp
+ * ManipulateArchive class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <string.h>
+#include <iostream>
+#include <fstream>
+#include <memory>
+#include <string>
+#include <set>
+
+#include <boost/filesystem.hpp>
+
+#include <archive.h>
+#include <archive_entry.h>
+
+#include "ManipulateArchive.h"
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+#include "core/FlowFile.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property ManipulateArchive::Operation("Operation", "Operation to perform on the archive", "");
+core::Property ManipulateArchive::Target("Target", "The path within the archive to perform the operation on", "");
+core::Property ManipulateArchive::Destination("Destination", "Destination for operations (move or copy) which result in new entries", "");
+core::Property ManipulateArchive::Before("Before", "For operations which result in new entries, places the new entry before the entry specified by this property", "");
+core::Property ManipulateArchive::After("After", "For operations which result in new entries, places the new entry after the entry specified by this property", "");
+core::Relationship ManipulateArchive::Success("success", "success operational on the flow record");
+
+char const* ManipulateArchive::OPERATION_REMOVE = "remove";
+char const* ManipulateArchive::OPERATION_COPY = "copy";
+char const* ManipulateArchive::OPERATION_MOVE = "move";
+char const* ManipulateArchive::OPERATION_TOUCH = "touch";
+
+void ManipulateArchive::initialize() {
+ //! Set the supported properties
+ std::set<core::Property> properties;
+ properties.insert(Operation);
+ properties.insert(Target);
+ properties.insert(Destination);
+ properties.insert(Before);
+ properties.insert(After);
+ setSupportedProperties(properties);
+
+ //! Set the supported relationships
+ std::set<core::Relationship> relationships;
+ relationships.insert(Success);
+ setSupportedRelationships(relationships);
+}
+
+void ManipulateArchive::onTrigger(core::ProcessContext *context, core::ProcessSession *session) {
+ std::shared_ptr<core::FlowFile> flowFile = session->get();
+
+ if (!flowFile) {
+ return;
+ }
+
+ std::string operation;
+ context->getProperty(Operation.getName(), operation);
+
+ std::string targetEntry;
+ context->getProperty(Target.getName(), targetEntry);
+
+ std::string destination;
+ context->getProperty(Destination.getName(), destination);
+
+ std::string before;
+ context->getProperty(Before.getName(), before);
+
+ std::string after;
+ context->getProperty(After.getName(), after);
+
+ // TODO(calebj) Validate properties
--- End diff --
Can these be gathered once, in onSchedule, and validated there?
---
[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/156#discussion_r147533673
--- Diff: extensions/libarchive/ManipulateArchive.cpp ---
@@ -0,0 +1,309 @@
+/**
+ * @file ManipulateArchive.cpp
+ * ManipulateArchive class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <string.h>
+#include <iostream>
+#include <fstream>
+#include <memory>
+#include <string>
+#include <set>
+
+#include <boost/filesystem.hpp>
+
+#include <archive.h>
+#include <archive_entry.h>
+
+#include "ManipulateArchive.h"
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+#include "core/FlowFile.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property ManipulateArchive::Operation("Operation", "Operation to perform on the archive", "");
+core::Property ManipulateArchive::Target("Target", "The path within the archive to perform the operation on", "");
+core::Property ManipulateArchive::Destination("Destination", "Destination for operations (move or copy) which result in new entries", "");
+core::Property ManipulateArchive::Before("Before", "For operations which result in new entries, places the new entry before the entry specified by this property", "");
+core::Property ManipulateArchive::After("After", "For operations which result in new entries, places the new entry after the entry specified by this property", "");
+core::Relationship ManipulateArchive::Success("success", "success operational on the flow record");
+
+char const* ManipulateArchive::OPERATION_REMOVE = "remove";
+char const* ManipulateArchive::OPERATION_COPY = "copy";
+char const* ManipulateArchive::OPERATION_MOVE = "move";
+char const* ManipulateArchive::OPERATION_TOUCH = "touch";
+
+void ManipulateArchive::initialize() {
+ //! Set the supported properties
+ std::set<core::Property> properties;
+ properties.insert(Operation);
+ properties.insert(Target);
+ properties.insert(Destination);
+ properties.insert(Before);
+ properties.insert(After);
+ setSupportedProperties(properties);
+
+ //! Set the supported relationships
+ std::set<core::Relationship> relationships;
+ relationships.insert(Success);
+ setSupportedRelationships(relationships);
+}
+
+void ManipulateArchive::onTrigger(core::ProcessContext *context, core::ProcessSession *session) {
+ std::shared_ptr<core::FlowFile> flowFile = session->get();
+
+ if (!flowFile) {
+ return;
+ }
+
+ std::string operation;
+ context->getProperty(Operation.getName(), operation);
+
+ std::string targetEntry;
+ context->getProperty(Target.getName(), targetEntry);
+
+ std::string destination;
+ context->getProperty(Destination.getName(), destination);
+
+ std::string before;
+ context->getProperty(Before.getName(), before);
+
+ std::string after;
+ context->getProperty(After.getName(), after);
+
+ // TODO(calebj) Validate properties
+
+ FocusArchiveEntry::ArchiveMetadata archiveMetadata;
+ FocusArchiveEntry::ReadCallback readCallback(this, &archiveMetadata);
+ session->read(flowFile, &readCallback);
+
+ logger_->log_info("ManipulateArchive performing operation %s on %s", operation.c_str(), targetEntry.c_str());
+
+ // Perform operation: REMOVE
+ if (operation == OPERATION_REMOVE) {
+ for (auto it = archiveMetadata.entryMetadata.begin(); it != archiveMetadata.entryMetadata.end();) {
+ if ((*it).entryName == targetEntry) {
+ logger_->log_info("ManipulateArchive found entry %s for removal", targetEntry.c_str());
+ std::remove((*it).tmpFileName.c_str());
+ it = archiveMetadata.entryMetadata.erase(it);
+ break;
+ } else {
+ it++;
+ }
+ }
+ }
+
+ // Perform operation: COPY
+ if (operation == OPERATION_COPY) {
+ bool found = false;
+ FocusArchiveEntry::ArchiveEntryMetadata copy;
+
+ // Find item to copy
+ for (auto it = archiveMetadata.entryMetadata.begin(); it != archiveMetadata.entryMetadata.end();) {
+ if ((*it).entryName == targetEntry) {
+ logger_->log_info("ManipulateArchive found entry %s to copy", targetEntry.c_str());
+ copy = *it;
+ found = true;
+ break;
+ } else {
+ it++;
+ }
+ }
+
+ if (found) {
+ // Copy tmp file
+ const auto origTmpFileName = copy.tmpFileName;
+ const auto newTmpFileName = boost::filesystem::unique_path().native();
+ copy.tmpFileName = newTmpFileName;
+
+ {
+ std::ifstream src(origTmpFileName, std::ios::binary);
+ std::ofstream dst(newTmpFileName, std::ios::binary);
+ dst << src.rdbuf();
+ }
+
+ copy.entryName = destination;
+
+ // Update metadata
+ if (after != "") {
+ for (auto it = archiveMetadata.entryMetadata.begin();;) {
+ if (it == archiveMetadata.entryMetadata.end()) {
+ logger_->log_info("ManipulateArchive could not find entry %s to insert copy after, so copy will be appended to end of archive", after.c_str());
+ archiveMetadata.entryMetadata.insert(it, copy);
+ break;
+ }
+
+ if ((*it).entryName == after) {
+ logger_->log_info("ManipulateArchive found entry %s to insert copy after", after.c_str());
+ it++;
+ archiveMetadata.entryMetadata.insert(it, copy);
+ break;
+ } else {
+ it++;
+ }
+ }
+ } else {
+ for (auto it = archiveMetadata.entryMetadata.begin();;) {
+ if (it == archiveMetadata.entryMetadata.end()) {
+ logger_->log_info("ManipulateArchive could not find entry %s to insert copy before, so copy will be appended to end of archive", before.c_str());
+ archiveMetadata.entryMetadata.insert(it, copy);
+ break;
+ }
+
+ if ((*it).entryName == before) {
+ logger_->log_info("ManipulateArchive found entry %s to insert copy before", before.c_str());
+ archiveMetadata.entryMetadata.insert(it, copy);
+
+ break;
+ } else {
+ it++;
+ }
+ }
+ }
+ } else {
+ logger_->log_info("ManipulateArchive could not find entry %s to copy", targetEntry.c_str());
+ }
+ }
+
+ // Perform operation: MOVE
+ if (operation == OPERATION_MOVE) {
+ bool found = false;
+ FocusArchiveEntry::ArchiveEntryMetadata moveEntry;
+
+ // Find item to move
+ for (auto it = archiveMetadata.entryMetadata.begin(); it != archiveMetadata.entryMetadata.end();) {
+ if ((*it).entryName == targetEntry) {
+ logger_->log_info("ManipulateArchive found entry %s to move", targetEntry.c_str());
+ moveEntry = *it;
+ it = archiveMetadata.entryMetadata.erase(it);
+ found = true;
+ break;
+ } else {
+ it++;
+ }
+ }
+
+ if (found) {
+ moveEntry.entryName = destination;
+
+ // Update metadata
+ if (after != "") {
+ for (auto it = archiveMetadata.entryMetadata.begin();;) {
+ if (it == archiveMetadata.entryMetadata.end()) {
+ logger_->log_info("ManipulateArchive could not find entry %s to move entry after, so entry will be appended to end of archive", after.c_str());
+ archiveMetadata.entryMetadata.insert(it, moveEntry);
+ break;
+ }
+
+ if ((*it).entryName == after) {
+ logger_->log_info("ManipulateArchive found entry %s to move entry after", after.c_str());
+ it++;
+ archiveMetadata.entryMetadata.insert(it, moveEntry);
+ break;
+ } else {
+ it++;
+ }
+ }
+ } else {
+ for (auto it = archiveMetadata.entryMetadata.begin();;) {
+ if (it == archiveMetadata.entryMetadata.end()) {
+ logger_->log_info("ManipulateArchive could not find entry %s to move entry before, so entry will be appended to end of archive", before.c_str());
+ archiveMetadata.entryMetadata.insert(it, moveEntry);
+ break;
+ }
+
+ if ((*it).entryName == before) {
+ logger_->log_info("ManipulateArchive found entry %s to move entry before", before.c_str());
+ archiveMetadata.entryMetadata.insert(it, moveEntry);
+
+ break;
+ } else {
+ it++;
+ }
+ }
+ }
+ } else {
+ logger_->log_info("ManipulateArchive could not find entry %s to move", targetEntry.c_str());
+ }
+ }
+
+ // Perform operation: TOUCH
+ if (operation == OPERATION_TOUCH) {
+ FocusArchiveEntry::ArchiveEntryMetadata touchEntry;
+ touchEntry.entryName = targetEntry;
+ touchEntry.entryType = AE_IFREG;
+ touchEntry.entrySize = 0;
+ touchEntry.entryMTime = time(nullptr);
+ touchEntry.entryMTimeNsec = 0;
+ touchEntry.entryUID = 0;
+ touchEntry.entryGID = 0;
+ touchEntry.entryPerm = 0777;
+
+ // Update metadata
+ if (after != "") {
--- End diff --
All of these loops are essentially the same thing, can we combine them into a function and reuse the code?
---
[GitHub] nifi-minifi-cpp issue #156: MINIFICPP-268 Implement ManipulateArchive proces...
Posted by calebj <gi...@git.apache.org>.
Github user calebj commented on the issue:
https://github.com/apache/nifi-minifi-cpp/pull/156
Is this waiting on #181? I have another branch based on that code I can push this PR if needed. Otherwise, is there an estimate as to when this might be merged?
---
[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/156#discussion_r147532806
--- Diff: extensions/libarchive/ManipulateArchive.cpp ---
@@ -0,0 +1,309 @@
+/**
+ * @file ManipulateArchive.cpp
+ * ManipulateArchive class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <string.h>
+#include <iostream>
+#include <fstream>
+#include <memory>
+#include <string>
+#include <set>
+
+#include <boost/filesystem.hpp>
+
+#include <archive.h>
+#include <archive_entry.h>
+
+#include "ManipulateArchive.h"
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+#include "core/FlowFile.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property ManipulateArchive::Operation("Operation", "Operation to perform on the archive", "");
+core::Property ManipulateArchive::Target("Target", "The path within the archive to perform the operation on", "");
+core::Property ManipulateArchive::Destination("Destination", "Destination for operations (move or copy) which result in new entries", "");
+core::Property ManipulateArchive::Before("Before", "For operations which result in new entries, places the new entry before the entry specified by this property", "");
+core::Property ManipulateArchive::After("After", "For operations which result in new entries, places the new entry after the entry specified by this property", "");
+core::Relationship ManipulateArchive::Success("success", "success operational on the flow record");
+
+char const* ManipulateArchive::OPERATION_REMOVE = "remove";
+char const* ManipulateArchive::OPERATION_COPY = "copy";
+char const* ManipulateArchive::OPERATION_MOVE = "move";
+char const* ManipulateArchive::OPERATION_TOUCH = "touch";
+
+void ManipulateArchive::initialize() {
+ //! Set the supported properties
+ std::set<core::Property> properties;
+ properties.insert(Operation);
+ properties.insert(Target);
+ properties.insert(Destination);
+ properties.insert(Before);
+ properties.insert(After);
+ setSupportedProperties(properties);
+
+ //! Set the supported relationships
+ std::set<core::Relationship> relationships;
+ relationships.insert(Success);
+ setSupportedRelationships(relationships);
+}
+
+void ManipulateArchive::onTrigger(core::ProcessContext *context, core::ProcessSession *session) {
+ std::shared_ptr<core::FlowFile> flowFile = session->get();
+
+ if (!flowFile) {
+ return;
+ }
+
+ std::string operation;
+ context->getProperty(Operation.getName(), operation);
+
+ std::string targetEntry;
+ context->getProperty(Target.getName(), targetEntry);
+
+ std::string destination;
+ context->getProperty(Destination.getName(), destination);
+
+ std::string before;
+ context->getProperty(Before.getName(), before);
+
+ std::string after;
+ context->getProperty(After.getName(), after);
+
+ // TODO(calebj) Validate properties
+
+ FocusArchiveEntry::ArchiveMetadata archiveMetadata;
+ FocusArchiveEntry::ReadCallback readCallback(this, &archiveMetadata);
+ session->read(flowFile, &readCallback);
+
+ logger_->log_info("ManipulateArchive performing operation %s on %s", operation.c_str(), targetEntry.c_str());
+
+ // Perform operation: REMOVE
+ if (operation == OPERATION_REMOVE) {
+ for (auto it = archiveMetadata.entryMetadata.begin(); it != archiveMetadata.entryMetadata.end();) {
+ if ((*it).entryName == targetEntry) {
+ logger_->log_info("ManipulateArchive found entry %s for removal", targetEntry.c_str());
+ std::remove((*it).tmpFileName.c_str());
+ it = archiveMetadata.entryMetadata.erase(it);
+ break;
+ } else {
+ it++;
+ }
+ }
+ }
+
+ // Perform operation: COPY
+ if (operation == OPERATION_COPY) {
+ bool found = false;
+ FocusArchiveEntry::ArchiveEntryMetadata copy;
+
+ // Find item to copy
+ for (auto it = archiveMetadata.entryMetadata.begin(); it != archiveMetadata.entryMetadata.end();) {
+ if ((*it).entryName == targetEntry) {
+ logger_->log_info("ManipulateArchive found entry %s to copy", targetEntry.c_str());
+ copy = *it;
+ found = true;
+ break;
+ } else {
+ it++;
+ }
+ }
+
+ if (found) {
+ // Copy tmp file
+ const auto origTmpFileName = copy.tmpFileName;
+ const auto newTmpFileName = boost::filesystem::unique_path().native();
+ copy.tmpFileName = newTmpFileName;
+
+ {
+ std::ifstream src(origTmpFileName, std::ios::binary);
+ std::ofstream dst(newTmpFileName, std::ios::binary);
+ dst << src.rdbuf();
+ }
+
+ copy.entryName = destination;
+
+ // Update metadata
+ if (after != "") {
+ for (auto it = archiveMetadata.entryMetadata.begin();;) {
--- End diff --
minor nit ;;
---
[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/156#discussion_r148075762
--- Diff: extensions/libarchive/ManipulateArchive.cpp ---
@@ -0,0 +1,218 @@
+/**
+ * @file ManipulateArchive.cpp
+ * ManipulateArchive class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <string.h>
+#include <iostream>
+#include <fstream>
+#include <memory>
+#include <string>
+#include <set>
+#include <list>
+#include <algorithm>
+
+#include <boost/filesystem.hpp>
+
+#include <archive.h>
+#include <archive_entry.h>
+
+#include "ManipulateArchive.h"
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+#include "core/FlowFile.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property ManipulateArchive::Operation("Operation", "Operation to perform on the archive (touch, remove, copy, move).", "");
+core::Property ManipulateArchive::Target("Target", "An existing entry within the archive to perform the operation on.", "");
+core::Property ManipulateArchive::Destination("Destination", "Destination for operations (touch, move or copy) which result in new entries.", "");
+core::Property ManipulateArchive::Before("Before", "For operations which result in new entries, places the new entry before the entry specified by this property.", "");
+core::Property ManipulateArchive::After("After", "For operations which result in new entries, places the new entry after the entry specified by this property.", "");
+core::Relationship ManipulateArchive::Success("success", "FlowFiles will be transferred to the success relationship if the operation succeeds.");
+core::Relationship ManipulateArchive::Failure("failure", "FlowFiles will be transferred to the failure relationship if the operation fails.");
+
+char const* ManipulateArchive::OPERATION_REMOVE = "remove";
+char const* ManipulateArchive::OPERATION_COPY = "copy";
+char const* ManipulateArchive::OPERATION_MOVE = "move";
+char const* ManipulateArchive::OPERATION_TOUCH = "touch";
+
+void ManipulateArchive::initialize() {
+ //! Set the supported properties
+ std::set<core::Property> properties;
+ properties.insert(Operation);
+ properties.insert(Target);
+ properties.insert(Destination);
+ properties.insert(Before);
+ properties.insert(After);
+ setSupportedProperties(properties);
+
+ //! Set the supported relationships
+ std::set<core::Relationship> relationships;
+ relationships.insert(Success);
+ relationships.insert(Failure);
+ setSupportedRelationships(relationships);
+}
+
+void ManipulateArchive::onSchedule(core::ProcessContext *context, core::ProcessSessionFactory *sessionFactory) {
+ context->getProperty(Operation.getName(), operation_);
+ std::transform(operation_.begin(), operation_.end(), operation_.begin(), ::tolower);
+
+ bool op_create = operation_ == OPERATION_COPY ||
+ operation_ == OPERATION_MOVE ||
+ operation_ == OPERATION_TOUCH;
+
+ // Operation must be one of copy, move, touch or remove
+ assert(op_create || (operation_ == OPERATION_REMOVE));
+
+ context->getProperty(Target.getName(), targetEntry_);
+ context->getProperty(Destination.getName(), destination_);
+
+ // All operations which create new entries require a set destination
+ assert(op_create != destination_.empty());
+
+ // The only operation that doesn't require an existing target is touch
+ assert((operation_ == OPERATION_TOUCH) == targetEntry_.empty());
+
+ context->getProperty(Before.getName(), before_);
+ context->getProperty(After.getName(), after_);
+
+ // Users may specify one or none of before or after, but never both.
+ assert(!(before_.size() && after_.size()));
+}
+
+void ManipulateArchive::onTrigger(core::ProcessContext *context, core::ProcessSession *session) {
+ std::shared_ptr<core::FlowFile> flowFile = session->get();
+
+ if (!flowFile) {
+ return;
+ }
+
+ FocusArchiveEntry::ArchiveMetadata archiveMetadata;
+ FocusArchiveEntry::ReadCallback readCallback(this, &archiveMetadata);
+ session->read(flowFile, &readCallback);
+
+ auto entries_end = archiveMetadata.entryMetadata.end();
+
+ auto target_position = scanArchiveEntries(archiveMetadata, targetEntry_);
+
+ if (target_position == entries_end && operation_ != OPERATION_TOUCH) {
+ logger_->log_warn("ManipulateArchive could not find entry %s to %s!",
+ targetEntry_.c_str(), operation_.c_str());
+ session->transfer(flowFile, Failure);
+ return;
+ } else {
+ logger_->log_info("ManipulateArchive found %s for %s.",
+ targetEntry_.c_str(), operation_.c_str());
+ }
+
+ if (!destination_.empty()) {
+ auto dest_position = scanArchiveEntries(archiveMetadata, destination_);
+ if (dest_position != entries_end) {
+ logger_->log_warn("ManipulateArchive cannot perform %s to existing destination_ %s!",
+ operation_.c_str(), destination_.c_str());
+ session->transfer(flowFile, Failure);
+ return;
+ }
+ }
+
+ std::list<FocusArchiveEntry::ArchiveEntryMetadata>::iterator position;
+
+ // Small speedup for when neither before or after are provided or needed
+ if ((before_.empty() && after_.empty()) || operation_ == OPERATION_REMOVE) {
+ position = entries_end;
+ } else {
+ std::string positionEntry = after_.empty() ? before_ : after_;
+ position = scanArchiveEntries(archiveMetadata, positionEntry);
+
+ if (position == entries_end)
+ logger_->log_warn("ManipulateArchive could not find entry %s to "
+ "perform %s %s; appending to end of archive...",
--- End diff --
FYI You do not need to use c_str() when using the logger.
---
[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/156#discussion_r148075278
--- Diff: extensions/libarchive/ManipulateArchive.cpp ---
@@ -0,0 +1,218 @@
+/**
+ * @file ManipulateArchive.cpp
+ * ManipulateArchive class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <string.h>
+#include <iostream>
+#include <fstream>
+#include <memory>
+#include <string>
+#include <set>
+#include <list>
+#include <algorithm>
+
+#include <boost/filesystem.hpp>
+
+#include <archive.h>
+#include <archive_entry.h>
+
+#include "ManipulateArchive.h"
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+#include "core/FlowFile.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property ManipulateArchive::Operation("Operation", "Operation to perform on the archive (touch, remove, copy, move).", "");
+core::Property ManipulateArchive::Target("Target", "An existing entry within the archive to perform the operation on.", "");
+core::Property ManipulateArchive::Destination("Destination", "Destination for operations (touch, move or copy) which result in new entries.", "");
+core::Property ManipulateArchive::Before("Before", "For operations which result in new entries, places the new entry before the entry specified by this property.", "");
+core::Property ManipulateArchive::After("After", "For operations which result in new entries, places the new entry after the entry specified by this property.", "");
+core::Relationship ManipulateArchive::Success("success", "FlowFiles will be transferred to the success relationship if the operation succeeds.");
+core::Relationship ManipulateArchive::Failure("failure", "FlowFiles will be transferred to the failure relationship if the operation fails.");
+
+char const* ManipulateArchive::OPERATION_REMOVE = "remove";
+char const* ManipulateArchive::OPERATION_COPY = "copy";
+char const* ManipulateArchive::OPERATION_MOVE = "move";
+char const* ManipulateArchive::OPERATION_TOUCH = "touch";
+
+void ManipulateArchive::initialize() {
+ //! Set the supported properties
+ std::set<core::Property> properties;
+ properties.insert(Operation);
+ properties.insert(Target);
+ properties.insert(Destination);
+ properties.insert(Before);
+ properties.insert(After);
+ setSupportedProperties(properties);
+
+ //! Set the supported relationships
+ std::set<core::Relationship> relationships;
+ relationships.insert(Success);
+ relationships.insert(Failure);
+ setSupportedRelationships(relationships);
+}
+
+void ManipulateArchive::onSchedule(core::ProcessContext *context, core::ProcessSessionFactory *sessionFactory) {
+ context->getProperty(Operation.getName(), operation_);
+ std::transform(operation_.begin(), operation_.end(), operation_.begin(), ::tolower);
+
+ bool op_create = operation_ == OPERATION_COPY ||
+ operation_ == OPERATION_MOVE ||
+ operation_ == OPERATION_TOUCH;
+
+ // Operation must be one of copy, move, touch or remove
+ assert(op_create || (operation_ == OPERATION_REMOVE));
--- End diff --
The usage of assert here is intriguing. Do you not want to avoid running this processor and log an error by throwing an exception if NDEBUG is defined?
---
[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/156#discussion_r148073780
--- Diff: CMakeLists.txt ---
@@ -111,38 +111,38 @@ add_subdirectory(libminifi)
#function(createExtension extensionCondition extensionGuard extensionName description dirName)
## Add http-curl extensions
-createExtension(DISABLE_CURL
- HTTP-CURL
- "HTTP CURL"
- "This enables RESTProtocol, InvokeHTTP, and the HTTPClient for Site to Site"
- "extensions/http-curl"
- "${TEST_DIR}/curl-tests")
+createExtension(DISABLE_CURL
+ HTTP-CURL
--- End diff --
we need to add a skip to this so this doesn't happen when the linter runs. thanks
---
[GitHub] nifi-minifi-cpp issue #156: MINIFICPP-268 Implement ManipulateArchive proces...
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on the issue:
https://github.com/apache/nifi-minifi-cpp/pull/156
@calebj yes, we're removing Boost dependencies.
---
[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/156#discussion_r147533090
--- Diff: extensions/libarchive/ManipulateArchive.cpp ---
@@ -0,0 +1,309 @@
+/**
+ * @file ManipulateArchive.cpp
+ * ManipulateArchive class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <string.h>
+#include <iostream>
+#include <fstream>
+#include <memory>
+#include <string>
+#include <set>
+
+#include <boost/filesystem.hpp>
+
+#include <archive.h>
+#include <archive_entry.h>
+
+#include "ManipulateArchive.h"
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+#include "core/FlowFile.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property ManipulateArchive::Operation("Operation", "Operation to perform on the archive", "");
+core::Property ManipulateArchive::Target("Target", "The path within the archive to perform the operation on", "");
+core::Property ManipulateArchive::Destination("Destination", "Destination for operations (move or copy) which result in new entries", "");
+core::Property ManipulateArchive::Before("Before", "For operations which result in new entries, places the new entry before the entry specified by this property", "");
+core::Property ManipulateArchive::After("After", "For operations which result in new entries, places the new entry after the entry specified by this property", "");
+core::Relationship ManipulateArchive::Success("success", "success operational on the flow record");
+
+char const* ManipulateArchive::OPERATION_REMOVE = "remove";
+char const* ManipulateArchive::OPERATION_COPY = "copy";
+char const* ManipulateArchive::OPERATION_MOVE = "move";
+char const* ManipulateArchive::OPERATION_TOUCH = "touch";
+
+void ManipulateArchive::initialize() {
+ //! Set the supported properties
+ std::set<core::Property> properties;
+ properties.insert(Operation);
+ properties.insert(Target);
+ properties.insert(Destination);
+ properties.insert(Before);
+ properties.insert(After);
+ setSupportedProperties(properties);
+
+ //! Set the supported relationships
+ std::set<core::Relationship> relationships;
+ relationships.insert(Success);
+ setSupportedRelationships(relationships);
+}
+
+void ManipulateArchive::onTrigger(core::ProcessContext *context, core::ProcessSession *session) {
+ std::shared_ptr<core::FlowFile> flowFile = session->get();
+
+ if (!flowFile) {
+ return;
+ }
+
+ std::string operation;
+ context->getProperty(Operation.getName(), operation);
+
+ std::string targetEntry;
+ context->getProperty(Target.getName(), targetEntry);
+
+ std::string destination;
+ context->getProperty(Destination.getName(), destination);
+
+ std::string before;
+ context->getProperty(Before.getName(), before);
+
+ std::string after;
+ context->getProperty(After.getName(), after);
+
+ // TODO(calebj) Validate properties
+
+ FocusArchiveEntry::ArchiveMetadata archiveMetadata;
+ FocusArchiveEntry::ReadCallback readCallback(this, &archiveMetadata);
+ session->read(flowFile, &readCallback);
+
+ logger_->log_info("ManipulateArchive performing operation %s on %s", operation.c_str(), targetEntry.c_str());
+
+ // Perform operation: REMOVE
+ if (operation == OPERATION_REMOVE) {
+ for (auto it = archiveMetadata.entryMetadata.begin(); it != archiveMetadata.entryMetadata.end();) {
+ if ((*it).entryName == targetEntry) {
+ logger_->log_info("ManipulateArchive found entry %s for removal", targetEntry.c_str());
+ std::remove((*it).tmpFileName.c_str());
+ it = archiveMetadata.entryMetadata.erase(it);
+ break;
+ } else {
+ it++;
+ }
+ }
+ }
+
+ // Perform operation: COPY
+ if (operation == OPERATION_COPY) {
+ bool found = false;
+ FocusArchiveEntry::ArchiveEntryMetadata copy;
+
+ // Find item to copy
+ for (auto it = archiveMetadata.entryMetadata.begin(); it != archiveMetadata.entryMetadata.end();) {
--- End diff --
Can clean this up with the range based loop, but that's not something that's absolutely necessary.
---
[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/156#discussion_r147533391
--- Diff: extensions/libarchive/ManipulateArchive.cpp ---
@@ -0,0 +1,309 @@
+/**
+ * @file ManipulateArchive.cpp
+ * ManipulateArchive class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <string.h>
+#include <iostream>
+#include <fstream>
+#include <memory>
+#include <string>
+#include <set>
+
+#include <boost/filesystem.hpp>
+
+#include <archive.h>
+#include <archive_entry.h>
+
+#include "ManipulateArchive.h"
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+#include "core/FlowFile.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property ManipulateArchive::Operation("Operation", "Operation to perform on the archive", "");
+core::Property ManipulateArchive::Target("Target", "The path within the archive to perform the operation on", "");
+core::Property ManipulateArchive::Destination("Destination", "Destination for operations (move or copy) which result in new entries", "");
+core::Property ManipulateArchive::Before("Before", "For operations which result in new entries, places the new entry before the entry specified by this property", "");
+core::Property ManipulateArchive::After("After", "For operations which result in new entries, places the new entry after the entry specified by this property", "");
+core::Relationship ManipulateArchive::Success("success", "success operational on the flow record");
+
+char const* ManipulateArchive::OPERATION_REMOVE = "remove";
+char const* ManipulateArchive::OPERATION_COPY = "copy";
+char const* ManipulateArchive::OPERATION_MOVE = "move";
+char const* ManipulateArchive::OPERATION_TOUCH = "touch";
+
+void ManipulateArchive::initialize() {
+ //! Set the supported properties
+ std::set<core::Property> properties;
+ properties.insert(Operation);
+ properties.insert(Target);
+ properties.insert(Destination);
+ properties.insert(Before);
+ properties.insert(After);
+ setSupportedProperties(properties);
+
+ //! Set the supported relationships
+ std::set<core::Relationship> relationships;
+ relationships.insert(Success);
+ setSupportedRelationships(relationships);
+}
+
+void ManipulateArchive::onTrigger(core::ProcessContext *context, core::ProcessSession *session) {
+ std::shared_ptr<core::FlowFile> flowFile = session->get();
+
+ if (!flowFile) {
+ return;
+ }
+
+ std::string operation;
+ context->getProperty(Operation.getName(), operation);
+
+ std::string targetEntry;
+ context->getProperty(Target.getName(), targetEntry);
+
+ std::string destination;
+ context->getProperty(Destination.getName(), destination);
+
+ std::string before;
+ context->getProperty(Before.getName(), before);
+
+ std::string after;
+ context->getProperty(After.getName(), after);
+
+ // TODO(calebj) Validate properties
+
+ FocusArchiveEntry::ArchiveMetadata archiveMetadata;
+ FocusArchiveEntry::ReadCallback readCallback(this, &archiveMetadata);
+ session->read(flowFile, &readCallback);
+
+ logger_->log_info("ManipulateArchive performing operation %s on %s", operation.c_str(), targetEntry.c_str());
+
+ // Perform operation: REMOVE
+ if (operation == OPERATION_REMOVE) {
+ for (auto it = archiveMetadata.entryMetadata.begin(); it != archiveMetadata.entryMetadata.end();) {
+ if ((*it).entryName == targetEntry) {
+ logger_->log_info("ManipulateArchive found entry %s for removal", targetEntry.c_str());
+ std::remove((*it).tmpFileName.c_str());
--- End diff --
In the event that volatile repositories are used in environments without persistent storage should we disable this suite of extensions? As I understand it, these operations can be performed in memory, right @minifirocks?
Should we put make an option to exclude this and other processors for those without persistent storage?
---
[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...
Posted by calebj <gi...@git.apache.org>.
Github user calebj commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/156#discussion_r148094619
--- Diff: CMakeLists.txt ---
@@ -111,38 +111,38 @@ add_subdirectory(libminifi)
#function(createExtension extensionCondition extensionGuard extensionName description dirName)
## Add http-curl extensions
-createExtension(DISABLE_CURL
- HTTP-CURL
- "HTTP CURL"
- "This enables RESTProtocol, InvokeHTTP, and the HTTPClient for Site to Site"
- "extensions/http-curl"
- "${TEST_DIR}/curl-tests")
+createExtension(DISABLE_CURL
+ HTTP-CURL
--- End diff --
I don't follow. What do you need me to change?
---
[GitHub] nifi-minifi-cpp issue #156: MINIFICPP-268 Implement ManipulateArchive proces...
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on the issue:
https://github.com/apache/nifi-minifi-cpp/pull/156
@calebj I can make the changes to boost if you're fine with it when I merge it, unless you are doing that now.
---