You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by calebj <gi...@git.apache.org> on 2017/10/27 20:13:52 UTC

[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...

GitHub user calebj opened a pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/156

    MINIFICPP-268 Implement ManipulateArchive processor

    Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
    
    In order to streamline the review of the contribution we ask you
    to ensure the following steps have been taken:
    
    ### For all changes:
    - [x] Is there a JIRA ticket associated with this PR? Is it referenced
         in the commit message?
    
    - [x] Does your PR title start with MINIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
    
    - [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
    
    - [x] Is your initial contribution a single, squashed commit?
    
    ### For code changes:
    - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
    - [ ] If applicable, have you updated the LICENSE file?
    - [ ] If applicable, have you updated the NOTICE file?
    
    ### For documentation related changes:
    - [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
    
    ### Note:
    Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/NiFiLocal/nifi-minifi-cpp ManipulateArchive

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/nifi-minifi-cpp/pull/156.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #156
    
----

----


---

[GitHub] nifi-minifi-cpp issue #156: MINIFICPP-268 Implement ManipulateArchive proces...

Posted by calebj <gi...@git.apache.org>.
Github user calebj commented on the issue:

    https://github.com/apache/nifi-minifi-cpp/pull/156
  
    Should be ready now. Check to see if the exceptions and error output on misconfigurations fits what you want.


---

[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...

Posted by calebj <gi...@git.apache.org>.
Github user calebj commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/156#discussion_r147534442
  
    --- Diff: extensions/libarchive/ManipulateArchive.cpp ---
    @@ -0,0 +1,309 @@
    +/**
    + * @file ManipulateArchive.cpp
    + * ManipulateArchive class implementation
    + *
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +#include <string.h>
    +#include <iostream>
    +#include <fstream>
    +#include <memory>
    +#include <string>
    +#include <set>
    +
    +#include <boost/filesystem.hpp>
    +
    +#include <archive.h>
    +#include <archive_entry.h>
    +
    +#include "ManipulateArchive.h"
    +#include "core/ProcessContext.h"
    +#include "core/ProcessSession.h"
    +#include "core/FlowFile.h"
    +
    +namespace org {
    +namespace apache {
    +namespace nifi {
    +namespace minifi {
    +namespace processors {
    +
    +core::Property ManipulateArchive::Operation("Operation", "Operation to perform on the archive", "");
    +core::Property ManipulateArchive::Target("Target", "The path within the archive to perform the operation on", "");
    +core::Property ManipulateArchive::Destination("Destination", "Destination for operations (move or copy) which result in new entries", "");
    +core::Property ManipulateArchive::Before("Before", "For operations which result in new entries, places the new entry before the entry specified by this property", "");
    +core::Property ManipulateArchive::After("After", "For operations which result in new entries, places the new entry after the entry specified by this property", "");
    +core::Relationship ManipulateArchive::Success("success", "success operational on the flow record");
    +
    +char const* ManipulateArchive::OPERATION_REMOVE = "remove";
    +char const* ManipulateArchive::OPERATION_COPY =   "copy";
    +char const* ManipulateArchive::OPERATION_MOVE =   "move";
    +char const* ManipulateArchive::OPERATION_TOUCH =  "touch";
    +
    +void ManipulateArchive::initialize() {
    +    //! Set the supported properties
    +    std::set<core::Property> properties;
    +    properties.insert(Operation);
    +    properties.insert(Target);
    +    properties.insert(Destination);
    +    properties.insert(Before);
    +    properties.insert(After);
    +    setSupportedProperties(properties);
    +
    +    //! Set the supported relationships
    +    std::set<core::Relationship> relationships;
    +    relationships.insert(Success);
    +    setSupportedRelationships(relationships);
    +}
    +
    +void ManipulateArchive::onTrigger(core::ProcessContext *context, core::ProcessSession *session) {
    +    std::shared_ptr<core::FlowFile> flowFile = session->get();
    +
    +    if (!flowFile) {
    +        return;
    +    }
    +
    +    std::string operation;
    +    context->getProperty(Operation.getName(), operation);
    +
    +    std::string targetEntry;
    +    context->getProperty(Target.getName(), targetEntry);
    +
    +    std::string destination;
    +    context->getProperty(Destination.getName(), destination);
    +
    +    std::string before;
    +    context->getProperty(Before.getName(), before);
    +
    +    std::string after;
    +    context->getProperty(After.getName(), after);
    +
    +    // TODO(calebj) Validate properties
    +
    +    FocusArchiveEntry::ArchiveMetadata archiveMetadata;
    +    FocusArchiveEntry::ReadCallback readCallback(this, &archiveMetadata);
    +    session->read(flowFile, &readCallback);
    +
    +    logger_->log_info("ManipulateArchive performing operation %s on %s", operation.c_str(), targetEntry.c_str());
    +
    +    // Perform operation: REMOVE
    +    if (operation == OPERATION_REMOVE) {
    +        for (auto it = archiveMetadata.entryMetadata.begin(); it != archiveMetadata.entryMetadata.end();) {
    +            if ((*it).entryName == targetEntry) {
    +                logger_->log_info("ManipulateArchive found entry %s for removal", targetEntry.c_str());
    +                std::remove((*it).tmpFileName.c_str());
    +                it = archiveMetadata.entryMetadata.erase(it);
    +                break;
    +            } else {
    +                it++;
    +            }
    +        }
    +    }
    +
    +    // Perform operation: COPY
    +    if (operation == OPERATION_COPY) {
    +        bool found = false;
    +        FocusArchiveEntry::ArchiveEntryMetadata copy;
    +
    +        // Find item to copy
    +        for (auto it = archiveMetadata.entryMetadata.begin(); it != archiveMetadata.entryMetadata.end();) {
    +            if ((*it).entryName == targetEntry) {
    +                logger_->log_info("ManipulateArchive found entry %s to copy", targetEntry.c_str());
    +                copy = *it;
    +                found = true;
    +                break;
    +            } else {
    +                it++;
    +            }
    +        }
    +
    +        if (found) {
    +            // Copy tmp file
    +            const auto origTmpFileName = copy.tmpFileName;
    +            const auto newTmpFileName = boost::filesystem::unique_path().native();
    +            copy.tmpFileName = newTmpFileName;
    +
    +            {
    +                std::ifstream src(origTmpFileName, std::ios::binary);
    +                std::ofstream dst(newTmpFileName, std::ios::binary);
    +                dst << src.rdbuf();
    +            }
    +
    +            copy.entryName = destination;
    +
    +            // Update metadata
    +            if (after != "") {
    +                for (auto it = archiveMetadata.entryMetadata.begin();;) {
    +                    if (it == archiveMetadata.entryMetadata.end()) {
    +                        logger_->log_info("ManipulateArchive could not find entry %s to insert copy after, so copy will be appended to end of archive", after.c_str());
    +                        archiveMetadata.entryMetadata.insert(it, copy);
    +                        break;
    +                    }
    +
    +                    if ((*it).entryName == after) {
    +                        logger_->log_info("ManipulateArchive found entry %s to insert copy after", after.c_str());
    +                        it++;
    +                        archiveMetadata.entryMetadata.insert(it, copy);
    +                        break;
    +                    } else {
    +                        it++;
    +                    }
    +                }
    +            } else {
    +                for (auto it = archiveMetadata.entryMetadata.begin();;) {
    +                    if (it == archiveMetadata.entryMetadata.end()) {
    +                        logger_->log_info("ManipulateArchive could not find entry %s to insert copy before, so copy will be appended to end of archive", before.c_str());
    +                        archiveMetadata.entryMetadata.insert(it, copy);
    +                        break;
    +                    }
    +
    +                    if ((*it).entryName == before) {
    +                        logger_->log_info("ManipulateArchive found entry %s to insert copy before", before.c_str());
    +                        archiveMetadata.entryMetadata.insert(it, copy);
    +
    +                        break;
    +                    } else {
    +                        it++;
    +                    }
    +                }
    +            }
    +        } else {
    +            logger_->log_info("ManipulateArchive could not find entry %s to copy", targetEntry.c_str());
    +        }
    +    }
    +
    +    // Perform operation: MOVE
    +    if (operation == OPERATION_MOVE) {
    +        bool found = false;
    +        FocusArchiveEntry::ArchiveEntryMetadata moveEntry;
    +
    +        // Find item to move
    +        for (auto it = archiveMetadata.entryMetadata.begin(); it != archiveMetadata.entryMetadata.end();) {
    +            if ((*it).entryName == targetEntry) {
    +                logger_->log_info("ManipulateArchive found entry %s to move", targetEntry.c_str());
    +                moveEntry = *it;
    +                it = archiveMetadata.entryMetadata.erase(it);
    +                found = true;
    +                break;
    +            } else {
    +                it++;
    +            }
    +        }
    +
    +        if (found) {
    +            moveEntry.entryName = destination;
    +
    +            // Update metadata
    +            if (after != "") {
    +                for (auto it = archiveMetadata.entryMetadata.begin();;) {
    +                    if (it == archiveMetadata.entryMetadata.end()) {
    +                        logger_->log_info("ManipulateArchive could not find entry %s to move entry after, so entry will be appended to end of archive", after.c_str());
    +                        archiveMetadata.entryMetadata.insert(it, moveEntry);
    +                        break;
    +                    }
    +
    +                    if ((*it).entryName == after) {
    +                        logger_->log_info("ManipulateArchive found entry %s to move entry after", after.c_str());
    +                        it++;
    +                        archiveMetadata.entryMetadata.insert(it, moveEntry);
    +                        break;
    +                    } else {
    +                        it++;
    +                    }
    +                }
    +            } else {
    +                for (auto it = archiveMetadata.entryMetadata.begin();;) {
    +                    if (it == archiveMetadata.entryMetadata.end()) {
    +                        logger_->log_info("ManipulateArchive could not find entry %s to move entry before, so entry will be appended to end of archive", before.c_str());
    +                        archiveMetadata.entryMetadata.insert(it, moveEntry);
    +                        break;
    +                    }
    +
    +                    if ((*it).entryName == before) {
    +                        logger_->log_info("ManipulateArchive found entry %s to move entry before", before.c_str());
    +                        archiveMetadata.entryMetadata.insert(it, moveEntry);
    +
    +                        break;
    +                    } else {
    +                        it++;
    +                    }
    +                }
    +            }
    +        } else {
    +            logger_->log_info("ManipulateArchive could not find entry %s to move", targetEntry.c_str());
    +        }
    +    }
    +
    +    // Perform operation: TOUCH
    +    if (operation == OPERATION_TOUCH) {
    +        FocusArchiveEntry::ArchiveEntryMetadata touchEntry;
    +        touchEntry.entryName = targetEntry;
    +        touchEntry.entryType = AE_IFREG;
    +        touchEntry.entrySize = 0;
    +        touchEntry.entryMTime = time(nullptr);
    +        touchEntry.entryMTimeNsec = 0;
    +        touchEntry.entryUID = 0;
    +        touchEntry.entryGID = 0;
    +        touchEntry.entryPerm = 0777;
    +
    +        // Update metadata
    +        if (after != "") {
    --- End diff --
    
    Indeed. That's the next step; the first was getting the other processor merged, and then get this building on the ever-so-picky Xcode.


---

[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...

Posted by calebj <gi...@git.apache.org>.
Github user calebj commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/156#discussion_r147534705
  
    --- Diff: extensions/libarchive/ManipulateArchive.cpp ---
    @@ -0,0 +1,309 @@
    +/**
    + * @file ManipulateArchive.cpp
    + * ManipulateArchive class implementation
    + *
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +#include <string.h>
    +#include <iostream>
    +#include <fstream>
    +#include <memory>
    +#include <string>
    +#include <set>
    +
    +#include <boost/filesystem.hpp>
    +
    +#include <archive.h>
    +#include <archive_entry.h>
    +
    +#include "ManipulateArchive.h"
    +#include "core/ProcessContext.h"
    +#include "core/ProcessSession.h"
    +#include "core/FlowFile.h"
    +
    +namespace org {
    +namespace apache {
    +namespace nifi {
    +namespace minifi {
    +namespace processors {
    +
    +core::Property ManipulateArchive::Operation("Operation", "Operation to perform on the archive", "");
    +core::Property ManipulateArchive::Target("Target", "The path within the archive to perform the operation on", "");
    +core::Property ManipulateArchive::Destination("Destination", "Destination for operations (move or copy) which result in new entries", "");
    +core::Property ManipulateArchive::Before("Before", "For operations which result in new entries, places the new entry before the entry specified by this property", "");
    +core::Property ManipulateArchive::After("After", "For operations which result in new entries, places the new entry after the entry specified by this property", "");
    +core::Relationship ManipulateArchive::Success("success", "success operational on the flow record");
    +
    +char const* ManipulateArchive::OPERATION_REMOVE = "remove";
    +char const* ManipulateArchive::OPERATION_COPY =   "copy";
    +char const* ManipulateArchive::OPERATION_MOVE =   "move";
    +char const* ManipulateArchive::OPERATION_TOUCH =  "touch";
    +
    +void ManipulateArchive::initialize() {
    +    //! Set the supported properties
    +    std::set<core::Property> properties;
    +    properties.insert(Operation);
    +    properties.insert(Target);
    +    properties.insert(Destination);
    +    properties.insert(Before);
    +    properties.insert(After);
    +    setSupportedProperties(properties);
    +
    +    //! Set the supported relationships
    +    std::set<core::Relationship> relationships;
    +    relationships.insert(Success);
    +    setSupportedRelationships(relationships);
    +}
    +
    +void ManipulateArchive::onTrigger(core::ProcessContext *context, core::ProcessSession *session) {
    +    std::shared_ptr<core::FlowFile> flowFile = session->get();
    +
    +    if (!flowFile) {
    +        return;
    +    }
    +
    +    std::string operation;
    +    context->getProperty(Operation.getName(), operation);
    +
    +    std::string targetEntry;
    +    context->getProperty(Target.getName(), targetEntry);
    +
    +    std::string destination;
    +    context->getProperty(Destination.getName(), destination);
    +
    +    std::string before;
    +    context->getProperty(Before.getName(), before);
    +
    +    std::string after;
    +    context->getProperty(After.getName(), after);
    +
    +    // TODO(calebj) Validate properties
    +
    +    FocusArchiveEntry::ArchiveMetadata archiveMetadata;
    +    FocusArchiveEntry::ReadCallback readCallback(this, &archiveMetadata);
    +    session->read(flowFile, &readCallback);
    +
    +    logger_->log_info("ManipulateArchive performing operation %s on %s", operation.c_str(), targetEntry.c_str());
    +
    +    // Perform operation: REMOVE
    +    if (operation == OPERATION_REMOVE) {
    +        for (auto it = archiveMetadata.entryMetadata.begin(); it != archiveMetadata.entryMetadata.end();) {
    +            if ((*it).entryName == targetEntry) {
    +                logger_->log_info("ManipulateArchive found entry %s for removal", targetEntry.c_str());
    +                std::remove((*it).tmpFileName.c_str());
    --- End diff --
    
    I'm leaning towards yes. There's no surefire way to determine the uncompressed size of input files.


---

[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/nifi-minifi-cpp/pull/156


---

[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/156#discussion_r147532903
  
    --- Diff: extensions/libarchive/ManipulateArchive.cpp ---
    @@ -0,0 +1,309 @@
    +/**
    + * @file ManipulateArchive.cpp
    + * ManipulateArchive class implementation
    + *
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +#include <string.h>
    +#include <iostream>
    +#include <fstream>
    +#include <memory>
    +#include <string>
    +#include <set>
    +
    +#include <boost/filesystem.hpp>
    +
    +#include <archive.h>
    +#include <archive_entry.h>
    +
    +#include "ManipulateArchive.h"
    +#include "core/ProcessContext.h"
    +#include "core/ProcessSession.h"
    +#include "core/FlowFile.h"
    +
    +namespace org {
    +namespace apache {
    +namespace nifi {
    +namespace minifi {
    +namespace processors {
    +
    +core::Property ManipulateArchive::Operation("Operation", "Operation to perform on the archive", "");
    +core::Property ManipulateArchive::Target("Target", "The path within the archive to perform the operation on", "");
    +core::Property ManipulateArchive::Destination("Destination", "Destination for operations (move or copy) which result in new entries", "");
    +core::Property ManipulateArchive::Before("Before", "For operations which result in new entries, places the new entry before the entry specified by this property", "");
    +core::Property ManipulateArchive::After("After", "For operations which result in new entries, places the new entry after the entry specified by this property", "");
    +core::Relationship ManipulateArchive::Success("success", "success operational on the flow record");
    +
    +char const* ManipulateArchive::OPERATION_REMOVE = "remove";
    +char const* ManipulateArchive::OPERATION_COPY =   "copy";
    +char const* ManipulateArchive::OPERATION_MOVE =   "move";
    +char const* ManipulateArchive::OPERATION_TOUCH =  "touch";
    +
    +void ManipulateArchive::initialize() {
    +    //! Set the supported properties
    +    std::set<core::Property> properties;
    +    properties.insert(Operation);
    +    properties.insert(Target);
    +    properties.insert(Destination);
    +    properties.insert(Before);
    +    properties.insert(After);
    +    setSupportedProperties(properties);
    +
    +    //! Set the supported relationships
    +    std::set<core::Relationship> relationships;
    +    relationships.insert(Success);
    +    setSupportedRelationships(relationships);
    +}
    +
    +void ManipulateArchive::onTrigger(core::ProcessContext *context, core::ProcessSession *session) {
    +    std::shared_ptr<core::FlowFile> flowFile = session->get();
    +
    +    if (!flowFile) {
    +        return;
    +    }
    +
    +    std::string operation;
    +    context->getProperty(Operation.getName(), operation);
    +
    +    std::string targetEntry;
    +    context->getProperty(Target.getName(), targetEntry);
    +
    +    std::string destination;
    +    context->getProperty(Destination.getName(), destination);
    +
    +    std::string before;
    +    context->getProperty(Before.getName(), before);
    +
    +    std::string after;
    +    context->getProperty(After.getName(), after);
    +
    +    // TODO(calebj) Validate properties
    --- End diff --
    
    Can these be gathered once, in onSchedule, and validated there?


---

[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/156#discussion_r147533673
  
    --- Diff: extensions/libarchive/ManipulateArchive.cpp ---
    @@ -0,0 +1,309 @@
    +/**
    + * @file ManipulateArchive.cpp
    + * ManipulateArchive class implementation
    + *
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +#include <string.h>
    +#include <iostream>
    +#include <fstream>
    +#include <memory>
    +#include <string>
    +#include <set>
    +
    +#include <boost/filesystem.hpp>
    +
    +#include <archive.h>
    +#include <archive_entry.h>
    +
    +#include "ManipulateArchive.h"
    +#include "core/ProcessContext.h"
    +#include "core/ProcessSession.h"
    +#include "core/FlowFile.h"
    +
    +namespace org {
    +namespace apache {
    +namespace nifi {
    +namespace minifi {
    +namespace processors {
    +
    +core::Property ManipulateArchive::Operation("Operation", "Operation to perform on the archive", "");
    +core::Property ManipulateArchive::Target("Target", "The path within the archive to perform the operation on", "");
    +core::Property ManipulateArchive::Destination("Destination", "Destination for operations (move or copy) which result in new entries", "");
    +core::Property ManipulateArchive::Before("Before", "For operations which result in new entries, places the new entry before the entry specified by this property", "");
    +core::Property ManipulateArchive::After("After", "For operations which result in new entries, places the new entry after the entry specified by this property", "");
    +core::Relationship ManipulateArchive::Success("success", "success operational on the flow record");
    +
    +char const* ManipulateArchive::OPERATION_REMOVE = "remove";
    +char const* ManipulateArchive::OPERATION_COPY =   "copy";
    +char const* ManipulateArchive::OPERATION_MOVE =   "move";
    +char const* ManipulateArchive::OPERATION_TOUCH =  "touch";
    +
    +void ManipulateArchive::initialize() {
    +    //! Set the supported properties
    +    std::set<core::Property> properties;
    +    properties.insert(Operation);
    +    properties.insert(Target);
    +    properties.insert(Destination);
    +    properties.insert(Before);
    +    properties.insert(After);
    +    setSupportedProperties(properties);
    +
    +    //! Set the supported relationships
    +    std::set<core::Relationship> relationships;
    +    relationships.insert(Success);
    +    setSupportedRelationships(relationships);
    +}
    +
    +void ManipulateArchive::onTrigger(core::ProcessContext *context, core::ProcessSession *session) {
    +    std::shared_ptr<core::FlowFile> flowFile = session->get();
    +
    +    if (!flowFile) {
    +        return;
    +    }
    +
    +    std::string operation;
    +    context->getProperty(Operation.getName(), operation);
    +
    +    std::string targetEntry;
    +    context->getProperty(Target.getName(), targetEntry);
    +
    +    std::string destination;
    +    context->getProperty(Destination.getName(), destination);
    +
    +    std::string before;
    +    context->getProperty(Before.getName(), before);
    +
    +    std::string after;
    +    context->getProperty(After.getName(), after);
    +
    +    // TODO(calebj) Validate properties
    +
    +    FocusArchiveEntry::ArchiveMetadata archiveMetadata;
    +    FocusArchiveEntry::ReadCallback readCallback(this, &archiveMetadata);
    +    session->read(flowFile, &readCallback);
    +
    +    logger_->log_info("ManipulateArchive performing operation %s on %s", operation.c_str(), targetEntry.c_str());
    +
    +    // Perform operation: REMOVE
    +    if (operation == OPERATION_REMOVE) {
    +        for (auto it = archiveMetadata.entryMetadata.begin(); it != archiveMetadata.entryMetadata.end();) {
    +            if ((*it).entryName == targetEntry) {
    +                logger_->log_info("ManipulateArchive found entry %s for removal", targetEntry.c_str());
    +                std::remove((*it).tmpFileName.c_str());
    +                it = archiveMetadata.entryMetadata.erase(it);
    +                break;
    +            } else {
    +                it++;
    +            }
    +        }
    +    }
    +
    +    // Perform operation: COPY
    +    if (operation == OPERATION_COPY) {
    +        bool found = false;
    +        FocusArchiveEntry::ArchiveEntryMetadata copy;
    +
    +        // Find item to copy
    +        for (auto it = archiveMetadata.entryMetadata.begin(); it != archiveMetadata.entryMetadata.end();) {
    +            if ((*it).entryName == targetEntry) {
    +                logger_->log_info("ManipulateArchive found entry %s to copy", targetEntry.c_str());
    +                copy = *it;
    +                found = true;
    +                break;
    +            } else {
    +                it++;
    +            }
    +        }
    +
    +        if (found) {
    +            // Copy tmp file
    +            const auto origTmpFileName = copy.tmpFileName;
    +            const auto newTmpFileName = boost::filesystem::unique_path().native();
    +            copy.tmpFileName = newTmpFileName;
    +
    +            {
    +                std::ifstream src(origTmpFileName, std::ios::binary);
    +                std::ofstream dst(newTmpFileName, std::ios::binary);
    +                dst << src.rdbuf();
    +            }
    +
    +            copy.entryName = destination;
    +
    +            // Update metadata
    +            if (after != "") {
    +                for (auto it = archiveMetadata.entryMetadata.begin();;) {
    +                    if (it == archiveMetadata.entryMetadata.end()) {
    +                        logger_->log_info("ManipulateArchive could not find entry %s to insert copy after, so copy will be appended to end of archive", after.c_str());
    +                        archiveMetadata.entryMetadata.insert(it, copy);
    +                        break;
    +                    }
    +
    +                    if ((*it).entryName == after) {
    +                        logger_->log_info("ManipulateArchive found entry %s to insert copy after", after.c_str());
    +                        it++;
    +                        archiveMetadata.entryMetadata.insert(it, copy);
    +                        break;
    +                    } else {
    +                        it++;
    +                    }
    +                }
    +            } else {
    +                for (auto it = archiveMetadata.entryMetadata.begin();;) {
    +                    if (it == archiveMetadata.entryMetadata.end()) {
    +                        logger_->log_info("ManipulateArchive could not find entry %s to insert copy before, so copy will be appended to end of archive", before.c_str());
    +                        archiveMetadata.entryMetadata.insert(it, copy);
    +                        break;
    +                    }
    +
    +                    if ((*it).entryName == before) {
    +                        logger_->log_info("ManipulateArchive found entry %s to insert copy before", before.c_str());
    +                        archiveMetadata.entryMetadata.insert(it, copy);
    +
    +                        break;
    +                    } else {
    +                        it++;
    +                    }
    +                }
    +            }
    +        } else {
    +            logger_->log_info("ManipulateArchive could not find entry %s to copy", targetEntry.c_str());
    +        }
    +    }
    +
    +    // Perform operation: MOVE
    +    if (operation == OPERATION_MOVE) {
    +        bool found = false;
    +        FocusArchiveEntry::ArchiveEntryMetadata moveEntry;
    +
    +        // Find item to move
    +        for (auto it = archiveMetadata.entryMetadata.begin(); it != archiveMetadata.entryMetadata.end();) {
    +            if ((*it).entryName == targetEntry) {
    +                logger_->log_info("ManipulateArchive found entry %s to move", targetEntry.c_str());
    +                moveEntry = *it;
    +                it = archiveMetadata.entryMetadata.erase(it);
    +                found = true;
    +                break;
    +            } else {
    +                it++;
    +            }
    +        }
    +
    +        if (found) {
    +            moveEntry.entryName = destination;
    +
    +            // Update metadata
    +            if (after != "") {
    +                for (auto it = archiveMetadata.entryMetadata.begin();;) {
    +                    if (it == archiveMetadata.entryMetadata.end()) {
    +                        logger_->log_info("ManipulateArchive could not find entry %s to move entry after, so entry will be appended to end of archive", after.c_str());
    +                        archiveMetadata.entryMetadata.insert(it, moveEntry);
    +                        break;
    +                    }
    +
    +                    if ((*it).entryName == after) {
    +                        logger_->log_info("ManipulateArchive found entry %s to move entry after", after.c_str());
    +                        it++;
    +                        archiveMetadata.entryMetadata.insert(it, moveEntry);
    +                        break;
    +                    } else {
    +                        it++;
    +                    }
    +                }
    +            } else {
    +                for (auto it = archiveMetadata.entryMetadata.begin();;) {
    +                    if (it == archiveMetadata.entryMetadata.end()) {
    +                        logger_->log_info("ManipulateArchive could not find entry %s to move entry before, so entry will be appended to end of archive", before.c_str());
    +                        archiveMetadata.entryMetadata.insert(it, moveEntry);
    +                        break;
    +                    }
    +
    +                    if ((*it).entryName == before) {
    +                        logger_->log_info("ManipulateArchive found entry %s to move entry before", before.c_str());
    +                        archiveMetadata.entryMetadata.insert(it, moveEntry);
    +
    +                        break;
    +                    } else {
    +                        it++;
    +                    }
    +                }
    +            }
    +        } else {
    +            logger_->log_info("ManipulateArchive could not find entry %s to move", targetEntry.c_str());
    +        }
    +    }
    +
    +    // Perform operation: TOUCH
    +    if (operation == OPERATION_TOUCH) {
    +        FocusArchiveEntry::ArchiveEntryMetadata touchEntry;
    +        touchEntry.entryName = targetEntry;
    +        touchEntry.entryType = AE_IFREG;
    +        touchEntry.entrySize = 0;
    +        touchEntry.entryMTime = time(nullptr);
    +        touchEntry.entryMTimeNsec = 0;
    +        touchEntry.entryUID = 0;
    +        touchEntry.entryGID = 0;
    +        touchEntry.entryPerm = 0777;
    +
    +        // Update metadata
    +        if (after != "") {
    --- End diff --
    
    All of these loops are essentially the same thing, can we combine them into a function and reuse the code? 


---

[GitHub] nifi-minifi-cpp issue #156: MINIFICPP-268 Implement ManipulateArchive proces...

Posted by calebj <gi...@git.apache.org>.
Github user calebj commented on the issue:

    https://github.com/apache/nifi-minifi-cpp/pull/156
  
    Is this waiting on #181? I have another branch based on that code I can push this PR if needed. Otherwise, is there an estimate as to when this might be merged?


---

[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/156#discussion_r147532806
  
    --- Diff: extensions/libarchive/ManipulateArchive.cpp ---
    @@ -0,0 +1,309 @@
    +/**
    + * @file ManipulateArchive.cpp
    + * ManipulateArchive class implementation
    + *
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +#include <string.h>
    +#include <iostream>
    +#include <fstream>
    +#include <memory>
    +#include <string>
    +#include <set>
    +
    +#include <boost/filesystem.hpp>
    +
    +#include <archive.h>
    +#include <archive_entry.h>
    +
    +#include "ManipulateArchive.h"
    +#include "core/ProcessContext.h"
    +#include "core/ProcessSession.h"
    +#include "core/FlowFile.h"
    +
    +namespace org {
    +namespace apache {
    +namespace nifi {
    +namespace minifi {
    +namespace processors {
    +
    +core::Property ManipulateArchive::Operation("Operation", "Operation to perform on the archive", "");
    +core::Property ManipulateArchive::Target("Target", "The path within the archive to perform the operation on", "");
    +core::Property ManipulateArchive::Destination("Destination", "Destination for operations (move or copy) which result in new entries", "");
    +core::Property ManipulateArchive::Before("Before", "For operations which result in new entries, places the new entry before the entry specified by this property", "");
    +core::Property ManipulateArchive::After("After", "For operations which result in new entries, places the new entry after the entry specified by this property", "");
    +core::Relationship ManipulateArchive::Success("success", "success operational on the flow record");
    +
    +char const* ManipulateArchive::OPERATION_REMOVE = "remove";
    +char const* ManipulateArchive::OPERATION_COPY =   "copy";
    +char const* ManipulateArchive::OPERATION_MOVE =   "move";
    +char const* ManipulateArchive::OPERATION_TOUCH =  "touch";
    +
    +void ManipulateArchive::initialize() {
    +    //! Set the supported properties
    +    std::set<core::Property> properties;
    +    properties.insert(Operation);
    +    properties.insert(Target);
    +    properties.insert(Destination);
    +    properties.insert(Before);
    +    properties.insert(After);
    +    setSupportedProperties(properties);
    +
    +    //! Set the supported relationships
    +    std::set<core::Relationship> relationships;
    +    relationships.insert(Success);
    +    setSupportedRelationships(relationships);
    +}
    +
    +void ManipulateArchive::onTrigger(core::ProcessContext *context, core::ProcessSession *session) {
    +    std::shared_ptr<core::FlowFile> flowFile = session->get();
    +
    +    if (!flowFile) {
    +        return;
    +    }
    +
    +    std::string operation;
    +    context->getProperty(Operation.getName(), operation);
    +
    +    std::string targetEntry;
    +    context->getProperty(Target.getName(), targetEntry);
    +
    +    std::string destination;
    +    context->getProperty(Destination.getName(), destination);
    +
    +    std::string before;
    +    context->getProperty(Before.getName(), before);
    +
    +    std::string after;
    +    context->getProperty(After.getName(), after);
    +
    +    // TODO(calebj) Validate properties
    +
    +    FocusArchiveEntry::ArchiveMetadata archiveMetadata;
    +    FocusArchiveEntry::ReadCallback readCallback(this, &archiveMetadata);
    +    session->read(flowFile, &readCallback);
    +
    +    logger_->log_info("ManipulateArchive performing operation %s on %s", operation.c_str(), targetEntry.c_str());
    +
    +    // Perform operation: REMOVE
    +    if (operation == OPERATION_REMOVE) {
    +        for (auto it = archiveMetadata.entryMetadata.begin(); it != archiveMetadata.entryMetadata.end();) {
    +            if ((*it).entryName == targetEntry) {
    +                logger_->log_info("ManipulateArchive found entry %s for removal", targetEntry.c_str());
    +                std::remove((*it).tmpFileName.c_str());
    +                it = archiveMetadata.entryMetadata.erase(it);
    +                break;
    +            } else {
    +                it++;
    +            }
    +        }
    +    }
    +
    +    // Perform operation: COPY
    +    if (operation == OPERATION_COPY) {
    +        bool found = false;
    +        FocusArchiveEntry::ArchiveEntryMetadata copy;
    +
    +        // Find item to copy
    +        for (auto it = archiveMetadata.entryMetadata.begin(); it != archiveMetadata.entryMetadata.end();) {
    +            if ((*it).entryName == targetEntry) {
    +                logger_->log_info("ManipulateArchive found entry %s to copy", targetEntry.c_str());
    +                copy = *it;
    +                found = true;
    +                break;
    +            } else {
    +                it++;
    +            }
    +        }
    +
    +        if (found) {
    +            // Copy tmp file
    +            const auto origTmpFileName = copy.tmpFileName;
    +            const auto newTmpFileName = boost::filesystem::unique_path().native();
    +            copy.tmpFileName = newTmpFileName;
    +
    +            {
    +                std::ifstream src(origTmpFileName, std::ios::binary);
    +                std::ofstream dst(newTmpFileName, std::ios::binary);
    +                dst << src.rdbuf();
    +            }
    +
    +            copy.entryName = destination;
    +
    +            // Update metadata
    +            if (after != "") {
    +                for (auto it = archiveMetadata.entryMetadata.begin();;) {
    --- End diff --
    
    minor nit ;;


---

[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/156#discussion_r148075762
  
    --- Diff: extensions/libarchive/ManipulateArchive.cpp ---
    @@ -0,0 +1,218 @@
    +/**
    + * @file ManipulateArchive.cpp
    + * ManipulateArchive class implementation
    + *
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +#include <string.h>
    +#include <iostream>
    +#include <fstream>
    +#include <memory>
    +#include <string>
    +#include <set>
    +#include <list>
    +#include <algorithm>
    +
    +#include <boost/filesystem.hpp>
    +
    +#include <archive.h>
    +#include <archive_entry.h>
    +
    +#include "ManipulateArchive.h"
    +#include "core/ProcessContext.h"
    +#include "core/ProcessSession.h"
    +#include "core/FlowFile.h"
    +
    +namespace org {
    +namespace apache {
    +namespace nifi {
    +namespace minifi {
    +namespace processors {
    +
    +core::Property ManipulateArchive::Operation("Operation", "Operation to perform on the archive (touch, remove, copy, move).", "");
    +core::Property ManipulateArchive::Target("Target", "An existing entry within the archive to perform the operation on.", "");
    +core::Property ManipulateArchive::Destination("Destination", "Destination for operations (touch, move or copy) which result in new entries.", "");
    +core::Property ManipulateArchive::Before("Before", "For operations which result in new entries, places the new entry before the entry specified by this property.", "");
    +core::Property ManipulateArchive::After("After", "For operations which result in new entries, places the new entry after the entry specified by this property.", "");
    +core::Relationship ManipulateArchive::Success("success", "FlowFiles will be transferred to the success relationship if the operation succeeds.");
    +core::Relationship ManipulateArchive::Failure("failure", "FlowFiles will be transferred to the failure relationship if the operation fails.");
    +
    +char const* ManipulateArchive::OPERATION_REMOVE = "remove";
    +char const* ManipulateArchive::OPERATION_COPY =   "copy";
    +char const* ManipulateArchive::OPERATION_MOVE =   "move";
    +char const* ManipulateArchive::OPERATION_TOUCH =  "touch";
    +
    +void ManipulateArchive::initialize() {
    +    //! Set the supported properties
    +    std::set<core::Property> properties;
    +    properties.insert(Operation);
    +    properties.insert(Target);
    +    properties.insert(Destination);
    +    properties.insert(Before);
    +    properties.insert(After);
    +    setSupportedProperties(properties);
    +
    +    //! Set the supported relationships
    +    std::set<core::Relationship> relationships;
    +    relationships.insert(Success);
    +    relationships.insert(Failure);
    +    setSupportedRelationships(relationships);
    +}
    +
    +void ManipulateArchive::onSchedule(core::ProcessContext *context, core::ProcessSessionFactory *sessionFactory) {
    +    context->getProperty(Operation.getName(), operation_);
    +    std::transform(operation_.begin(), operation_.end(), operation_.begin(), ::tolower);
    +
    +    bool op_create = operation_ == OPERATION_COPY ||
    +                     operation_ == OPERATION_MOVE ||
    +                     operation_ == OPERATION_TOUCH;
    +
    +    // Operation must be one of copy, move, touch or remove
    +    assert(op_create || (operation_ == OPERATION_REMOVE));
    +
    +    context->getProperty(Target.getName(), targetEntry_);
    +    context->getProperty(Destination.getName(), destination_);
    +
    +    // All operations which create new entries require a set destination
    +    assert(op_create != destination_.empty());
    +
    +    // The only operation that doesn't require an existing target is touch
    +    assert((operation_ == OPERATION_TOUCH) == targetEntry_.empty());
    +
    +    context->getProperty(Before.getName(), before_);
    +    context->getProperty(After.getName(), after_);
    +
    +    // Users may specify one or none of before or after, but never both.
    +    assert(!(before_.size() && after_.size()));
    +}
    +
    +void ManipulateArchive::onTrigger(core::ProcessContext *context, core::ProcessSession *session) {
    +    std::shared_ptr<core::FlowFile> flowFile = session->get();
    +
    +    if (!flowFile) {
    +        return;
    +    }
    +
    +    FocusArchiveEntry::ArchiveMetadata archiveMetadata;
    +    FocusArchiveEntry::ReadCallback readCallback(this, &archiveMetadata);
    +    session->read(flowFile, &readCallback);
    +
    +    auto entries_end = archiveMetadata.entryMetadata.end();
    +
    +    auto target_position = scanArchiveEntries(archiveMetadata, targetEntry_);
    +
    +    if (target_position == entries_end && operation_ != OPERATION_TOUCH) {
    +        logger_->log_warn("ManipulateArchive could not find entry %s to %s!",
    +                          targetEntry_.c_str(), operation_.c_str());
    +        session->transfer(flowFile, Failure);
    +        return;
    +    } else {
    +        logger_->log_info("ManipulateArchive found %s for %s.",
    +                          targetEntry_.c_str(), operation_.c_str());
    +    }
    +
    +    if (!destination_.empty()) {
    +        auto dest_position = scanArchiveEntries(archiveMetadata, destination_);
    +        if (dest_position != entries_end) {
    +            logger_->log_warn("ManipulateArchive cannot perform %s to existing destination_ %s!",
    +                              operation_.c_str(), destination_.c_str());
    +            session->transfer(flowFile, Failure);
    +            return;
    +        }
    +    }
    +
    +    std::list<FocusArchiveEntry::ArchiveEntryMetadata>::iterator position;
    +
    +    // Small speedup for when neither before or after are provided or needed
    +    if ((before_.empty() && after_.empty()) || operation_ == OPERATION_REMOVE) {
    +        position = entries_end;
    +    } else {
    +        std::string positionEntry = after_.empty() ? before_ : after_;
    +        position = scanArchiveEntries(archiveMetadata, positionEntry);
    +
    +        if (position == entries_end)
    +            logger_->log_warn("ManipulateArchive could not find entry %s to "
    +                              "perform %s %s; appending to end of archive...",
    --- End diff --
    
    FYI You do not need to use c_str() when using the logger. 


---

[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/156#discussion_r148075278
  
    --- Diff: extensions/libarchive/ManipulateArchive.cpp ---
    @@ -0,0 +1,218 @@
    +/**
    + * @file ManipulateArchive.cpp
    + * ManipulateArchive class implementation
    + *
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +#include <string.h>
    +#include <iostream>
    +#include <fstream>
    +#include <memory>
    +#include <string>
    +#include <set>
    +#include <list>
    +#include <algorithm>
    +
    +#include <boost/filesystem.hpp>
    +
    +#include <archive.h>
    +#include <archive_entry.h>
    +
    +#include "ManipulateArchive.h"
    +#include "core/ProcessContext.h"
    +#include "core/ProcessSession.h"
    +#include "core/FlowFile.h"
    +
    +namespace org {
    +namespace apache {
    +namespace nifi {
    +namespace minifi {
    +namespace processors {
    +
    +core::Property ManipulateArchive::Operation("Operation", "Operation to perform on the archive (touch, remove, copy, move).", "");
    +core::Property ManipulateArchive::Target("Target", "An existing entry within the archive to perform the operation on.", "");
    +core::Property ManipulateArchive::Destination("Destination", "Destination for operations (touch, move or copy) which result in new entries.", "");
    +core::Property ManipulateArchive::Before("Before", "For operations which result in new entries, places the new entry before the entry specified by this property.", "");
    +core::Property ManipulateArchive::After("After", "For operations which result in new entries, places the new entry after the entry specified by this property.", "");
    +core::Relationship ManipulateArchive::Success("success", "FlowFiles will be transferred to the success relationship if the operation succeeds.");
    +core::Relationship ManipulateArchive::Failure("failure", "FlowFiles will be transferred to the failure relationship if the operation fails.");
    +
    +char const* ManipulateArchive::OPERATION_REMOVE = "remove";
    +char const* ManipulateArchive::OPERATION_COPY =   "copy";
    +char const* ManipulateArchive::OPERATION_MOVE =   "move";
    +char const* ManipulateArchive::OPERATION_TOUCH =  "touch";
    +
    +void ManipulateArchive::initialize() {
    +    //! Set the supported properties
    +    std::set<core::Property> properties;
    +    properties.insert(Operation);
    +    properties.insert(Target);
    +    properties.insert(Destination);
    +    properties.insert(Before);
    +    properties.insert(After);
    +    setSupportedProperties(properties);
    +
    +    //! Set the supported relationships
    +    std::set<core::Relationship> relationships;
    +    relationships.insert(Success);
    +    relationships.insert(Failure);
    +    setSupportedRelationships(relationships);
    +}
    +
    +void ManipulateArchive::onSchedule(core::ProcessContext *context, core::ProcessSessionFactory *sessionFactory) {
    +    context->getProperty(Operation.getName(), operation_);
    +    std::transform(operation_.begin(), operation_.end(), operation_.begin(), ::tolower);
    +
    +    bool op_create = operation_ == OPERATION_COPY ||
    +                     operation_ == OPERATION_MOVE ||
    +                     operation_ == OPERATION_TOUCH;
    +
    +    // Operation must be one of copy, move, touch or remove
    +    assert(op_create || (operation_ == OPERATION_REMOVE));
    --- End diff --
    
    The usage of assert here is intriguing. Do you not want to avoid running this processor and log an error by throwing an exception if NDEBUG is defined? 


---

[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/156#discussion_r148073780
  
    --- Diff: CMakeLists.txt ---
    @@ -111,38 +111,38 @@ add_subdirectory(libminifi)
     #function(createExtension extensionCondition extensionGuard extensionName description dirName)
     
     ## Add http-curl extensions
    -createExtension(DISABLE_CURL 
    -				HTTP-CURL 
    -				"HTTP CURL" 
    -				"This enables RESTProtocol, InvokeHTTP, and the HTTPClient for Site to Site" 
    -				"extensions/http-curl"
    -				"${TEST_DIR}/curl-tests")
    +createExtension(DISABLE_CURL
    +                HTTP-CURL
    --- End diff --
    
    we need to add a skip to this so this doesn't happen when the linter runs. thanks


---

[GitHub] nifi-minifi-cpp issue #156: MINIFICPP-268 Implement ManipulateArchive proces...

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on the issue:

    https://github.com/apache/nifi-minifi-cpp/pull/156
  
    @calebj yes, we're removing Boost dependencies. 


---

[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/156#discussion_r147533090
  
    --- Diff: extensions/libarchive/ManipulateArchive.cpp ---
    @@ -0,0 +1,309 @@
    +/**
    + * @file ManipulateArchive.cpp
    + * ManipulateArchive class implementation
    + *
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +#include <string.h>
    +#include <iostream>
    +#include <fstream>
    +#include <memory>
    +#include <string>
    +#include <set>
    +
    +#include <boost/filesystem.hpp>
    +
    +#include <archive.h>
    +#include <archive_entry.h>
    +
    +#include "ManipulateArchive.h"
    +#include "core/ProcessContext.h"
    +#include "core/ProcessSession.h"
    +#include "core/FlowFile.h"
    +
    +namespace org {
    +namespace apache {
    +namespace nifi {
    +namespace minifi {
    +namespace processors {
    +
    +core::Property ManipulateArchive::Operation("Operation", "Operation to perform on the archive", "");
    +core::Property ManipulateArchive::Target("Target", "The path within the archive to perform the operation on", "");
    +core::Property ManipulateArchive::Destination("Destination", "Destination for operations (move or copy) which result in new entries", "");
    +core::Property ManipulateArchive::Before("Before", "For operations which result in new entries, places the new entry before the entry specified by this property", "");
    +core::Property ManipulateArchive::After("After", "For operations which result in new entries, places the new entry after the entry specified by this property", "");
    +core::Relationship ManipulateArchive::Success("success", "success operational on the flow record");
    +
    +char const* ManipulateArchive::OPERATION_REMOVE = "remove";
    +char const* ManipulateArchive::OPERATION_COPY =   "copy";
    +char const* ManipulateArchive::OPERATION_MOVE =   "move";
    +char const* ManipulateArchive::OPERATION_TOUCH =  "touch";
    +
    +void ManipulateArchive::initialize() {
    +    //! Set the supported properties
    +    std::set<core::Property> properties;
    +    properties.insert(Operation);
    +    properties.insert(Target);
    +    properties.insert(Destination);
    +    properties.insert(Before);
    +    properties.insert(After);
    +    setSupportedProperties(properties);
    +
    +    //! Set the supported relationships
    +    std::set<core::Relationship> relationships;
    +    relationships.insert(Success);
    +    setSupportedRelationships(relationships);
    +}
    +
    +void ManipulateArchive::onTrigger(core::ProcessContext *context, core::ProcessSession *session) {
    +    std::shared_ptr<core::FlowFile> flowFile = session->get();
    +
    +    if (!flowFile) {
    +        return;
    +    }
    +
    +    std::string operation;
    +    context->getProperty(Operation.getName(), operation);
    +
    +    std::string targetEntry;
    +    context->getProperty(Target.getName(), targetEntry);
    +
    +    std::string destination;
    +    context->getProperty(Destination.getName(), destination);
    +
    +    std::string before;
    +    context->getProperty(Before.getName(), before);
    +
    +    std::string after;
    +    context->getProperty(After.getName(), after);
    +
    +    // TODO(calebj) Validate properties
    +
    +    FocusArchiveEntry::ArchiveMetadata archiveMetadata;
    +    FocusArchiveEntry::ReadCallback readCallback(this, &archiveMetadata);
    +    session->read(flowFile, &readCallback);
    +
    +    logger_->log_info("ManipulateArchive performing operation %s on %s", operation.c_str(), targetEntry.c_str());
    +
    +    // Perform operation: REMOVE
    +    if (operation == OPERATION_REMOVE) {
    +        for (auto it = archiveMetadata.entryMetadata.begin(); it != archiveMetadata.entryMetadata.end();) {
    +            if ((*it).entryName == targetEntry) {
    +                logger_->log_info("ManipulateArchive found entry %s for removal", targetEntry.c_str());
    +                std::remove((*it).tmpFileName.c_str());
    +                it = archiveMetadata.entryMetadata.erase(it);
    +                break;
    +            } else {
    +                it++;
    +            }
    +        }
    +    }
    +
    +    // Perform operation: COPY
    +    if (operation == OPERATION_COPY) {
    +        bool found = false;
    +        FocusArchiveEntry::ArchiveEntryMetadata copy;
    +
    +        // Find item to copy
    +        for (auto it = archiveMetadata.entryMetadata.begin(); it != archiveMetadata.entryMetadata.end();) {
    --- End diff --
    
    Can clean this up with the range based loop, but that's not something that's absolutely necessary. 


---

[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/156#discussion_r147533391
  
    --- Diff: extensions/libarchive/ManipulateArchive.cpp ---
    @@ -0,0 +1,309 @@
    +/**
    + * @file ManipulateArchive.cpp
    + * ManipulateArchive class implementation
    + *
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +#include <string.h>
    +#include <iostream>
    +#include <fstream>
    +#include <memory>
    +#include <string>
    +#include <set>
    +
    +#include <boost/filesystem.hpp>
    +
    +#include <archive.h>
    +#include <archive_entry.h>
    +
    +#include "ManipulateArchive.h"
    +#include "core/ProcessContext.h"
    +#include "core/ProcessSession.h"
    +#include "core/FlowFile.h"
    +
    +namespace org {
    +namespace apache {
    +namespace nifi {
    +namespace minifi {
    +namespace processors {
    +
    +core::Property ManipulateArchive::Operation("Operation", "Operation to perform on the archive", "");
    +core::Property ManipulateArchive::Target("Target", "The path within the archive to perform the operation on", "");
    +core::Property ManipulateArchive::Destination("Destination", "Destination for operations (move or copy) which result in new entries", "");
    +core::Property ManipulateArchive::Before("Before", "For operations which result in new entries, places the new entry before the entry specified by this property", "");
    +core::Property ManipulateArchive::After("After", "For operations which result in new entries, places the new entry after the entry specified by this property", "");
    +core::Relationship ManipulateArchive::Success("success", "success operational on the flow record");
    +
    +char const* ManipulateArchive::OPERATION_REMOVE = "remove";
    +char const* ManipulateArchive::OPERATION_COPY =   "copy";
    +char const* ManipulateArchive::OPERATION_MOVE =   "move";
    +char const* ManipulateArchive::OPERATION_TOUCH =  "touch";
    +
    +void ManipulateArchive::initialize() {
    +    //! Set the supported properties
    +    std::set<core::Property> properties;
    +    properties.insert(Operation);
    +    properties.insert(Target);
    +    properties.insert(Destination);
    +    properties.insert(Before);
    +    properties.insert(After);
    +    setSupportedProperties(properties);
    +
    +    //! Set the supported relationships
    +    std::set<core::Relationship> relationships;
    +    relationships.insert(Success);
    +    setSupportedRelationships(relationships);
    +}
    +
    +void ManipulateArchive::onTrigger(core::ProcessContext *context, core::ProcessSession *session) {
    +    std::shared_ptr<core::FlowFile> flowFile = session->get();
    +
    +    if (!flowFile) {
    +        return;
    +    }
    +
    +    std::string operation;
    +    context->getProperty(Operation.getName(), operation);
    +
    +    std::string targetEntry;
    +    context->getProperty(Target.getName(), targetEntry);
    +
    +    std::string destination;
    +    context->getProperty(Destination.getName(), destination);
    +
    +    std::string before;
    +    context->getProperty(Before.getName(), before);
    +
    +    std::string after;
    +    context->getProperty(After.getName(), after);
    +
    +    // TODO(calebj) Validate properties
    +
    +    FocusArchiveEntry::ArchiveMetadata archiveMetadata;
    +    FocusArchiveEntry::ReadCallback readCallback(this, &archiveMetadata);
    +    session->read(flowFile, &readCallback);
    +
    +    logger_->log_info("ManipulateArchive performing operation %s on %s", operation.c_str(), targetEntry.c_str());
    +
    +    // Perform operation: REMOVE
    +    if (operation == OPERATION_REMOVE) {
    +        for (auto it = archiveMetadata.entryMetadata.begin(); it != archiveMetadata.entryMetadata.end();) {
    +            if ((*it).entryName == targetEntry) {
    +                logger_->log_info("ManipulateArchive found entry %s for removal", targetEntry.c_str());
    +                std::remove((*it).tmpFileName.c_str());
    --- End diff --
    
    In the event that volatile repositories are used in environments without persistent storage should we disable this suite of extensions? As I understand it, these operations can be performed in memory, right @minifirocks?
    
    Should we put make an option to exclude this and other processors for those without persistent storage?


---

[GitHub] nifi-minifi-cpp pull request #156: MINIFICPP-268 Implement ManipulateArchive...

Posted by calebj <gi...@git.apache.org>.
Github user calebj commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/156#discussion_r148094619
  
    --- Diff: CMakeLists.txt ---
    @@ -111,38 +111,38 @@ add_subdirectory(libminifi)
     #function(createExtension extensionCondition extensionGuard extensionName description dirName)
     
     ## Add http-curl extensions
    -createExtension(DISABLE_CURL 
    -				HTTP-CURL 
    -				"HTTP CURL" 
    -				"This enables RESTProtocol, InvokeHTTP, and the HTTPClient for Site to Site" 
    -				"extensions/http-curl"
    -				"${TEST_DIR}/curl-tests")
    +createExtension(DISABLE_CURL
    +                HTTP-CURL
    --- End diff --
    
    I don't follow. What do you need me to change?


---

[GitHub] nifi-minifi-cpp issue #156: MINIFICPP-268 Implement ManipulateArchive proces...

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on the issue:

    https://github.com/apache/nifi-minifi-cpp/pull/156
  
    @calebj I can make the changes to boost if you're fine with it when I merge it, unless you are doing that now. 


---