You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2020/04/01 00:45:17 UTC

[GitHub] [nifi-minifi-cpp] bakaid opened a new pull request #747: MINIFICPP-1036 - libarchive extension improvements

bakaid opened a new pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747
 
 
   This commit
    - transfers the libarchive compilation to an ExternalProject
    - adds bundled liblzma and bzip2
    - makes the libarchive extension compile on Windows
    - adds the "Encapsulate in TAR" Property to CompressContent
    - adds raw GZIP compression support to CompressContent
   
   Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [ ] Is there a JIRA ticket associated with this PR? Is it referenced
        in the commit message?
   
   - [ ] Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
   
   - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
   
   - [ ] Is your initial contribution a single, squashed commit?
   
   ### For code changes:
   - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the LICENSE file?
   - [ ] If applicable, have you updated the NOTICE file?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r404000232
 
 

 ##########
 File path: libminifi/include/io/ZlibStream.h
 ##########
 @@ -0,0 +1,105 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef LIBMINIFI_INCLUDE_IO_ZLIBSTREAM_H_
+#define LIBMINIFI_INCLUDE_IO_ZLIBSTREAM_H_
+#include <cstdint>
+#include <vector>
+
+#include <zlib.h>
+
+#include "BaseStream.h"
+#include "core/logging/LoggerConfiguration.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace io {
+
+enum class ZlibStreamState : uint8_t {
+  UNINITIALIZED,
+  INITIALIZED,
+  ERRORED,
+  FINISHED
+};
+
+class ZlibBaseStream : public BaseStream {
+ public:
+  ZlibBaseStream();
+  ZlibBaseStream(DataStream* other);
+
+  ZlibBaseStream(const ZlibBaseStream&) = delete;
+  ZlibBaseStream& operator=(const ZlibBaseStream&) = delete;
+  ZlibBaseStream(ZlibBaseStream&& other) = delete;
+  ZlibBaseStream& operator=(ZlibBaseStream&& other) = delete;
+
+  ~ZlibBaseStream() override = default;
+
+  virtual bool isFinished() const;
+
+ protected:
+  ZlibStreamState state_{ZlibStreamState::UNINITIALIZED};
+  z_stream strm_{};
+  std::vector<uint8_t> outputBuffer_;
+};
+
+class ZlibCompressStream : public ZlibBaseStream {
+ public:
+  ZlibCompressStream(bool gzip = true, int level = Z_DEFAULT_COMPRESSION);
+  ZlibCompressStream(DataStream* other, bool gzip = true, int level = Z_DEFAULT_COMPRESSION);
 
 Review comment:
   I could, although I don't really see the point in this case.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r403975877
 
 

 ##########
 File path: extensions/libarchive/CompressContent.cpp
 ##########
 @@ -39,6 +39,11 @@ core::Property CompressContent::CompressLevel("Compression Level", "The compress
 core::Property CompressContent::CompressMode("Mode", "Indicates whether the processor should compress content or decompress content.", MODE_COMPRESS);
 core::Property CompressContent::CompressFormat("Compression Format", "The compression format to use.", COMPRESSION_FORMAT_ATTRIBUTE);
 core::Property CompressContent::UpdateFileName("Update Filename", "Determines if filename extension need to be updated", "false");
+core::Property CompressContent::EncapsulateInTar("Encapsulate in TAR", "If true, on compression the FlowFile is added to a TAR archive and then compressed, "
 
 Review comment:
   I completely agree, but because all other Properties used the older API, I didn't want this to stand out.
   However, after using this processor through C2, I am convinced that converting all Properties to use validators in this processor will worth the (not significant) effort, so I'll do that.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r401559213
 
 

 ##########
 File path: libminifi/src/io/ZlibStream.cpp
 ##########
 @@ -0,0 +1,202 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "io/ZlibStream.h"
+#include "Exception.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace io {
+
+/* ZlibBaseStream */
+
+ZlibBaseStream::ZlibBaseStream()
+    : ZlibBaseStream(this) {
+}
+
+ZlibBaseStream::ZlibBaseStream(DataStream* other)
+    : BaseStream(other)
+    , outputBuffer_(16384U) {
+  strm_.zalloc = Z_NULL;
+  strm_.zfree = Z_NULL;
+  strm_.opaque = Z_NULL;
+}
+
+bool ZlibBaseStream::isFinished() const {
+  return state_ == ZlibStreamState::FINISHED;
+}
+
+/* ZlibCompressStream */
+
+ZlibCompressStream::ZlibCompressStream(bool gzip, int level)
+  : ZlibCompressStream(this, gzip, level) {
+}
+
+ZlibCompressStream::ZlibCompressStream(DataStream* other, bool gzip, int level)
+  : ZlibBaseStream(other)
+  , logger_(logging::LoggerFactory<ZlibCompressStream>::getLogger()) {
+  int ret = deflateInit2(
+      &strm_,
+      level,
+      Z_DEFLATED /* method */,
+      15 + (gzip ? 16 : 0) /* windowBits */,
+      8 /* memLevel */,
+      Z_DEFAULT_STRATEGY /* strategy */);
+  if (ret != Z_OK) {
+    logger_->log_error("Failed to initialize z_stream with deflateInit2, error code: %d", ret);
+    throw Exception(ExceptionType::GENERAL_EXCEPTION, "zlib deflateInit2 failed");
+  }
+
+  state_ = ZlibStreamState::INITIALIZED;
+}
+
+ZlibCompressStream::~ZlibCompressStream() {
+  if (state_ != ZlibStreamState::UNINITIALIZED) {
 
 Review comment:
   These ifs don't hurt at all, but given the ctor either trows or inits it properly, I can't image any unitialized scenario here.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r404616322
 
 

 ##########
 File path: libminifi/test/archive-tests/CompressContentTests.cpp
 ##########
 @@ -932,3 +938,107 @@ TEST_CASE("DecompressFileXYLZMA", "[compressfiletest8]") {
   }
 }
 
+TEST_CASE("RawGzipCompressionDecompression", "[compressfiletest8]") {
+  TestController testController;
+  LogTestController::getInstance().setTrace<org::apache::nifi::minifi::processors::CompressContent>();
+  LogTestController::getInstance().setTrace<org::apache::nifi::minifi::processors::PutFile>();
+
+  // Create temporary directories
+  char format_src[] = "/tmp/archives.XXXXXX";
+  std::string src_dir = testController.createTempDirectory(format_src);
+  REQUIRE(!src_dir.empty());
+
+  char format_dst[] = "/tmp/archived.XXXXXX";
+  std::string dst_dir = testController.createTempDirectory(format_dst);
+  REQUIRE(!dst_dir.empty());
+
+  // Define files
+  std::string src_file = utils::file::FileUtils::concat_path(src_dir, "src.txt");
+  std::string compressed_file = utils::file::FileUtils::concat_path(dst_dir, "src.txt.gz");
+  std::string decompressed_file = utils::file::FileUtils::concat_path(dst_dir, "src.txt");
+
+  // Build MiNiFi processing graph
+  auto plan = testController.createPlan();
+  auto get_file = plan->addProcessor(
+      "GetFile",
+      "GetFile");
+  auto compress_content = plan->addProcessor(
+      "CompressContent",
+      "CompressContent",
+      core::Relationship("success", "d"),
+      true);
+  auto put_compressed = plan->addProcessor(
+      "PutFile",
+      "PutFile",
+      core::Relationship("success", "d"),
+      true);
+  auto decompress_content = plan->addProcessor(
+      "CompressContent",
+      "CompressContent",
+      core::Relationship("success", "d"),
+      true);
+  auto put_decompressed = plan->addProcessor(
+      "PutFile",
+      "PutFile",
+      core::Relationship("success", "d"),
+      true);
+
+  // Configure GetFile processor
+  plan->setProperty(get_file, "Input Directory", src_dir);
+
+  // Configure CompressContent processor for compression
+  plan->setProperty(compress_content, "Mode", MODE_COMPRESS);
+  plan->setProperty(compress_content, "Compression Format", COMPRESSION_FORMAT_GZIP);
+  plan->setProperty(compress_content, "Update Filename", "true");
+  plan->setProperty(compress_content, "Encapsulate in TAR", "false");
+
+  // Configure first PutFile processor
+  plan->setProperty(put_compressed, "Directory", dst_dir);
+
+  // Configure CompressContent processor for decompression
+  plan->setProperty(decompress_content, "Mode", MODE_DECOMPRESS);
+  plan->setProperty(decompress_content, "Compression Format", COMPRESSION_FORMAT_GZIP);
+  plan->setProperty(decompress_content, "Update Filename", "true");
+  plan->setProperty(decompress_content, "Encapsulate in TAR", "false");
+
+  // Configure second PutFile processor
+  plan->setProperty(put_decompressed, "Directory", dst_dir);
+
+  // Create source file
+  std::string content;
+  SECTION("Empty content") {
+  }
+  SECTION("Short content") {
+    content = "Repeated repeated repeated repeated repeated stuff.";
+  }
+  SECTION("Long content") {
+    std::stringstream content_ss;
+    for (size_t i = 0U; i < 1024 * 1024U; i++) {
+      content_ss << "foobar";
+    }
+    content = content_ss.str();
+  }
+
+  std::fstream file;
+  file.open(src_file, std::ios::out);
+  file << content;
+  file.close();
 
 Review comment:
   Could be a oneliner: `std::ofstream{src_file} << content;`. Just an idea, I don't mind if it stays like this.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r401570806
 
 

 ##########
 File path: thirdparty/liblzma/liblzma.patch
 ##########
 @@ -0,0 +1,118 @@
+diff -rupN orig/CMakeLists.txt patched/CMakeLists.txt
+--- orig/CMakeLists.txt	1970-01-01 01:00:00.000000000 +0100
++++ patched/CMakeLists.txt	2020-03-31 21:52:39.000000000 +0200
+@@ -0,0 +1,114 @@
++cmake_minimum_required(VERSION 3.7)
++
++project(liblzma)
++
++set(SOURCES src/common/tuklib_cpucores.c
 
 Review comment:
   Do we patch to only compile the subset of files/features we need?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r401550668
 
 

 ##########
 File path: extensions/libarchive/CompressContent.h
 ##########
 @@ -341,6 +346,73 @@ class CompressContent: public core::Processor {
     }
   };
 
+  class GzipWriteCallback : public OutputStreamCallback {
+   public:
+    GzipWriteCallback(std::string compress_mode, int64_t compress_level, std::shared_ptr<core::FlowFile> flow, std::shared_ptr<core::ProcessSession> session)
+      : logger_(logging::LoggerFactory<CompressContent>::getLogger())
+      , compress_mode_(std::move(compress_mode))
+      , compress_level_(compress_level)
+      , flow_(std::move(flow))
+      , session_(std::move(session))
+    {
+    }
+
+    ~GzipWriteCallback() override = default;
 
 Review comment:
   Why? :)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r401925721
 
 

 ##########
 File path: extensions/libarchive/CompressContent.h
 ##########
 @@ -341,6 +346,73 @@ class CompressContent: public core::Processor {
     }
   };
 
+  class GzipWriteCallback : public OutputStreamCallback {
+   public:
+    GzipWriteCallback(std::string compress_mode, int64_t compress_level, std::shared_ptr<core::FlowFile> flow, std::shared_ptr<core::ProcessSession> session)
+      : logger_(logging::LoggerFactory<CompressContent>::getLogger())
+      , compress_mode_(std::move(compress_mode))
+      , compress_level_(compress_level)
+      , flow_(std::move(flow))
+      , session_(std::move(session))
+    {
+    }
+
+    ~GzipWriteCallback() override = default;
+
+    std::shared_ptr<logging::Logger> logger_;
+    std::string compress_mode_;
+    int64_t compress_level_;
+    std::shared_ptr<core::FlowFile> flow_;
+    std::shared_ptr<core::ProcessSession> session_;
+    bool success_{false};
+
+    int64_t process(std::shared_ptr<io::BaseStream> outputStream) override {
+      class ReadCallback : public InputStreamCallback {
+       public:
+        ReadCallback(GzipWriteCallback& writer, std::shared_ptr<io::BaseStream> outputStream)
+          : writer_(writer)
+          , outputStream_(std::move(outputStream)) {
+        }
+
+        int64_t process(std::shared_ptr<io::BaseStream> inputStream) override {
+          std::vector<uint8_t> buffer(16 * 1024U);
+          int64_t read_size = 0;
+          while (read_size < writer_.flow_->getSize()) {
+            int ret = inputStream->read(buffer.data(), buffer.size());
+            if (ret < 0) {
+              return -1;
+            } else if (ret == 0) {
+              break;
+            } else {
+              if (outputStream_->writeData(buffer.data(), ret) != ret) {
+                return -1;
+              }
+              read_size += ret;
+            }
+          }
+          outputStream_->closeStream();
 
 Review comment:
   When do we need to close a stream? We don't close `inputStream` and we only close `outputStream` on successful process. Shouldn't we always or never close `outputStream`?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r401549512
 
 

 ##########
 File path: extensions/libarchive/CompressContent.cpp
 ##########
 @@ -74,7 +80,13 @@ void CompressContent::onSchedule(core::ProcessContext *context, core::ProcessSes
   if (context->getProperty(UpdateFileName.getName(), value) && !value.empty()) {
     org::apache::nifi::minifi::utils::StringUtils::StringToBool(value, updateFileName_);
   }
-  logger_->log_info("Compress Content: Mode [%s] Format [%s] Level [%d] UpdateFileName [%d]", compressMode_, compressFormat_, compressLevel_, updateFileName_);
+  value = "";
+  if (context->getProperty(EncapsulateInTar.getName(), value) && !value.empty()) {
 
 Review comment:
   In case the property has a bool validator and a default value, you can skip the if and the string to bool conversion, just call the bool templated getter. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r404155844
 
 

 ##########
 File path: libminifi/include/io/ZlibStream.h
 ##########
 @@ -0,0 +1,105 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef LIBMINIFI_INCLUDE_IO_ZLIBSTREAM_H_
+#define LIBMINIFI_INCLUDE_IO_ZLIBSTREAM_H_
+#include <cstdint>
+#include <vector>
+
+#include <zlib.h>
+
+#include "BaseStream.h"
+#include "core/logging/LoggerConfiguration.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace io {
+
+enum class ZlibStreamState : uint8_t {
+  UNINITIALIZED,
+  INITIALIZED,
+  ERRORED,
+  FINISHED
+};
+
+class ZlibBaseStream : public BaseStream {
+ public:
+  ZlibBaseStream();
+  ZlibBaseStream(DataStream* other);
+
+  ZlibBaseStream(const ZlibBaseStream&) = delete;
+  ZlibBaseStream& operator=(const ZlibBaseStream&) = delete;
+  ZlibBaseStream(ZlibBaseStream&& other) = delete;
+  ZlibBaseStream& operator=(ZlibBaseStream&& other) = delete;
+
+  ~ZlibBaseStream() override = default;
+
+  virtual bool isFinished() const;
+
+ protected:
+  ZlibStreamState state_{ZlibStreamState::UNINITIALIZED};
+  z_stream strm_{};
+  std::vector<uint8_t> outputBuffer_;
+};
+
+class ZlibCompressStream : public ZlibBaseStream {
+ public:
+  ZlibCompressStream(bool gzip = true, int level = Z_DEFAULT_COMPRESSION);
+  ZlibCompressStream(DataStream* other, bool gzip = true, int level = Z_DEFAULT_COMPRESSION);
 
 Review comment:
   Fair, will mark them explicit.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r403997623
 
 

 ##########
 File path: libminifi/src/io/ZlibStream.cpp
 ##########
 @@ -0,0 +1,202 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "io/ZlibStream.h"
+#include "Exception.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace io {
+
+/* ZlibBaseStream */
+
+ZlibBaseStream::ZlibBaseStream()
+    : ZlibBaseStream(this) {
+}
+
+ZlibBaseStream::ZlibBaseStream(DataStream* other)
+    : BaseStream(other)
+    , outputBuffer_(16384U) {
+  strm_.zalloc = Z_NULL;
+  strm_.zfree = Z_NULL;
+  strm_.opaque = Z_NULL;
+}
+
+bool ZlibBaseStream::isFinished() const {
+  return state_ == ZlibStreamState::FINISHED;
+}
+
+/* ZlibCompressStream */
+
+ZlibCompressStream::ZlibCompressStream(bool gzip, int level)
+  : ZlibCompressStream(this, gzip, level) {
+}
+
+ZlibCompressStream::ZlibCompressStream(DataStream* other, bool gzip, int level)
+  : ZlibBaseStream(other)
+  , logger_(logging::LoggerFactory<ZlibCompressStream>::getLogger()) {
+  int ret = deflateInit2(
+      &strm_,
+      level,
+      Z_DEFLATED /* method */,
+      15 + (gzip ? 16 : 0) /* windowBits */,
+      8 /* memLevel */,
+      Z_DEFAULT_STRATEGY /* strategy */);
+  if (ret != Z_OK) {
+    logger_->log_error("Failed to initialize z_stream with deflateInit2, error code: %d", ret);
+    throw Exception(ExceptionType::GENERAL_EXCEPTION, "zlib deflateInit2 failed");
+  }
+
+  state_ = ZlibStreamState::INITIALIZED;
+}
+
+ZlibCompressStream::~ZlibCompressStream() {
+  if (state_ != ZlibStreamState::UNINITIALIZED) {
+    deflateEnd(&strm_);
+  }
+}
+
+int ZlibCompressStream::writeData(uint8_t* value, int size) {
+  if (state_ != ZlibStreamState::INITIALIZED) {
+    logger_->log_error("writeData called in invalid ZlibCompressStream state, state is %hhu", state_);
+    return -1;
+  }
+
+  strm_.next_in = value;
+  strm_.avail_in = size;
+
+  /*
+   * deflate consumes all input data it can (i.e. if it has enough output buffer it never leaves input data unconsumed)
+   * and fills the output buffer to the brim every time it can. This means that the proper way to use deflate is to
+   * watch avail_out: once deflate does not have to fill it to the brim, then it has consumed all data we have provided
+   * to it, and does not need more output buffer for the time being.
+   * When we have no more input data, we must call deflate with Z_FINISH to make it empty its internal buffers and
+   * close the compressed stream.
+   */
+  do {
+    logger_->log_trace("writeData has %u B of input data left", strm_.avail_in);
+
+    strm_.next_out = outputBuffer_.data();
+    strm_.avail_out = outputBuffer_.size();
+
+    int flush = value == nullptr ? Z_FINISH : Z_NO_FLUSH;
+    logger_->log_trace("calling deflate with flush %d", flush);
+
+    int ret = deflate(&strm_, flush);
+    if (ret == Z_STREAM_ERROR) {
+      logger_->log_error("deflate failed, error code: %d", ret);
+      state_ = ZlibStreamState::ERRORED;
+      return -1;
+    }
+    int output_size = outputBuffer_.size() - strm_.avail_out;
+    logger_->log_trace("deflate produced %d B of output data", output_size);
+    if (BaseStream::writeData(outputBuffer_.data(), output_size) != output_size) {
+      logger_->log_error("Failed to write to underlying stream");
+      state_ = ZlibStreamState::ERRORED;
+      return -1;
+    }
+  } while (strm_.avail_out == 0);
+
+  return size;
+}
+
+void ZlibCompressStream::closeStream() {
+  if (state_ == ZlibStreamState::INITIALIZED) {
+    if (writeData(nullptr, 0U) == 0) {
+      state_ = ZlibStreamState::FINISHED;
+    }
+  }
+}
+
+/* ZlibDecompressStream */
+
+ZlibDecompressStream::ZlibDecompressStream(bool gzip)
+  : ZlibDecompressStream(this, gzip) {
+}
+
+ZlibDecompressStream::ZlibDecompressStream(DataStream* other, bool gzip)
+    : ZlibBaseStream(other)
+    , logger_(logging::LoggerFactory<ZlibDecompressStream>::getLogger()) {
+  int ret = inflateInit2(&strm_, 15 + (gzip ? 16 : 0) /* windowBits */);
+  if (ret != Z_OK) {
+    logger_->log_error("Failed to initialize z_stream with inflateInit2, error code: %d", ret);
+    throw Exception(ExceptionType::GENERAL_EXCEPTION, "zlib inflateInit2 failed");
+  }
+
+  state_ = ZlibStreamState::INITIALIZED;
+}
+
+ZlibDecompressStream::~ZlibDecompressStream() {
+  if (state_ != ZlibStreamState::UNINITIALIZED) {
 
 Review comment:
   Would it make sense to make `state_ != UNINITIALIZED` a postcondition of the ctor and remove the UNINITIALIZED state? It seems to be already the case, so we could remove all possibility of related dead code by removing the enum value. We don't need to prepare for any usage of an object, including destruction, during object creation.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r404032200
 
 

 ##########
 File path: libminifi/test/unit/ZlibStreamTests.cpp
 ##########
 @@ -0,0 +1,105 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <utility>
+#include <string>
+#include <random>
+#include <algorithm>
+#include "../TestBase.h"
+#include "io/ZlibStream.h"
+#include "utils/StringUtils.h"
+
+namespace io = org::apache::nifi::minifi::io;
+
+TEST_CASE("gzip compression and decompression", "[basic]") {
+  /* Compression*/
+  io::ZlibCompressStream compressStream;
+
+  std::string original;
+  SECTION("Empty") {
+  }
+  SECTION("Simple content in one write") {
+    REQUIRE(-1 != compressStream.write(reinterpret_cast<uint8_t*>(const_cast<char*>("foobar")), strlen("foobar")));
+    original += "foobar";
+  }
+  SECTION("Simple content in two writes") {
+    REQUIRE(-1 != compressStream.write(reinterpret_cast<uint8_t*>(const_cast<char*>("foo")), strlen("foo")));
+    original += "foo";
+    REQUIRE(-1 != compressStream.write(reinterpret_cast<uint8_t*>(const_cast<char*>("bar")), strlen("bar")));
 
 Review comment:
   It must return the number of bytes consumed, will change to an exact check.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r401760087
 
 

 ##########
 File path: libminifi/src/io/ZlibStream.cpp
 ##########
 @@ -0,0 +1,202 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "io/ZlibStream.h"
+#include "Exception.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace io {
+
+/* ZlibBaseStream */
+
+ZlibBaseStream::ZlibBaseStream()
+    : ZlibBaseStream(this) {
+}
+
+ZlibBaseStream::ZlibBaseStream(DataStream* other)
+    : BaseStream(other)
+    , outputBuffer_(16384U) {
+  strm_.zalloc = Z_NULL;
+  strm_.zfree = Z_NULL;
+  strm_.opaque = Z_NULL;
+}
+
+bool ZlibBaseStream::isFinished() const {
+  return state_ == ZlibStreamState::FINISHED;
+}
+
+/* ZlibCompressStream */
+
+ZlibCompressStream::ZlibCompressStream(bool gzip, int level)
+  : ZlibCompressStream(this, gzip, level) {
+}
+
+ZlibCompressStream::ZlibCompressStream(DataStream* other, bool gzip, int level)
+  : ZlibBaseStream(other)
+  , logger_(logging::LoggerFactory<ZlibCompressStream>::getLogger()) {
+  int ret = deflateInit2(
+      &strm_,
+      level,
+      Z_DEFLATED /* method */,
+      15 + (gzip ? 16 : 0) /* windowBits */,
+      8 /* memLevel */,
+      Z_DEFAULT_STRATEGY /* strategy */);
+  if (ret != Z_OK) {
+    logger_->log_error("Failed to initialize z_stream with deflateInit2, error code: %d", ret);
+    throw Exception(ExceptionType::GENERAL_EXCEPTION, "zlib deflateInit2 failed");
+  }
+
+  state_ = ZlibStreamState::INITIALIZED;
+}
+
+ZlibCompressStream::~ZlibCompressStream() {
+  if (state_ != ZlibStreamState::UNINITIALIZED) {
+    deflateEnd(&strm_);
+  }
+}
+
+int ZlibCompressStream::writeData(uint8_t* value, int size) {
+  if (state_ != ZlibStreamState::INITIALIZED) {
+    logger_->log_error("writeData called in invalid ZlibCompressStream state, state is %hhu", state_);
+    return -1;
+  }
+
+  strm_.next_in = value;
+  strm_.avail_in = size;
+
+  /*
+   * deflate consumes all input data it can (i.e. if it has enough output buffer it never leaves input data unconsumed)
+   * and fills the output buffer to the brim every time it can. This means that the proper way to use deflate is to
+   * watch avail_out: once deflate does not have to fill it to the brim, then it has consumed all data we have provided
+   * to it, and does not need more output buffer for the time being.
+   * When we have no more input data, we must call deflate with Z_FINISH to make it empty its internal buffers and
+   * close the compressed stream.
+   */
+  do {
+    logger_->log_trace("writeData has %u B of input data left", strm_.avail_in);
+
+    strm_.next_out = outputBuffer_.data();
+    strm_.avail_out = outputBuffer_.size();
+
+    int flush = value == nullptr ? Z_FINISH : Z_NO_FLUSH;
+    logger_->log_trace("calling deflate with flush %d", flush);
+
+    int ret = deflate(&strm_, flush);
+    if (ret == Z_STREAM_ERROR) {
+      logger_->log_error("deflate failed, error code: %d", ret);
+      state_ = ZlibStreamState::ERRORED;
+      return -1;
+    }
+    int output_size = outputBuffer_.size() - strm_.avail_out;
+    logger_->log_trace("deflate produced %d B of output data", output_size);
+    if (BaseStream::writeData(outputBuffer_.data(), output_size) != output_size) {
+      logger_->log_error("Failed to write to underlying stream");
+      state_ = ZlibStreamState::ERRORED;
+      return -1;
+    }
+  } while (strm_.avail_out == 0);
+
+  return size;
+}
+
+void ZlibCompressStream::closeStream() {
+  if (state_ == ZlibStreamState::INITIALIZED) {
+    if (writeData(nullptr, 0U) == 0) {
+      state_ = ZlibStreamState::FINISHED;
+    }
+  }
+}
+
+/* ZlibDecompressStream */
+
+ZlibDecompressStream::ZlibDecompressStream(bool gzip)
+  : ZlibDecompressStream(this, gzip) {
+}
+
+ZlibDecompressStream::ZlibDecompressStream(DataStream* other, bool gzip)
+    : ZlibBaseStream(other)
+    , logger_(logging::LoggerFactory<ZlibDecompressStream>::getLogger()) {
+  int ret = inflateInit2(&strm_, 15 + (gzip ? 16 : 0) /* windowBits */);
+  if (ret != Z_OK) {
+    logger_->log_error("Failed to initialize z_stream with inflateInit2, error code: %d", ret);
+    throw Exception(ExceptionType::GENERAL_EXCEPTION, "zlib inflateInit2 failed");
+  }
+
+  state_ = ZlibStreamState::INITIALIZED;
+}
+
+ZlibDecompressStream::~ZlibDecompressStream() {
+  if (state_ != ZlibStreamState::UNINITIALIZED) {
 
 Review comment:
   inflateEnd frees the resources allocated by zlib, so if we called inflateInit and it succeeded we must call inflateEnd, even if there as an error.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r401591099
 
 

 ##########
 File path: thirdparty/bzip2/bzip2.patch
 ##########
 @@ -0,0 +1,31 @@
+diff -rupN orig/CMakeLists.txt patched/CMakeLists.txt
 
 Review comment:
   bzip2 only provides a Makefile which is not very usable on Windows, and it is a pretty simple project so I created a CMake file to compile it on all platforms. Will add a comment.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r401562109
 
 

 ##########
 File path: libminifi/src/io/ZlibStream.cpp
 ##########
 @@ -0,0 +1,202 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "io/ZlibStream.h"
+#include "Exception.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace io {
+
+/* ZlibBaseStream */
+
+ZlibBaseStream::ZlibBaseStream()
+    : ZlibBaseStream(this) {
+}
+
+ZlibBaseStream::ZlibBaseStream(DataStream* other)
+    : BaseStream(other)
+    , outputBuffer_(16384U) {
+  strm_.zalloc = Z_NULL;
+  strm_.zfree = Z_NULL;
+  strm_.opaque = Z_NULL;
+}
+
+bool ZlibBaseStream::isFinished() const {
+  return state_ == ZlibStreamState::FINISHED;
+}
+
+/* ZlibCompressStream */
+
+ZlibCompressStream::ZlibCompressStream(bool gzip, int level)
+  : ZlibCompressStream(this, gzip, level) {
+}
+
+ZlibCompressStream::ZlibCompressStream(DataStream* other, bool gzip, int level)
+  : ZlibBaseStream(other)
+  , logger_(logging::LoggerFactory<ZlibCompressStream>::getLogger()) {
+  int ret = deflateInit2(
+      &strm_,
+      level,
+      Z_DEFLATED /* method */,
+      15 + (gzip ? 16 : 0) /* windowBits */,
+      8 /* memLevel */,
+      Z_DEFAULT_STRATEGY /* strategy */);
+  if (ret != Z_OK) {
+    logger_->log_error("Failed to initialize z_stream with deflateInit2, error code: %d", ret);
+    throw Exception(ExceptionType::GENERAL_EXCEPTION, "zlib deflateInit2 failed");
+  }
+
+  state_ = ZlibStreamState::INITIALIZED;
+}
+
+ZlibCompressStream::~ZlibCompressStream() {
+  if (state_ != ZlibStreamState::UNINITIALIZED) {
+    deflateEnd(&strm_);
+  }
+}
+
+int ZlibCompressStream::writeData(uint8_t* value, int size) {
+  if (state_ != ZlibStreamState::INITIALIZED) {
+    logger_->log_error("writeData called in invalid ZlibCompressStream state, state is %hhu", state_);
+    return -1;
+  }
+
+  strm_.next_in = value;
+  strm_.avail_in = size;
+
+  /*
+   * deflate consumes all input data it can (i.e. if it has enough output buffer it never leaves input data unconsumed)
+   * and fills the output buffer to the brim every time it can. This means that the proper way to use deflate is to
+   * watch avail_out: once deflate does not have to fill it to the brim, then it has consumed all data we have provided
+   * to it, and does not need more output buffer for the time being.
+   * When we have no more input data, we must call deflate with Z_FINISH to make it empty its internal buffers and
+   * close the compressed stream.
+   */
+  do {
+    logger_->log_trace("writeData has %u B of input data left", strm_.avail_in);
+
+    strm_.next_out = outputBuffer_.data();
+    strm_.avail_out = outputBuffer_.size();
+
+    int flush = value == nullptr ? Z_FINISH : Z_NO_FLUSH;
+    logger_->log_trace("calling deflate with flush %d", flush);
+
+    int ret = deflate(&strm_, flush);
+    if (ret == Z_STREAM_ERROR) {
+      logger_->log_error("deflate failed, error code: %d", ret);
+      state_ = ZlibStreamState::ERRORED;
+      return -1;
+    }
+    int output_size = outputBuffer_.size() - strm_.avail_out;
+    logger_->log_trace("deflate produced %d B of output data", output_size);
+    if (BaseStream::writeData(outputBuffer_.data(), output_size) != output_size) {
+      logger_->log_error("Failed to write to underlying stream");
+      state_ = ZlibStreamState::ERRORED;
+      return -1;
+    }
+  } while (strm_.avail_out == 0);
+
+  return size;
+}
+
+void ZlibCompressStream::closeStream() {
+  if (state_ == ZlibStreamState::INITIALIZED) {
 
 Review comment:
   In case error happened during write and close is called, what happens? Or we don't expect it to happen at all?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r401570146
 
 

 ##########
 File path: thirdparty/bzip2/bzip2.patch
 ##########
 @@ -0,0 +1,31 @@
+diff -rupN orig/CMakeLists.txt patched/CMakeLists.txt
 
 Review comment:
   A justification for this file (in a comment or sg) would be nice. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r401565642
 
 

 ##########
 File path: libminifi/src/io/ZlibStream.cpp
 ##########
 @@ -0,0 +1,202 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "io/ZlibStream.h"
+#include "Exception.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace io {
+
+/* ZlibBaseStream */
+
+ZlibBaseStream::ZlibBaseStream()
+    : ZlibBaseStream(this) {
+}
+
+ZlibBaseStream::ZlibBaseStream(DataStream* other)
+    : BaseStream(other)
+    , outputBuffer_(16384U) {
+  strm_.zalloc = Z_NULL;
+  strm_.zfree = Z_NULL;
+  strm_.opaque = Z_NULL;
+}
+
+bool ZlibBaseStream::isFinished() const {
+  return state_ == ZlibStreamState::FINISHED;
+}
+
+/* ZlibCompressStream */
+
+ZlibCompressStream::ZlibCompressStream(bool gzip, int level)
+  : ZlibCompressStream(this, gzip, level) {
+}
+
+ZlibCompressStream::ZlibCompressStream(DataStream* other, bool gzip, int level)
+  : ZlibBaseStream(other)
+  , logger_(logging::LoggerFactory<ZlibCompressStream>::getLogger()) {
+  int ret = deflateInit2(
+      &strm_,
+      level,
+      Z_DEFLATED /* method */,
+      15 + (gzip ? 16 : 0) /* windowBits */,
+      8 /* memLevel */,
+      Z_DEFAULT_STRATEGY /* strategy */);
+  if (ret != Z_OK) {
+    logger_->log_error("Failed to initialize z_stream with deflateInit2, error code: %d", ret);
+    throw Exception(ExceptionType::GENERAL_EXCEPTION, "zlib deflateInit2 failed");
+  }
+
+  state_ = ZlibStreamState::INITIALIZED;
+}
+
+ZlibCompressStream::~ZlibCompressStream() {
+  if (state_ != ZlibStreamState::UNINITIALIZED) {
+    deflateEnd(&strm_);
+  }
+}
+
+int ZlibCompressStream::writeData(uint8_t* value, int size) {
+  if (state_ != ZlibStreamState::INITIALIZED) {
+    logger_->log_error("writeData called in invalid ZlibCompressStream state, state is %hhu", state_);
+    return -1;
+  }
+
+  strm_.next_in = value;
+  strm_.avail_in = size;
+
+  /*
+   * deflate consumes all input data it can (i.e. if it has enough output buffer it never leaves input data unconsumed)
+   * and fills the output buffer to the brim every time it can. This means that the proper way to use deflate is to
+   * watch avail_out: once deflate does not have to fill it to the brim, then it has consumed all data we have provided
+   * to it, and does not need more output buffer for the time being.
+   * When we have no more input data, we must call deflate with Z_FINISH to make it empty its internal buffers and
+   * close the compressed stream.
+   */
+  do {
+    logger_->log_trace("writeData has %u B of input data left", strm_.avail_in);
+
+    strm_.next_out = outputBuffer_.data();
+    strm_.avail_out = outputBuffer_.size();
+
+    int flush = value == nullptr ? Z_FINISH : Z_NO_FLUSH;
+    logger_->log_trace("calling deflate with flush %d", flush);
+
+    int ret = deflate(&strm_, flush);
+    if (ret == Z_STREAM_ERROR) {
+      logger_->log_error("deflate failed, error code: %d", ret);
+      state_ = ZlibStreamState::ERRORED;
+      return -1;
+    }
+    int output_size = outputBuffer_.size() - strm_.avail_out;
+    logger_->log_trace("deflate produced %d B of output data", output_size);
+    if (BaseStream::writeData(outputBuffer_.data(), output_size) != output_size) {
+      logger_->log_error("Failed to write to underlying stream");
+      state_ = ZlibStreamState::ERRORED;
+      return -1;
+    }
+  } while (strm_.avail_out == 0);
+
+  return size;
+}
+
+void ZlibCompressStream::closeStream() {
+  if (state_ == ZlibStreamState::INITIALIZED) {
+    if (writeData(nullptr, 0U) == 0) {
+      state_ = ZlibStreamState::FINISHED;
+    }
+  }
+}
+
+/* ZlibDecompressStream */
+
+ZlibDecompressStream::ZlibDecompressStream(bool gzip)
+  : ZlibDecompressStream(this, gzip) {
+}
+
+ZlibDecompressStream::ZlibDecompressStream(DataStream* other, bool gzip)
+    : ZlibBaseStream(other)
+    , logger_(logging::LoggerFactory<ZlibDecompressStream>::getLogger()) {
+  int ret = inflateInit2(&strm_, 15 + (gzip ? 16 : 0) /* windowBits */);
+  if (ret != Z_OK) {
+    logger_->log_error("Failed to initialize z_stream with inflateInit2, error code: %d", ret);
+    throw Exception(ExceptionType::GENERAL_EXCEPTION, "zlib inflateInit2 failed");
+  }
+
+  state_ = ZlibStreamState::INITIALIZED;
+}
+
+ZlibDecompressStream::~ZlibDecompressStream() {
+  if (state_ != ZlibStreamState::UNINITIALIZED) {
 
 Review comment:
   The same here, uninited state can't happen, although I wonder if inflateEnd should be called in case decompress failed and we have an error state.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] arpadboda closed pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
arpadboda closed pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r404032273
 
 

 ##########
 File path: libminifi/test/archive-tests/CompressContentTests.cpp
 ##########
 @@ -932,3 +938,107 @@ TEST_CASE("DecompressFileXYLZMA", "[compressfiletest8]") {
   }
 }
 
+TEST_CASE("RawGzipCompressionDecompression", "[compressfiletest8]") {
+  TestController testController;
+  LogTestController::getInstance().setTrace<org::apache::nifi::minifi::processors::CompressContent>();
+  LogTestController::getInstance().setTrace<org::apache::nifi::minifi::processors::PutFile>();
+
+  // Create temporary directories
+  char format_src[] = "/tmp/archives.XXXXXX";
+  std::string src_dir = testController.createTempDirectory(format_src);
+  REQUIRE(!src_dir.empty());
+
+  char format_dst[] = "/tmp/archived.XXXXXX";
+  std::string dst_dir = testController.createTempDirectory(format_dst);
+  REQUIRE(!dst_dir.empty());
+
+  // Define files
+  std::string src_file = utils::file::FileUtils::concat_path(src_dir, "src.txt");
+  std::string compressed_file = utils::file::FileUtils::concat_path(dst_dir, "src.txt.gz");
+  std::string decompressed_file = utils::file::FileUtils::concat_path(dst_dir, "src.txt");
+
+  // Build MiNiFi processing graph
+  auto plan = testController.createPlan();
+  auto get_file = plan->addProcessor(
+      "GetFile",
+      "GetFile");
+  auto compress_content = plan->addProcessor(
+      "CompressContent",
+      "CompressContent",
+      core::Relationship("success", "d"),
+      true);
+  auto put_compressed = plan->addProcessor(
+      "PutFile",
+      "PutFile",
+      core::Relationship("success", "d"),
+      true);
+  auto decompress_content = plan->addProcessor(
+      "CompressContent",
+      "CompressContent",
+      core::Relationship("success", "d"),
+      true);
+  auto put_decompressed = plan->addProcessor(
+      "PutFile",
+      "PutFile",
+      core::Relationship("success", "d"),
+      true);
+
+  // Configure GetFile processor
+  plan->setProperty(get_file, "Input Directory", src_dir);
+
+  // Configure CompressContent processor for compression
+  plan->setProperty(compress_content, "Mode", MODE_COMPRESS);
+  plan->setProperty(compress_content, "Compression Format", COMPRESSION_FORMAT_GZIP);
+  plan->setProperty(compress_content, "Update Filename", "true");
+  plan->setProperty(compress_content, "Encapsulate in TAR", "false");
+
+  // Configure first PutFile processor
+  plan->setProperty(put_compressed, "Directory", dst_dir);
+
+  // Configure CompressContent processor for decompression
+  plan->setProperty(decompress_content, "Mode", MODE_DECOMPRESS);
+  plan->setProperty(decompress_content, "Compression Format", COMPRESSION_FORMAT_GZIP);
+  plan->setProperty(decompress_content, "Update Filename", "true");
+  plan->setProperty(decompress_content, "Encapsulate in TAR", "false");
+
+  // Configure second PutFile processor
+  plan->setProperty(put_decompressed, "Directory", dst_dir);
+
+  // Create source file
+  std::string content;
+  SECTION("Empty content") {
+  }
+  SECTION("Short content") {
+    content = "Repeated repeated repeated repeated repeated stuff.";
+  }
+  SECTION("Long content") {
+    std::stringstream content_ss;
+    for (size_t i = 0U; i < 1024 * 1024U; i++) {
+      content_ss << "foobar";
+    }
+    content = content_ss.str();
+  }
+
+  std::fstream file;
+  file.open(src_file, std::ios::out);
+  file << content;
+  file.close();
+
+  // Run flow
+  testController.runSession(plan, true);
+
+  // Check compressed file
+  std::ifstream compressed(compressed_file, std::ios::in | std::ios::binary);
+  std::vector<uint8_t> compressed_content((std::istreambuf_iterator<char>(compressed)), std::istreambuf_iterator<char>());
+  REQUIRE(2 < compressed_content.size());
+  // gzip magic number
+  REQUIRE(0x1f == compressed_content[0]);
+  REQUIRE(0x8b == compressed_content[1]);
+
+  // Check decompressed file
+  std::ifstream decompressed(decompressed_file, std::ios::in | std::ios::binary);
+  std::string decompressed_content((std::istreambuf_iterator<char>(decompressed)), std::istreambuf_iterator<char>());
+  REQUIRE(content == decompressed_content);
 
 Review comment:
   Thanks!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r401591099
 
 

 ##########
 File path: thirdparty/bzip2/bzip2.patch
 ##########
 @@ -0,0 +1,31 @@
+diff -rupN orig/CMakeLists.txt patched/CMakeLists.txt
 
 Review comment:
   bzip2 only provides a Makefile which is not very usable on Windows, and it is a pretty simple project so I created a CMake file to compile it on all platforms.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r401759614
 
 

 ##########
 File path: libminifi/src/io/ZlibStream.cpp
 ##########
 @@ -0,0 +1,202 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "io/ZlibStream.h"
+#include "Exception.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace io {
+
+/* ZlibBaseStream */
+
+ZlibBaseStream::ZlibBaseStream()
+    : ZlibBaseStream(this) {
+}
+
+ZlibBaseStream::ZlibBaseStream(DataStream* other)
+    : BaseStream(other)
+    , outputBuffer_(16384U) {
+  strm_.zalloc = Z_NULL;
+  strm_.zfree = Z_NULL;
+  strm_.opaque = Z_NULL;
+}
+
+bool ZlibBaseStream::isFinished() const {
+  return state_ == ZlibStreamState::FINISHED;
+}
+
+/* ZlibCompressStream */
+
+ZlibCompressStream::ZlibCompressStream(bool gzip, int level)
+  : ZlibCompressStream(this, gzip, level) {
+}
+
+ZlibCompressStream::ZlibCompressStream(DataStream* other, bool gzip, int level)
+  : ZlibBaseStream(other)
+  , logger_(logging::LoggerFactory<ZlibCompressStream>::getLogger()) {
+  int ret = deflateInit2(
+      &strm_,
+      level,
+      Z_DEFLATED /* method */,
+      15 + (gzip ? 16 : 0) /* windowBits */,
+      8 /* memLevel */,
+      Z_DEFAULT_STRATEGY /* strategy */);
+  if (ret != Z_OK) {
+    logger_->log_error("Failed to initialize z_stream with deflateInit2, error code: %d", ret);
+    throw Exception(ExceptionType::GENERAL_EXCEPTION, "zlib deflateInit2 failed");
+  }
+
+  state_ = ZlibStreamState::INITIALIZED;
+}
+
+ZlibCompressStream::~ZlibCompressStream() {
+  if (state_ != ZlibStreamState::UNINITIALIZED) {
+    deflateEnd(&strm_);
+  }
+}
+
+int ZlibCompressStream::writeData(uint8_t* value, int size) {
+  if (state_ != ZlibStreamState::INITIALIZED) {
+    logger_->log_error("writeData called in invalid ZlibCompressStream state, state is %hhu", state_);
+    return -1;
+  }
+
+  strm_.next_in = value;
+  strm_.avail_in = size;
+
+  /*
+   * deflate consumes all input data it can (i.e. if it has enough output buffer it never leaves input data unconsumed)
+   * and fills the output buffer to the brim every time it can. This means that the proper way to use deflate is to
+   * watch avail_out: once deflate does not have to fill it to the brim, then it has consumed all data we have provided
+   * to it, and does not need more output buffer for the time being.
+   * When we have no more input data, we must call deflate with Z_FINISH to make it empty its internal buffers and
+   * close the compressed stream.
+   */
+  do {
+    logger_->log_trace("writeData has %u B of input data left", strm_.avail_in);
+
+    strm_.next_out = outputBuffer_.data();
+    strm_.avail_out = outputBuffer_.size();
+
+    int flush = value == nullptr ? Z_FINISH : Z_NO_FLUSH;
+    logger_->log_trace("calling deflate with flush %d", flush);
+
+    int ret = deflate(&strm_, flush);
+    if (ret == Z_STREAM_ERROR) {
+      logger_->log_error("deflate failed, error code: %d", ret);
+      state_ = ZlibStreamState::ERRORED;
+      return -1;
+    }
+    int output_size = outputBuffer_.size() - strm_.avail_out;
+    logger_->log_trace("deflate produced %d B of output data", output_size);
+    if (BaseStream::writeData(outputBuffer_.data(), output_size) != output_size) {
+      logger_->log_error("Failed to write to underlying stream");
+      state_ = ZlibStreamState::ERRORED;
+      return -1;
+    }
+  } while (strm_.avail_out == 0);
+
+  return size;
+}
+
+void ZlibCompressStream::closeStream() {
+  if (state_ == ZlibStreamState::INITIALIZED) {
 
 Review comment:
   It's perfectly okay to call `closeStream` if an error happens, and we will do nothing: we will keep the error state as it was.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r404213814
 
 

 ##########
 File path: libminifi/include/io/ZlibStream.h
 ##########
 @@ -0,0 +1,110 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef LIBMINIFI_INCLUDE_IO_ZLIBSTREAM_H_
+#define LIBMINIFI_INCLUDE_IO_ZLIBSTREAM_H_
+#include <cstdint>
+#include <vector>
+
+#include <zlib.h>
+
+#include "BaseStream.h"
+#include "core/logging/LoggerConfiguration.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace io {
+
+enum class ZlibCompressionFormat : uint8_t {
+  ZLIB,
+  GZIP
+};
+
+enum class ZlibStreamState : uint8_t {
+  UNINITIALIZED,
+  INITIALIZED,
+  ERRORED,
+  FINISHED
+};
+
+class ZlibBaseStream : public BaseStream {
+ public:
+  ZlibBaseStream();
+  ZlibBaseStream(DataStream* other);
+
+  ZlibBaseStream(const ZlibBaseStream&) = delete;
+  ZlibBaseStream& operator=(const ZlibBaseStream&) = delete;
+  ZlibBaseStream(ZlibBaseStream&& other) = delete;
+  ZlibBaseStream& operator=(ZlibBaseStream&& other) = delete;
 
 Review comment:
   Can we make these `protected`? It seems to me that this class was not meant to be instantiated by anything other than derived class ctors.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r404727720
 
 

 ##########
 File path: libminifi/test/archive-tests/CompressContentTests.cpp
 ##########
 @@ -932,3 +938,107 @@ TEST_CASE("DecompressFileXYLZMA", "[compressfiletest8]") {
   }
 }
 
+TEST_CASE("RawGzipCompressionDecompression", "[compressfiletest8]") {
+  TestController testController;
+  LogTestController::getInstance().setTrace<org::apache::nifi::minifi::processors::CompressContent>();
+  LogTestController::getInstance().setTrace<org::apache::nifi::minifi::processors::PutFile>();
+
+  // Create temporary directories
+  char format_src[] = "/tmp/archives.XXXXXX";
+  std::string src_dir = testController.createTempDirectory(format_src);
+  REQUIRE(!src_dir.empty());
+
+  char format_dst[] = "/tmp/archived.XXXXXX";
+  std::string dst_dir = testController.createTempDirectory(format_dst);
+  REQUIRE(!dst_dir.empty());
+
+  // Define files
+  std::string src_file = utils::file::FileUtils::concat_path(src_dir, "src.txt");
+  std::string compressed_file = utils::file::FileUtils::concat_path(dst_dir, "src.txt.gz");
+  std::string decompressed_file = utils::file::FileUtils::concat_path(dst_dir, "src.txt");
+
+  // Build MiNiFi processing graph
+  auto plan = testController.createPlan();
+  auto get_file = plan->addProcessor(
+      "GetFile",
+      "GetFile");
+  auto compress_content = plan->addProcessor(
+      "CompressContent",
+      "CompressContent",
+      core::Relationship("success", "d"),
+      true);
+  auto put_compressed = plan->addProcessor(
+      "PutFile",
+      "PutFile",
+      core::Relationship("success", "d"),
+      true);
+  auto decompress_content = plan->addProcessor(
+      "CompressContent",
+      "CompressContent",
+      core::Relationship("success", "d"),
+      true);
+  auto put_decompressed = plan->addProcessor(
+      "PutFile",
+      "PutFile",
+      core::Relationship("success", "d"),
+      true);
+
+  // Configure GetFile processor
+  plan->setProperty(get_file, "Input Directory", src_dir);
+
+  // Configure CompressContent processor for compression
+  plan->setProperty(compress_content, "Mode", MODE_COMPRESS);
+  plan->setProperty(compress_content, "Compression Format", COMPRESSION_FORMAT_GZIP);
+  plan->setProperty(compress_content, "Update Filename", "true");
+  plan->setProperty(compress_content, "Encapsulate in TAR", "false");
+
+  // Configure first PutFile processor
+  plan->setProperty(put_compressed, "Directory", dst_dir);
+
+  // Configure CompressContent processor for decompression
+  plan->setProperty(decompress_content, "Mode", MODE_DECOMPRESS);
+  plan->setProperty(decompress_content, "Compression Format", COMPRESSION_FORMAT_GZIP);
+  plan->setProperty(decompress_content, "Update Filename", "true");
+  plan->setProperty(decompress_content, "Encapsulate in TAR", "false");
+
+  // Configure second PutFile processor
+  plan->setProperty(put_decompressed, "Directory", dst_dir);
+
+  // Create source file
+  std::string content;
+  SECTION("Empty content") {
+  }
+  SECTION("Short content") {
+    content = "Repeated repeated repeated repeated repeated stuff.";
+  }
+  SECTION("Long content") {
+    std::stringstream content_ss;
+    for (size_t i = 0U; i < 1024 * 1024U; i++) {
+      content_ss << "foobar";
+    }
+    content = content_ss.str();
+  }
+
+  std::fstream file;
+  file.open(src_file, std::ios::out);
+  file << content;
+  file.close();
 
 Review comment:
   This is a test, I copied this from somewhere, I could not really be bothered whether its one line or four.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r404291902
 
 

 ##########
 File path: extensions/libarchive/ArchiveCommon.h
 ##########
 @@ -0,0 +1,29 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef EXTENSIONS_LIBARCHIVE_ARCHIVECOMMON_H_
+#define EXTENSIONS_LIBARCHIVE_ARCHIVECOMMON_H_
+
+#include <stdint.h>
+
+#ifdef WIN32
+#define mode_t uint16_t
+#define uid_t int64_t
+#define gid_t int64_t
 
 Review comment:
   Not sure, I think I was distracted by some third party header mess when doing this, typedef is the nice solution here, you are right. Changed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on issue #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on issue #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#issuecomment-610055836
 
 
   @arpadboda @szaszm Addressed review comments, it also builds for me on Travis now (on Xenial for some unfathomable reason some constexpr string literals in the sftp extension started to get ODR-used and therefore lacked storage and build failed with a link error - even though I haven't even touched the extension. Anyway, added storage for them in the last commit and it builds properly now.)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r401569407
 
 

 ##########
 File path: libminifi/test/unit/ZlibStreamTests.cpp
 ##########
 @@ -0,0 +1,105 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <utility>
+#include <string>
+#include <random>
+#include <algorithm>
+#include "../TestBase.h"
+#include "io/ZlibStream.h"
+#include "utils/StringUtils.h"
+
+namespace io = org::apache::nifi::minifi::io;
+
+TEST_CASE("gzip compression and decompression", "[basic]") {
+  /* Compression*/
+  io::ZlibCompressStream compressStream;
+
+  std::string original;
+  SECTION("Empty") {
+  }
+  SECTION("Simple content in one write") {
+    REQUIRE(-1 != compressStream.write(reinterpret_cast<uint8_t*>(const_cast<char*>("foobar")), strlen("foobar")));
+    original += "foobar";
+  }
+  SECTION("Simple content in two writes") {
+    REQUIRE(-1 != compressStream.write(reinterpret_cast<uint8_t*>(const_cast<char*>("foo")), strlen("foo")));
+    original += "foo";
+    REQUIRE(-1 != compressStream.write(reinterpret_cast<uint8_t*>(const_cast<char*>("bar")), strlen("bar")));
 
 Review comment:
   I think the return values should match the length of the string. 
   Or in case they return the number of compressed bytes written, let's check for grearter than zero. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r401555676
 
 

 ##########
 File path: libminifi/include/io/ZlibStream.h
 ##########
 @@ -0,0 +1,105 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef LIBMINIFI_INCLUDE_IO_ZLIBSTREAM_H_
+#define LIBMINIFI_INCLUDE_IO_ZLIBSTREAM_H_
+#include <cstdint>
+#include <vector>
+
+#include <zlib.h>
+
+#include "BaseStream.h"
+#include "core/logging/LoggerConfiguration.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace io {
+
+enum class ZlibStreamState : uint8_t {
+  UNINITIALIZED,
+  INITIALIZED,
+  ERRORED,
+  FINISHED
+};
+
+class ZlibBaseStream : public BaseStream {
+ public:
+  ZlibBaseStream();
+  ZlibBaseStream(DataStream* other);
+
+  ZlibBaseStream(const ZlibBaseStream&) = delete;
+  ZlibBaseStream& operator=(const ZlibBaseStream&) = delete;
+  ZlibBaseStream(ZlibBaseStream&& other) = delete;
+  ZlibBaseStream& operator=(ZlibBaseStream&& other) = delete;
+
+  ~ZlibBaseStream() override = default;
+
+  virtual bool isFinished() const;
+
+ protected:
+  ZlibStreamState state_{ZlibStreamState::UNINITIALIZED};
+  z_stream strm_{};
+  std::vector<uint8_t> outputBuffer_;
+};
+
+class ZlibCompressStream : public ZlibBaseStream {
+ public:
+  ZlibCompressStream(bool gzip = true, int level = Z_DEFAULT_COMPRESSION);
+  ZlibCompressStream(DataStream* other, bool gzip = true, int level = Z_DEFAULT_COMPRESSION);
 
 Review comment:
   These frighten me a bit as everything has a default value and ptr can be converted to bool implicitly, but not sure how it can be made better. Moving the stream to the end with an optional nullptr value?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r401760087
 
 

 ##########
 File path: libminifi/src/io/ZlibStream.cpp
 ##########
 @@ -0,0 +1,202 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "io/ZlibStream.h"
+#include "Exception.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace io {
+
+/* ZlibBaseStream */
+
+ZlibBaseStream::ZlibBaseStream()
+    : ZlibBaseStream(this) {
+}
+
+ZlibBaseStream::ZlibBaseStream(DataStream* other)
+    : BaseStream(other)
+    , outputBuffer_(16384U) {
+  strm_.zalloc = Z_NULL;
+  strm_.zfree = Z_NULL;
+  strm_.opaque = Z_NULL;
+}
+
+bool ZlibBaseStream::isFinished() const {
+  return state_ == ZlibStreamState::FINISHED;
+}
+
+/* ZlibCompressStream */
+
+ZlibCompressStream::ZlibCompressStream(bool gzip, int level)
+  : ZlibCompressStream(this, gzip, level) {
+}
+
+ZlibCompressStream::ZlibCompressStream(DataStream* other, bool gzip, int level)
+  : ZlibBaseStream(other)
+  , logger_(logging::LoggerFactory<ZlibCompressStream>::getLogger()) {
+  int ret = deflateInit2(
+      &strm_,
+      level,
+      Z_DEFLATED /* method */,
+      15 + (gzip ? 16 : 0) /* windowBits */,
+      8 /* memLevel */,
+      Z_DEFAULT_STRATEGY /* strategy */);
+  if (ret != Z_OK) {
+    logger_->log_error("Failed to initialize z_stream with deflateInit2, error code: %d", ret);
+    throw Exception(ExceptionType::GENERAL_EXCEPTION, "zlib deflateInit2 failed");
+  }
+
+  state_ = ZlibStreamState::INITIALIZED;
+}
+
+ZlibCompressStream::~ZlibCompressStream() {
+  if (state_ != ZlibStreamState::UNINITIALIZED) {
+    deflateEnd(&strm_);
+  }
+}
+
+int ZlibCompressStream::writeData(uint8_t* value, int size) {
+  if (state_ != ZlibStreamState::INITIALIZED) {
+    logger_->log_error("writeData called in invalid ZlibCompressStream state, state is %hhu", state_);
+    return -1;
+  }
+
+  strm_.next_in = value;
+  strm_.avail_in = size;
+
+  /*
+   * deflate consumes all input data it can (i.e. if it has enough output buffer it never leaves input data unconsumed)
+   * and fills the output buffer to the brim every time it can. This means that the proper way to use deflate is to
+   * watch avail_out: once deflate does not have to fill it to the brim, then it has consumed all data we have provided
+   * to it, and does not need more output buffer for the time being.
+   * When we have no more input data, we must call deflate with Z_FINISH to make it empty its internal buffers and
+   * close the compressed stream.
+   */
+  do {
+    logger_->log_trace("writeData has %u B of input data left", strm_.avail_in);
+
+    strm_.next_out = outputBuffer_.data();
+    strm_.avail_out = outputBuffer_.size();
+
+    int flush = value == nullptr ? Z_FINISH : Z_NO_FLUSH;
+    logger_->log_trace("calling deflate with flush %d", flush);
+
+    int ret = deflate(&strm_, flush);
+    if (ret == Z_STREAM_ERROR) {
+      logger_->log_error("deflate failed, error code: %d", ret);
+      state_ = ZlibStreamState::ERRORED;
+      return -1;
+    }
+    int output_size = outputBuffer_.size() - strm_.avail_out;
+    logger_->log_trace("deflate produced %d B of output data", output_size);
+    if (BaseStream::writeData(outputBuffer_.data(), output_size) != output_size) {
+      logger_->log_error("Failed to write to underlying stream");
+      state_ = ZlibStreamState::ERRORED;
+      return -1;
+    }
+  } while (strm_.avail_out == 0);
+
+  return size;
+}
+
+void ZlibCompressStream::closeStream() {
+  if (state_ == ZlibStreamState::INITIALIZED) {
+    if (writeData(nullptr, 0U) == 0) {
+      state_ = ZlibStreamState::FINISHED;
+    }
+  }
+}
+
+/* ZlibDecompressStream */
+
+ZlibDecompressStream::ZlibDecompressStream(bool gzip)
+  : ZlibDecompressStream(this, gzip) {
+}
+
+ZlibDecompressStream::ZlibDecompressStream(DataStream* other, bool gzip)
+    : ZlibBaseStream(other)
+    , logger_(logging::LoggerFactory<ZlibDecompressStream>::getLogger()) {
+  int ret = inflateInit2(&strm_, 15 + (gzip ? 16 : 0) /* windowBits */);
+  if (ret != Z_OK) {
+    logger_->log_error("Failed to initialize z_stream with inflateInit2, error code: %d", ret);
+    throw Exception(ExceptionType::GENERAL_EXCEPTION, "zlib inflateInit2 failed");
+  }
+
+  state_ = ZlibStreamState::INITIALIZED;
+}
+
+ZlibDecompressStream::~ZlibDecompressStream() {
+  if (state_ != ZlibStreamState::UNINITIALIZED) {
 
 Review comment:
   inflateEnd frees the resources allocated by zlib, so if we called inflateInit and it succeeded we must call inflateEnd, even if there was an error.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r404727823
 
 

 ##########
 File path: libminifi/include/io/ZlibStream.h
 ##########
 @@ -0,0 +1,110 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef LIBMINIFI_INCLUDE_IO_ZLIBSTREAM_H_
+#define LIBMINIFI_INCLUDE_IO_ZLIBSTREAM_H_
+#include <cstdint>
+#include <vector>
+
+#include <zlib.h>
+
+#include "BaseStream.h"
+#include "core/logging/LoggerConfiguration.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace io {
+
+enum class ZlibCompressionFormat : uint8_t {
+  ZLIB,
+  GZIP
+};
+
+enum class ZlibStreamState : uint8_t {
+  UNINITIALIZED,
+  INITIALIZED,
+  ERRORED,
+  FINISHED
+};
+
+class ZlibBaseStream : public BaseStream {
+ public:
+  ZlibBaseStream();
+  ZlibBaseStream(DataStream* other);
+
+  ZlibBaseStream(const ZlibBaseStream&) = delete;
+  ZlibBaseStream& operator=(const ZlibBaseStream&) = delete;
+  ZlibBaseStream(ZlibBaseStream&& other) = delete;
+  ZlibBaseStream& operator=(ZlibBaseStream&& other) = delete;
+
+  ~ZlibBaseStream() override = default;
 
 Review comment:
   Done.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r404032926
 
 

 ##########
 File path: extensions/libarchive/CompressContent.h
 ##########
 @@ -341,6 +346,73 @@ class CompressContent: public core::Processor {
     }
   };
 
+  class GzipWriteCallback : public OutputStreamCallback {
+   public:
+    GzipWriteCallback(std::string compress_mode, int64_t compress_level, std::shared_ptr<core::FlowFile> flow, std::shared_ptr<core::ProcessSession> session)
+      : logger_(logging::LoggerFactory<CompressContent>::getLogger())
+      , compress_mode_(std::move(compress_mode))
+      , compress_level_(compress_level)
+      , flow_(std::move(flow))
+      , session_(std::move(session))
+    {
+    }
+
+    ~GzipWriteCallback() override = default;
 
 Review comment:
   Force of habit, removed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r404087933
 
 

 ##########
 File path: libminifi/include/io/ZlibStream.h
 ##########
 @@ -0,0 +1,110 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef LIBMINIFI_INCLUDE_IO_ZLIBSTREAM_H_
+#define LIBMINIFI_INCLUDE_IO_ZLIBSTREAM_H_
+#include <cstdint>
+#include <vector>
+
+#include <zlib.h>
+
+#include "BaseStream.h"
+#include "core/logging/LoggerConfiguration.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace io {
+
+enum class ZlibCompressionFormat : uint8_t {
+  ZLIB,
+  GZIP
+};
+
+enum class ZlibStreamState : uint8_t {
+  UNINITIALIZED,
+  INITIALIZED,
+  ERRORED,
+  FINISHED
+};
+
+class ZlibBaseStream : public BaseStream {
+ public:
+  ZlibBaseStream();
+  ZlibBaseStream(DataStream* other);
+
+  ZlibBaseStream(const ZlibBaseStream&) = delete;
+  ZlibBaseStream& operator=(const ZlibBaseStream&) = delete;
+  ZlibBaseStream(ZlibBaseStream&& other) = delete;
+  ZlibBaseStream& operator=(ZlibBaseStream&& other) = delete;
+
+  ~ZlibBaseStream() override = default;
 
 Review comment:
   The one parameter ctor could be marked `explicit` and the dtor line is redundant.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r404879261
 
 

 ##########
 File path: extensions/libarchive/CompressContent.cpp
 ##########
 @@ -35,10 +35,32 @@ namespace nifi {
 namespace minifi {
 namespace processors {
 
-core::Property CompressContent::CompressLevel("Compression Level", "The compression level to use; this is valid only when using GZIP compression.", "1");
-core::Property CompressContent::CompressMode("Mode", "Indicates whether the processor should compress content or decompress content.", MODE_COMPRESS);
-core::Property CompressContent::CompressFormat("Compression Format", "The compression format to use.", COMPRESSION_FORMAT_ATTRIBUTE);
-core::Property CompressContent::UpdateFileName("Update Filename", "Determines if filename extension need to be updated", "false");
+core::Property CompressContent::CompressLevel(
+    core::PropertyBuilder::createProperty("Compression Level")->withDescription("The compression level to use; this is valid only when using GZIP compression.")
+        ->isRequired(false)->withDefaultValue<int>(1)->build());
+core::Property CompressContent::CompressMode(
+    core::PropertyBuilder::createProperty("Mode")->withDescription("Indicates whether the processor should compress content or decompress content.")
+        ->isRequired(false)->withAllowableValues<std::string>({MODE_COMPRESS, MODE_DECOMPRESS})->withDefaultValue(MODE_COMPRESS)->build());
+core::Property CompressContent::CompressFormat(
+    core::PropertyBuilder::createProperty("Compression Format")->withDescription("The compression format to use.")
+        ->isRequired(false)
+        ->withAllowableValues<std::string>({
+          COMPRESSION_FORMAT_ATTRIBUTE,
+          COMPRESSION_FORMAT_GZIP,
+          COMPRESSION_FORMAT_BZIP2,
+          COMPRESSION_FORMAT_XZ_LZMA2,
 
 Review comment:
   In that case, I was wrong.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r404007223
 
 

 ##########
 File path: libminifi/include/io/ZlibStream.h
 ##########
 @@ -0,0 +1,105 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef LIBMINIFI_INCLUDE_IO_ZLIBSTREAM_H_
+#define LIBMINIFI_INCLUDE_IO_ZLIBSTREAM_H_
+#include <cstdint>
+#include <vector>
+
+#include <zlib.h>
+
+#include "BaseStream.h"
+#include "core/logging/LoggerConfiguration.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace io {
+
+enum class ZlibStreamState : uint8_t {
+  UNINITIALIZED,
+  INITIALIZED,
+  ERRORED,
+  FINISHED
+};
+
+class ZlibBaseStream : public BaseStream {
+ public:
+  ZlibBaseStream();
+  ZlibBaseStream(DataStream* other);
+
+  ZlibBaseStream(const ZlibBaseStream&) = delete;
+  ZlibBaseStream& operator=(const ZlibBaseStream&) = delete;
+  ZlibBaseStream(ZlibBaseStream&& other) = delete;
+  ZlibBaseStream& operator=(ZlibBaseStream&& other) = delete;
+
+  ~ZlibBaseStream() override = default;
+
+  virtual bool isFinished() const;
+
+ protected:
+  ZlibStreamState state_{ZlibStreamState::UNINITIALIZED};
+  z_stream strm_{};
+  std::vector<uint8_t> outputBuffer_;
+};
+
+class ZlibCompressStream : public ZlibBaseStream {
+ public:
+  ZlibCompressStream(bool gzip = true, int level = Z_DEFAULT_COMPRESSION);
+  ZlibCompressStream(DataStream* other, bool gzip = true, int level = Z_DEFAULT_COMPRESSION);
 
 Review comment:
   To prevent implicit conversion from `bool`/`ZlibFormat` and `DataStream*`, as suggested in [C.46: By default, declare single-argument constructors explicit](http://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#Rc-explicit). (The guideline doesn't, but the reason covers all constructors that can be called with a single argument, i.e. including ctors with more parameters where all N>1-th parameter has a default argument.)
   
   Or if you intended the implicit conversion to be part of the API, I'm also fine with an explanation and possibly an example of intended usage added in a code comment.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r404726572
 
 

 ##########
 File path: extensions/libarchive/CompressContent.cpp
 ##########
 @@ -35,10 +35,32 @@ namespace nifi {
 namespace minifi {
 namespace processors {
 
-core::Property CompressContent::CompressLevel("Compression Level", "The compression level to use; this is valid only when using GZIP compression.", "1");
-core::Property CompressContent::CompressMode("Mode", "Indicates whether the processor should compress content or decompress content.", MODE_COMPRESS);
-core::Property CompressContent::CompressFormat("Compression Format", "The compression format to use.", COMPRESSION_FORMAT_ATTRIBUTE);
-core::Property CompressContent::UpdateFileName("Update Filename", "Determines if filename extension need to be updated", "false");
+core::Property CompressContent::CompressLevel(
+    core::PropertyBuilder::createProperty("Compression Level")->withDescription("The compression level to use; this is valid only when using GZIP compression.")
+        ->isRequired(false)->withDefaultValue<int>(1)->build());
+core::Property CompressContent::CompressMode(
+    core::PropertyBuilder::createProperty("Mode")->withDescription("Indicates whether the processor should compress content or decompress content.")
+        ->isRequired(false)->withAllowableValues<std::string>({MODE_COMPRESS, MODE_DECOMPRESS})->withDefaultValue(MODE_COMPRESS)->build());
+core::Property CompressContent::CompressFormat(
+    core::PropertyBuilder::createProperty("Compression Format")->withDescription("The compression format to use.")
+        ->isRequired(false)
+        ->withAllowableValues<std::string>({
+          COMPRESSION_FORMAT_ATTRIBUTE,
+          COMPRESSION_FORMAT_GZIP,
+          COMPRESSION_FORMAT_BZIP2,
+          COMPRESSION_FORMAT_XZ_LZMA2,
 
 Review comment:
   Could you elaborate? Any YAML config that didn't use values from this enum was erroneous and resulted in an error on onTrigger. How would this change break those configurations further?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r403987386
 
 

 ##########
 File path: libminifi/include/io/ZlibStream.h
 ##########
 @@ -0,0 +1,105 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef LIBMINIFI_INCLUDE_IO_ZLIBSTREAM_H_
+#define LIBMINIFI_INCLUDE_IO_ZLIBSTREAM_H_
+#include <cstdint>
+#include <vector>
+
+#include <zlib.h>
+
+#include "BaseStream.h"
+#include "core/logging/LoggerConfiguration.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace io {
+
+enum class ZlibStreamState : uint8_t {
+  UNINITIALIZED,
+  INITIALIZED,
+  ERRORED,
+  FINISHED
+};
+
+class ZlibBaseStream : public BaseStream {
+ public:
+  ZlibBaseStream();
+  ZlibBaseStream(DataStream* other);
+
+  ZlibBaseStream(const ZlibBaseStream&) = delete;
+  ZlibBaseStream& operator=(const ZlibBaseStream&) = delete;
+  ZlibBaseStream(ZlibBaseStream&& other) = delete;
+  ZlibBaseStream& operator=(ZlibBaseStream&& other) = delete;
+
+  ~ZlibBaseStream() override = default;
+
+  virtual bool isFinished() const;
+
+ protected:
+  ZlibStreamState state_{ZlibStreamState::UNINITIALIZED};
+  z_stream strm_{};
+  std::vector<uint8_t> outputBuffer_;
+};
+
+class ZlibCompressStream : public ZlibBaseStream {
+ public:
+  ZlibCompressStream(bool gzip = true, int level = Z_DEFAULT_COMPRESSION);
+  ZlibCompressStream(DataStream* other, bool gzip = true, int level = Z_DEFAULT_COMPRESSION);
 
 Review comment:
   Fair point. I'll replace the `bool gzip` with an enum class to the effect of `ZlibFormat format`. A pointer is not converted implicitly to an enum class, and it will also improve readability.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r401742645
 
 

 ##########
 File path: extensions/libarchive/ArchiveCommon.h
 ##########
 @@ -0,0 +1,29 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef EXTENSIONS_LIBARCHIVE_ARCHIVECOMMON_H_
+#define EXTENSIONS_LIBARCHIVE_ARCHIVECOMMON_H_
+
+#include <stdint.h>
+
+#ifdef WIN32
+#define mode_t uint16_t
+#define uid_t int64_t
+#define gid_t int64_t
 
 Review comment:
   why not `using` or `typedef` if it has to be C compatible?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r401590545
 
 

 ##########
 File path: thirdparty/liblzma/liblzma.patch
 ##########
 @@ -0,0 +1,118 @@
+diff -rupN orig/CMakeLists.txt patched/CMakeLists.txt
+--- orig/CMakeLists.txt	1970-01-01 01:00:00.000000000 +0100
++++ patched/CMakeLists.txt	2020-03-31 21:52:39.000000000 +0200
+@@ -0,0 +1,114 @@
++cmake_minimum_required(VERSION 3.7)
++
++project(liblzma)
++
++set(SOURCES src/common/tuklib_cpucores.c
 
 Review comment:
   We can't run the configure script on Windows, and the only thing lzma provides for Windows are msbuild files which do not integrate well into our CMake build system, so I wrote a windows-specific CMakeLists.txt for the project.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r401591739
 
 

 ##########
 File path: libminifi/src/io/ZlibStream.cpp
 ##########
 @@ -0,0 +1,202 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "io/ZlibStream.h"
+#include "Exception.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace io {
+
+/* ZlibBaseStream */
+
+ZlibBaseStream::ZlibBaseStream()
+    : ZlibBaseStream(this) {
+}
+
+ZlibBaseStream::ZlibBaseStream(DataStream* other)
+    : BaseStream(other)
+    , outputBuffer_(16384U) {
+  strm_.zalloc = Z_NULL;
+  strm_.zfree = Z_NULL;
+  strm_.opaque = Z_NULL;
+}
+
+bool ZlibBaseStream::isFinished() const {
+  return state_ == ZlibStreamState::FINISHED;
+}
+
+/* ZlibCompressStream */
+
+ZlibCompressStream::ZlibCompressStream(bool gzip, int level)
+  : ZlibCompressStream(this, gzip, level) {
+}
+
+ZlibCompressStream::ZlibCompressStream(DataStream* other, bool gzip, int level)
+  : ZlibBaseStream(other)
+  , logger_(logging::LoggerFactory<ZlibCompressStream>::getLogger()) {
+  int ret = deflateInit2(
+      &strm_,
+      level,
+      Z_DEFLATED /* method */,
+      15 + (gzip ? 16 : 0) /* windowBits */,
+      8 /* memLevel */,
+      Z_DEFAULT_STRATEGY /* strategy */);
+  if (ret != Z_OK) {
+    logger_->log_error("Failed to initialize z_stream with deflateInit2, error code: %d", ret);
+    throw Exception(ExceptionType::GENERAL_EXCEPTION, "zlib deflateInit2 failed");
+  }
+
+  state_ = ZlibStreamState::INITIALIZED;
+}
+
+ZlibCompressStream::~ZlibCompressStream() {
+  if (state_ != ZlibStreamState::UNINITIALIZED) {
+    deflateEnd(&strm_);
+  }
+}
+
+int ZlibCompressStream::writeData(uint8_t* value, int size) {
+  if (state_ != ZlibStreamState::INITIALIZED) {
+    logger_->log_error("writeData called in invalid ZlibCompressStream state, state is %hhu", state_);
+    return -1;
+  }
+
+  strm_.next_in = value;
+  strm_.avail_in = size;
+
+  /*
+   * deflate consumes all input data it can (i.e. if it has enough output buffer it never leaves input data unconsumed)
+   * and fills the output buffer to the brim every time it can. This means that the proper way to use deflate is to
+   * watch avail_out: once deflate does not have to fill it to the brim, then it has consumed all data we have provided
+   * to it, and does not need more output buffer for the time being.
+   * When we have no more input data, we must call deflate with Z_FINISH to make it empty its internal buffers and
+   * close the compressed stream.
+   */
+  do {
+    logger_->log_trace("writeData has %u B of input data left", strm_.avail_in);
+
+    strm_.next_out = outputBuffer_.data();
+    strm_.avail_out = outputBuffer_.size();
+
+    int flush = value == nullptr ? Z_FINISH : Z_NO_FLUSH;
+    logger_->log_trace("calling deflate with flush %d", flush);
+
+    int ret = deflate(&strm_, flush);
+    if (ret == Z_STREAM_ERROR) {
+      logger_->log_error("deflate failed, error code: %d", ret);
+      state_ = ZlibStreamState::ERRORED;
+      return -1;
+    }
+    int output_size = outputBuffer_.size() - strm_.avail_out;
+    logger_->log_trace("deflate produced %d B of output data", output_size);
+    if (BaseStream::writeData(outputBuffer_.data(), output_size) != output_size) {
+      logger_->log_error("Failed to write to underlying stream");
+      state_ = ZlibStreamState::ERRORED;
+      return -1;
+    }
+  } while (strm_.avail_out == 0);
+
+  return size;
+}
+
+void ZlibCompressStream::closeStream() {
+  if (state_ == ZlibStreamState::INITIALIZED) {
+    if (writeData(nullptr, 0U) == 0) {
+      state_ = ZlibStreamState::FINISHED;
+    }
+  }
+}
+
+/* ZlibDecompressStream */
+
+ZlibDecompressStream::ZlibDecompressStream(bool gzip)
+  : ZlibDecompressStream(this, gzip) {
+}
+
+ZlibDecompressStream::ZlibDecompressStream(DataStream* other, bool gzip)
+    : ZlibBaseStream(other)
+    , logger_(logging::LoggerFactory<ZlibDecompressStream>::getLogger()) {
+  int ret = inflateInit2(&strm_, 15 + (gzip ? 16 : 0) /* windowBits */);
+  if (ret != Z_OK) {
+    logger_->log_error("Failed to initialize z_stream with inflateInit2, error code: %d", ret);
+    throw Exception(ExceptionType::GENERAL_EXCEPTION, "zlib inflateInit2 failed");
+  }
+
+  state_ = ZlibStreamState::INITIALIZED;
+}
+
+ZlibDecompressStream::~ZlibDecompressStream() {
+  if (state_ != ZlibStreamState::UNINITIALIZED) {
+    inflateEnd(&strm_);
+  }
+}
+
+int ZlibDecompressStream::writeData(uint8_t* value, int size) {
+  if (state_ != ZlibStreamState::INITIALIZED) {
+    logger_->log_error("writeData called in invalid ZlibDecompressStream state, state is %hhu", state_);
+    return -1;
+  }
+
+  strm_.next_in = value;
+  strm_.avail_in = size;
+
+  /*
+   * inflate works similarly to deflate in that it will not leave input data unconsumed, and we have to watch avail_out,
+   * but in this case we do not have to close the stream, because it will detect the end of the compressed format
+   * and signal that it is ended by returning Z_STREAM_END and not accepting any more input data.
+   */
+  int ret;
+  do {
+    logger_->log_trace("writeData has %u B of input data left", strm_.avail_in);
+
+    strm_.next_out = outputBuffer_.data();
+    strm_.avail_out = outputBuffer_.size();
+
+    ret = inflate(&strm_, Z_NO_FLUSH);
+    if (ret == Z_STREAM_ERROR ||
+        ret == Z_NEED_DICT ||
+        ret == Z_DATA_ERROR ||
 
 Review comment:
   This is how it is designed, see the example usage: https://zlib.net/zlib_how.html

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r403766559
 
 

 ##########
 File path: libminifi/src/io/ZlibStream.cpp
 ##########
 @@ -0,0 +1,202 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "io/ZlibStream.h"
+#include "Exception.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace io {
+
+/* ZlibBaseStream */
+
+ZlibBaseStream::ZlibBaseStream()
+    : ZlibBaseStream(this) {
+}
+
+ZlibBaseStream::ZlibBaseStream(DataStream* other)
+    : BaseStream(other)
+    , outputBuffer_(16384U) {
+  strm_.zalloc = Z_NULL;
+  strm_.zfree = Z_NULL;
+  strm_.opaque = Z_NULL;
+}
+
+bool ZlibBaseStream::isFinished() const {
+  return state_ == ZlibStreamState::FINISHED;
+}
+
+/* ZlibCompressStream */
+
+ZlibCompressStream::ZlibCompressStream(bool gzip, int level)
+  : ZlibCompressStream(this, gzip, level) {
+}
+
+ZlibCompressStream::ZlibCompressStream(DataStream* other, bool gzip, int level)
+  : ZlibBaseStream(other)
+  , logger_(logging::LoggerFactory<ZlibCompressStream>::getLogger()) {
+  int ret = deflateInit2(
+      &strm_,
+      level,
+      Z_DEFLATED /* method */,
+      15 + (gzip ? 16 : 0) /* windowBits */,
+      8 /* memLevel */,
+      Z_DEFAULT_STRATEGY /* strategy */);
+  if (ret != Z_OK) {
+    logger_->log_error("Failed to initialize z_stream with deflateInit2, error code: %d", ret);
+    throw Exception(ExceptionType::GENERAL_EXCEPTION, "zlib deflateInit2 failed");
+  }
+
+  state_ = ZlibStreamState::INITIALIZED;
+}
+
+ZlibCompressStream::~ZlibCompressStream() {
+  if (state_ != ZlibStreamState::UNINITIALIZED) {
+    deflateEnd(&strm_);
+  }
+}
+
+int ZlibCompressStream::writeData(uint8_t* value, int size) {
+  if (state_ != ZlibStreamState::INITIALIZED) {
+    logger_->log_error("writeData called in invalid ZlibCompressStream state, state is %hhu", state_);
+    return -1;
+  }
+
+  strm_.next_in = value;
+  strm_.avail_in = size;
+
+  /*
+   * deflate consumes all input data it can (i.e. if it has enough output buffer it never leaves input data unconsumed)
+   * and fills the output buffer to the brim every time it can. This means that the proper way to use deflate is to
+   * watch avail_out: once deflate does not have to fill it to the brim, then it has consumed all data we have provided
+   * to it, and does not need more output buffer for the time being.
+   * When we have no more input data, we must call deflate with Z_FINISH to make it empty its internal buffers and
+   * close the compressed stream.
+   */
+  do {
+    logger_->log_trace("writeData has %u B of input data left", strm_.avail_in);
+
+    strm_.next_out = outputBuffer_.data();
+    strm_.avail_out = outputBuffer_.size();
+
+    int flush = value == nullptr ? Z_FINISH : Z_NO_FLUSH;
+    logger_->log_trace("calling deflate with flush %d", flush);
+
+    int ret = deflate(&strm_, flush);
+    if (ret == Z_STREAM_ERROR) {
+      logger_->log_error("deflate failed, error code: %d", ret);
+      state_ = ZlibStreamState::ERRORED;
+      return -1;
+    }
+    int output_size = outputBuffer_.size() - strm_.avail_out;
+    logger_->log_trace("deflate produced %d B of output data", output_size);
+    if (BaseStream::writeData(outputBuffer_.data(), output_size) != output_size) {
+      logger_->log_error("Failed to write to underlying stream");
+      state_ = ZlibStreamState::ERRORED;
+      return -1;
+    }
+  } while (strm_.avail_out == 0);
+
+  return size;
+}
+
+void ZlibCompressStream::closeStream() {
+  if (state_ == ZlibStreamState::INITIALIZED) {
+    if (writeData(nullptr, 0U) == 0) {
+      state_ = ZlibStreamState::FINISHED;
+    }
+  }
+}
+
+/* ZlibDecompressStream */
+
+ZlibDecompressStream::ZlibDecompressStream(bool gzip)
+  : ZlibDecompressStream(this, gzip) {
+}
+
+ZlibDecompressStream::ZlibDecompressStream(DataStream* other, bool gzip)
+    : ZlibBaseStream(other)
+    , logger_(logging::LoggerFactory<ZlibDecompressStream>::getLogger()) {
+  int ret = inflateInit2(&strm_, 15 + (gzip ? 16 : 0) /* windowBits */);
+  if (ret != Z_OK) {
+    logger_->log_error("Failed to initialize z_stream with inflateInit2, error code: %d", ret);
+    throw Exception(ExceptionType::GENERAL_EXCEPTION, "zlib inflateInit2 failed");
+  }
+
+  state_ = ZlibStreamState::INITIALIZED;
+}
+
+ZlibDecompressStream::~ZlibDecompressStream() {
+  if (state_ != ZlibStreamState::UNINITIALIZED) {
 
 Review comment:
   Okay, thanks, that makes sense.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r404727889
 
 

 ##########
 File path: libminifi/include/io/ZlibStream.h
 ##########
 @@ -0,0 +1,110 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef LIBMINIFI_INCLUDE_IO_ZLIBSTREAM_H_
+#define LIBMINIFI_INCLUDE_IO_ZLIBSTREAM_H_
+#include <cstdint>
+#include <vector>
+
+#include <zlib.h>
+
+#include "BaseStream.h"
+#include "core/logging/LoggerConfiguration.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace io {
+
+enum class ZlibCompressionFormat : uint8_t {
+  ZLIB,
+  GZIP
+};
+
+enum class ZlibStreamState : uint8_t {
+  UNINITIALIZED,
+  INITIALIZED,
+  ERRORED,
+  FINISHED
+};
+
+class ZlibBaseStream : public BaseStream {
+ public:
+  ZlibBaseStream();
+  ZlibBaseStream(DataStream* other);
+
+  ZlibBaseStream(const ZlibBaseStream&) = delete;
+  ZlibBaseStream& operator=(const ZlibBaseStream&) = delete;
+  ZlibBaseStream(ZlibBaseStream&& other) = delete;
+  ZlibBaseStream& operator=(ZlibBaseStream&& other) = delete;
 
 Review comment:
   Done.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on issue #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on issue #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#issuecomment-610326784
 
 
   Addressed new review comments.
   I would appreciate if we could move this PR along.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r401923330
 
 

 ##########
 File path: extensions/libarchive/CompressContent.h
 ##########
 @@ -341,6 +346,73 @@ class CompressContent: public core::Processor {
     }
   };
 
+  class GzipWriteCallback : public OutputStreamCallback {
+   public:
+    GzipWriteCallback(std::string compress_mode, int64_t compress_level, std::shared_ptr<core::FlowFile> flow, std::shared_ptr<core::ProcessSession> session)
+      : logger_(logging::LoggerFactory<CompressContent>::getLogger())
+      , compress_mode_(std::move(compress_mode))
+      , compress_level_(compress_level)
+      , flow_(std::move(flow))
+      , session_(std::move(session))
+    {
+    }
+
+    ~GzipWriteCallback() override = default;
+
+    std::shared_ptr<logging::Logger> logger_;
+    std::string compress_mode_;
+    int64_t compress_level_;
+    std::shared_ptr<core::FlowFile> flow_;
+    std::shared_ptr<core::ProcessSession> session_;
+    bool success_{false};
 
 Review comment:
   Loggers should also use an in-class/default member initializer as they are always initialized the same way (per class) and it reduces clutter in the ctor init list.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r404726572
 
 

 ##########
 File path: extensions/libarchive/CompressContent.cpp
 ##########
 @@ -35,10 +35,32 @@ namespace nifi {
 namespace minifi {
 namespace processors {
 
-core::Property CompressContent::CompressLevel("Compression Level", "The compression level to use; this is valid only when using GZIP compression.", "1");
-core::Property CompressContent::CompressMode("Mode", "Indicates whether the processor should compress content or decompress content.", MODE_COMPRESS);
-core::Property CompressContent::CompressFormat("Compression Format", "The compression format to use.", COMPRESSION_FORMAT_ATTRIBUTE);
-core::Property CompressContent::UpdateFileName("Update Filename", "Determines if filename extension need to be updated", "false");
+core::Property CompressContent::CompressLevel(
+    core::PropertyBuilder::createProperty("Compression Level")->withDescription("The compression level to use; this is valid only when using GZIP compression.")
+        ->isRequired(false)->withDefaultValue<int>(1)->build());
+core::Property CompressContent::CompressMode(
+    core::PropertyBuilder::createProperty("Mode")->withDescription("Indicates whether the processor should compress content or decompress content.")
+        ->isRequired(false)->withAllowableValues<std::string>({MODE_COMPRESS, MODE_DECOMPRESS})->withDefaultValue(MODE_COMPRESS)->build());
+core::Property CompressContent::CompressFormat(
+    core::PropertyBuilder::createProperty("Compression Format")->withDescription("The compression format to use.")
+        ->isRequired(false)
+        ->withAllowableValues<std::string>({
+          COMPRESSION_FORMAT_ATTRIBUTE,
+          COMPRESSION_FORMAT_GZIP,
+          COMPRESSION_FORMAT_BZIP2,
+          COMPRESSION_FORMAT_XZ_LZMA2,
 
 Review comment:
   Could you elaborate? Any YAML config that didn't use values from this enum was erroneous and resulted in an error on onTrigger. How this change would break those configurations further?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r401549110
 
 

 ##########
 File path: extensions/libarchive/CompressContent.cpp
 ##########
 @@ -39,6 +39,11 @@ core::Property CompressContent::CompressLevel("Compression Level", "The compress
 core::Property CompressContent::CompressMode("Mode", "Indicates whether the processor should compress content or decompress content.", MODE_COMPRESS);
 core::Property CompressContent::CompressFormat("Compression Format", "The compression format to use.", COMPRESSION_FORMAT_ATTRIBUTE);
 core::Property CompressContent::UpdateFileName("Update Filename", "Determines if filename extension need to be updated", "false");
+core::Property CompressContent::EncapsulateInTar("Encapsulate in TAR", "If true, on compression the FlowFile is added to a TAR archive and then compressed, "
 
 Review comment:
   I think using the propertybuilder would help adding a bool validator and a bool default true value

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r403989512
 
 

 ##########
 File path: libminifi/include/io/ZlibStream.h
 ##########
 @@ -0,0 +1,105 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef LIBMINIFI_INCLUDE_IO_ZLIBSTREAM_H_
+#define LIBMINIFI_INCLUDE_IO_ZLIBSTREAM_H_
+#include <cstdint>
+#include <vector>
+
+#include <zlib.h>
+
+#include "BaseStream.h"
+#include "core/logging/LoggerConfiguration.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace io {
+
+enum class ZlibStreamState : uint8_t {
+  UNINITIALIZED,
+  INITIALIZED,
+  ERRORED,
+  FINISHED
+};
+
+class ZlibBaseStream : public BaseStream {
+ public:
+  ZlibBaseStream();
+  ZlibBaseStream(DataStream* other);
+
+  ZlibBaseStream(const ZlibBaseStream&) = delete;
+  ZlibBaseStream& operator=(const ZlibBaseStream&) = delete;
+  ZlibBaseStream(ZlibBaseStream&& other) = delete;
+  ZlibBaseStream& operator=(ZlibBaseStream&& other) = delete;
+
+  ~ZlibBaseStream() override = default;
+
+  virtual bool isFinished() const;
+
+ protected:
+  ZlibStreamState state_{ZlibStreamState::UNINITIALIZED};
+  z_stream strm_{};
+  std::vector<uint8_t> outputBuffer_;
+};
+
+class ZlibCompressStream : public ZlibBaseStream {
+ public:
+  ZlibCompressStream(bool gzip = true, int level = Z_DEFAULT_COMPRESSION);
+  ZlibCompressStream(DataStream* other, bool gzip = true, int level = Z_DEFAULT_COMPRESSION);
 
 Review comment:
   Could you also mark both ctors as `explicit`? Since this is new code, it doesn't break API compatibility.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r403982252
 
 

 ##########
 File path: extensions/libarchive/CompressContent.h
 ##########
 @@ -341,6 +346,73 @@ class CompressContent: public core::Processor {
     }
   };
 
+  class GzipWriteCallback : public OutputStreamCallback {
+   public:
+    GzipWriteCallback(std::string compress_mode, int64_t compress_level, std::shared_ptr<core::FlowFile> flow, std::shared_ptr<core::ProcessSession> session)
+      : logger_(logging::LoggerFactory<CompressContent>::getLogger())
+      , compress_mode_(std::move(compress_mode))
+      , compress_level_(compress_level)
+      , flow_(std::move(flow))
+      , session_(std::move(session))
+    {
+    }
+
+    ~GzipWriteCallback() override = default;
+
+    std::shared_ptr<logging::Logger> logger_;
+    std::string compress_mode_;
+    int64_t compress_level_;
+    std::shared_ptr<core::FlowFile> flow_;
+    std::shared_ptr<core::ProcessSession> session_;
+    bool success_{false};
+
+    int64_t process(std::shared_ptr<io::BaseStream> outputStream) override {
+      class ReadCallback : public InputStreamCallback {
+       public:
+        ReadCallback(GzipWriteCallback& writer, std::shared_ptr<io::BaseStream> outputStream)
+          : writer_(writer)
+          , outputStream_(std::move(outputStream)) {
+        }
+
+        int64_t process(std::shared_ptr<io::BaseStream> inputStream) override {
+          std::vector<uint8_t> buffer(16 * 1024U);
+          int64_t read_size = 0;
+          while (read_size < writer_.flow_->getSize()) {
+            int ret = inputStream->read(buffer.data(), buffer.size());
+            if (ret < 0) {
+              return -1;
+            } else if (ret == 0) {
+              break;
+            } else {
+              if (outputStream_->writeData(buffer.data(), ret) != ret) {
+                return -1;
+              }
+              read_size += ret;
+            }
+          }
+          outputStream_->closeStream();
 
 Review comment:
   The generic answer is that our Stream API is not very highly (or properly) defined and whatever `closeStream` does, or when it should be called is implementation-defined.
   In this particular case we have to call `closeStream` in the successful case, because that's how we signal the end of the input data and force zlib to flush its buffers and write out the end of the compressed stream.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r401568716
 
 

 ##########
 File path: libminifi/test/archive-tests/CompressContentTests.cpp
 ##########
 @@ -932,3 +938,107 @@ TEST_CASE("DecompressFileXYLZMA", "[compressfiletest8]") {
   }
 }
 
+TEST_CASE("RawGzipCompressionDecompression", "[compressfiletest8]") {
+  TestController testController;
+  LogTestController::getInstance().setTrace<org::apache::nifi::minifi::processors::CompressContent>();
+  LogTestController::getInstance().setTrace<org::apache::nifi::minifi::processors::PutFile>();
+
+  // Create temporary directories
+  char format_src[] = "/tmp/archives.XXXXXX";
+  std::string src_dir = testController.createTempDirectory(format_src);
+  REQUIRE(!src_dir.empty());
+
+  char format_dst[] = "/tmp/archived.XXXXXX";
+  std::string dst_dir = testController.createTempDirectory(format_dst);
+  REQUIRE(!dst_dir.empty());
+
+  // Define files
+  std::string src_file = utils::file::FileUtils::concat_path(src_dir, "src.txt");
+  std::string compressed_file = utils::file::FileUtils::concat_path(dst_dir, "src.txt.gz");
+  std::string decompressed_file = utils::file::FileUtils::concat_path(dst_dir, "src.txt");
+
+  // Build MiNiFi processing graph
+  auto plan = testController.createPlan();
+  auto get_file = plan->addProcessor(
+      "GetFile",
+      "GetFile");
+  auto compress_content = plan->addProcessor(
+      "CompressContent",
+      "CompressContent",
+      core::Relationship("success", "d"),
+      true);
+  auto put_compressed = plan->addProcessor(
+      "PutFile",
+      "PutFile",
+      core::Relationship("success", "d"),
+      true);
+  auto decompress_content = plan->addProcessor(
+      "CompressContent",
+      "CompressContent",
+      core::Relationship("success", "d"),
+      true);
+  auto put_decompressed = plan->addProcessor(
+      "PutFile",
+      "PutFile",
+      core::Relationship("success", "d"),
+      true);
+
+  // Configure GetFile processor
+  plan->setProperty(get_file, "Input Directory", src_dir);
+
+  // Configure CompressContent processor for compression
+  plan->setProperty(compress_content, "Mode", MODE_COMPRESS);
+  plan->setProperty(compress_content, "Compression Format", COMPRESSION_FORMAT_GZIP);
+  plan->setProperty(compress_content, "Update Filename", "true");
+  plan->setProperty(compress_content, "Encapsulate in TAR", "false");
+
+  // Configure first PutFile processor
+  plan->setProperty(put_compressed, "Directory", dst_dir);
+
+  // Configure CompressContent processor for decompression
+  plan->setProperty(decompress_content, "Mode", MODE_DECOMPRESS);
+  plan->setProperty(decompress_content, "Compression Format", COMPRESSION_FORMAT_GZIP);
+  plan->setProperty(decompress_content, "Update Filename", "true");
+  plan->setProperty(decompress_content, "Encapsulate in TAR", "false");
+
+  // Configure second PutFile processor
+  plan->setProperty(put_decompressed, "Directory", dst_dir);
+
+  // Create source file
+  std::string content;
+  SECTION("Empty content") {
+  }
+  SECTION("Short content") {
+    content = "Repeated repeated repeated repeated repeated stuff.";
+  }
+  SECTION("Long content") {
+    std::stringstream content_ss;
+    for (size_t i = 0U; i < 1024 * 1024U; i++) {
+      content_ss << "foobar";
+    }
+    content = content_ss.str();
+  }
+
+  std::fstream file;
+  file.open(src_file, std::ios::out);
+  file << content;
+  file.close();
+
+  // Run flow
+  testController.runSession(plan, true);
+
+  // Check compressed file
+  std::ifstream compressed(compressed_file, std::ios::in | std::ios::binary);
+  std::vector<uint8_t> compressed_content((std::istreambuf_iterator<char>(compressed)), std::istreambuf_iterator<char>());
+  REQUIRE(2 < compressed_content.size());
+  // gzip magic number
+  REQUIRE(0x1f == compressed_content[0]);
+  REQUIRE(0x8b == compressed_content[1]);
+
+  // Check decompressed file
+  std::ifstream decompressed(decompressed_file, std::ios::in | std::ios::binary);
+  std::string decompressed_content((std::istreambuf_iterator<char>(decompressed)), std::istreambuf_iterator<char>());
+  REQUIRE(content == decompressed_content);
 
 Review comment:
   Nice test!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
arpadboda commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r401564545
 
 

 ##########
 File path: libminifi/src/io/ZlibStream.cpp
 ##########
 @@ -0,0 +1,202 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "io/ZlibStream.h"
+#include "Exception.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace io {
+
+/* ZlibBaseStream */
+
+ZlibBaseStream::ZlibBaseStream()
+    : ZlibBaseStream(this) {
+}
+
+ZlibBaseStream::ZlibBaseStream(DataStream* other)
+    : BaseStream(other)
+    , outputBuffer_(16384U) {
+  strm_.zalloc = Z_NULL;
+  strm_.zfree = Z_NULL;
+  strm_.opaque = Z_NULL;
+}
+
+bool ZlibBaseStream::isFinished() const {
+  return state_ == ZlibStreamState::FINISHED;
+}
+
+/* ZlibCompressStream */
+
+ZlibCompressStream::ZlibCompressStream(bool gzip, int level)
+  : ZlibCompressStream(this, gzip, level) {
+}
+
+ZlibCompressStream::ZlibCompressStream(DataStream* other, bool gzip, int level)
+  : ZlibBaseStream(other)
+  , logger_(logging::LoggerFactory<ZlibCompressStream>::getLogger()) {
+  int ret = deflateInit2(
+      &strm_,
+      level,
+      Z_DEFLATED /* method */,
+      15 + (gzip ? 16 : 0) /* windowBits */,
+      8 /* memLevel */,
+      Z_DEFAULT_STRATEGY /* strategy */);
+  if (ret != Z_OK) {
+    logger_->log_error("Failed to initialize z_stream with deflateInit2, error code: %d", ret);
+    throw Exception(ExceptionType::GENERAL_EXCEPTION, "zlib deflateInit2 failed");
+  }
+
+  state_ = ZlibStreamState::INITIALIZED;
+}
+
+ZlibCompressStream::~ZlibCompressStream() {
+  if (state_ != ZlibStreamState::UNINITIALIZED) {
+    deflateEnd(&strm_);
+  }
+}
+
+int ZlibCompressStream::writeData(uint8_t* value, int size) {
+  if (state_ != ZlibStreamState::INITIALIZED) {
+    logger_->log_error("writeData called in invalid ZlibCompressStream state, state is %hhu", state_);
+    return -1;
+  }
+
+  strm_.next_in = value;
+  strm_.avail_in = size;
+
+  /*
+   * deflate consumes all input data it can (i.e. if it has enough output buffer it never leaves input data unconsumed)
+   * and fills the output buffer to the brim every time it can. This means that the proper way to use deflate is to
+   * watch avail_out: once deflate does not have to fill it to the brim, then it has consumed all data we have provided
+   * to it, and does not need more output buffer for the time being.
+   * When we have no more input data, we must call deflate with Z_FINISH to make it empty its internal buffers and
+   * close the compressed stream.
+   */
+  do {
+    logger_->log_trace("writeData has %u B of input data left", strm_.avail_in);
+
+    strm_.next_out = outputBuffer_.data();
+    strm_.avail_out = outputBuffer_.size();
+
+    int flush = value == nullptr ? Z_FINISH : Z_NO_FLUSH;
+    logger_->log_trace("calling deflate with flush %d", flush);
+
+    int ret = deflate(&strm_, flush);
+    if (ret == Z_STREAM_ERROR) {
+      logger_->log_error("deflate failed, error code: %d", ret);
+      state_ = ZlibStreamState::ERRORED;
+      return -1;
+    }
+    int output_size = outputBuffer_.size() - strm_.avail_out;
+    logger_->log_trace("deflate produced %d B of output data", output_size);
+    if (BaseStream::writeData(outputBuffer_.data(), output_size) != output_size) {
+      logger_->log_error("Failed to write to underlying stream");
+      state_ = ZlibStreamState::ERRORED;
+      return -1;
+    }
+  } while (strm_.avail_out == 0);
+
+  return size;
+}
+
+void ZlibCompressStream::closeStream() {
+  if (state_ == ZlibStreamState::INITIALIZED) {
+    if (writeData(nullptr, 0U) == 0) {
+      state_ = ZlibStreamState::FINISHED;
+    }
+  }
+}
+
+/* ZlibDecompressStream */
+
+ZlibDecompressStream::ZlibDecompressStream(bool gzip)
+  : ZlibDecompressStream(this, gzip) {
+}
+
+ZlibDecompressStream::ZlibDecompressStream(DataStream* other, bool gzip)
+    : ZlibBaseStream(other)
+    , logger_(logging::LoggerFactory<ZlibDecompressStream>::getLogger()) {
+  int ret = inflateInit2(&strm_, 15 + (gzip ? 16 : 0) /* windowBits */);
+  if (ret != Z_OK) {
+    logger_->log_error("Failed to initialize z_stream with inflateInit2, error code: %d", ret);
+    throw Exception(ExceptionType::GENERAL_EXCEPTION, "zlib inflateInit2 failed");
+  }
+
+  state_ = ZlibStreamState::INITIALIZED;
+}
+
+ZlibDecompressStream::~ZlibDecompressStream() {
+  if (state_ != ZlibStreamState::UNINITIALIZED) {
+    inflateEnd(&strm_);
+  }
+}
+
+int ZlibDecompressStream::writeData(uint8_t* value, int size) {
+  if (state_ != ZlibStreamState::INITIALIZED) {
+    logger_->log_error("writeData called in invalid ZlibDecompressStream state, state is %hhu", state_);
+    return -1;
+  }
+
+  strm_.next_in = value;
+  strm_.avail_in = size;
+
+  /*
+   * inflate works similarly to deflate in that it will not leave input data unconsumed, and we have to watch avail_out,
+   * but in this case we do not have to close the stream, because it will detect the end of the compressed format
+   * and signal that it is ended by returning Z_STREAM_END and not accepting any more input data.
+   */
+  int ret;
+  do {
+    logger_->log_trace("writeData has %u B of input data left", strm_.avail_in);
+
+    strm_.next_out = outputBuffer_.data();
+    strm_.avail_out = outputBuffer_.size();
+
+    ret = inflate(&strm_, Z_NO_FLUSH);
+    if (ret == Z_STREAM_ERROR ||
+        ret == Z_NEED_DICT ||
+        ret == Z_DATA_ERROR ||
 
 Review comment:
   Bleh, is this API designed so badly?
   
   I would expect a success case or something, but handling some errors that look like a specific set of errors is weird.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r404032649
 
 

 ##########
 File path: extensions/libarchive/CompressContent.h
 ##########
 @@ -341,6 +346,73 @@ class CompressContent: public core::Processor {
     }
   };
 
+  class GzipWriteCallback : public OutputStreamCallback {
+   public:
+    GzipWriteCallback(std::string compress_mode, int64_t compress_level, std::shared_ptr<core::FlowFile> flow, std::shared_ptr<core::ProcessSession> session)
+      : logger_(logging::LoggerFactory<CompressContent>::getLogger())
+      , compress_mode_(std::move(compress_mode))
+      , compress_level_(compress_level)
+      , flow_(std::move(flow))
+      , session_(std::move(session))
+    {
+    }
+
+    ~GzipWriteCallback() override = default;
+
+    std::shared_ptr<logging::Logger> logger_;
+    std::string compress_mode_;
+    int64_t compress_level_;
+    std::shared_ptr<core::FlowFile> flow_;
+    std::shared_ptr<core::ProcessSession> session_;
+    bool success_{false};
 
 Review comment:
   Changed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r401759274
 
 

 ##########
 File path: libminifi/src/io/ZlibStream.cpp
 ##########
 @@ -0,0 +1,202 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "io/ZlibStream.h"
+#include "Exception.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace io {
+
+/* ZlibBaseStream */
+
+ZlibBaseStream::ZlibBaseStream()
+    : ZlibBaseStream(this) {
+}
+
+ZlibBaseStream::ZlibBaseStream(DataStream* other)
+    : BaseStream(other)
+    , outputBuffer_(16384U) {
+  strm_.zalloc = Z_NULL;
+  strm_.zfree = Z_NULL;
+  strm_.opaque = Z_NULL;
+}
+
+bool ZlibBaseStream::isFinished() const {
+  return state_ == ZlibStreamState::FINISHED;
+}
+
+/* ZlibCompressStream */
+
+ZlibCompressStream::ZlibCompressStream(bool gzip, int level)
+  : ZlibCompressStream(this, gzip, level) {
+}
+
+ZlibCompressStream::ZlibCompressStream(DataStream* other, bool gzip, int level)
+  : ZlibBaseStream(other)
+  , logger_(logging::LoggerFactory<ZlibCompressStream>::getLogger()) {
+  int ret = deflateInit2(
+      &strm_,
+      level,
+      Z_DEFLATED /* method */,
+      15 + (gzip ? 16 : 0) /* windowBits */,
+      8 /* memLevel */,
+      Z_DEFAULT_STRATEGY /* strategy */);
+  if (ret != Z_OK) {
+    logger_->log_error("Failed to initialize z_stream with deflateInit2, error code: %d", ret);
+    throw Exception(ExceptionType::GENERAL_EXCEPTION, "zlib deflateInit2 failed");
+  }
+
+  state_ = ZlibStreamState::INITIALIZED;
+}
+
+ZlibCompressStream::~ZlibCompressStream() {
+  if (state_ != ZlibStreamState::UNINITIALIZED) {
 
 Review comment:
   You are right, but I think it is better readable this way.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r404208801
 
 

 ##########
 File path: extensions/libarchive/CompressContent.cpp
 ##########
 @@ -35,10 +35,32 @@ namespace nifi {
 namespace minifi {
 namespace processors {
 
-core::Property CompressContent::CompressLevel("Compression Level", "The compression level to use; this is valid only when using GZIP compression.", "1");
-core::Property CompressContent::CompressMode("Mode", "Indicates whether the processor should compress content or decompress content.", MODE_COMPRESS);
-core::Property CompressContent::CompressFormat("Compression Format", "The compression format to use.", COMPRESSION_FORMAT_ATTRIBUTE);
-core::Property CompressContent::UpdateFileName("Update Filename", "Determines if filename extension need to be updated", "false");
+core::Property CompressContent::CompressLevel(
+    core::PropertyBuilder::createProperty("Compression Level")->withDescription("The compression level to use; this is valid only when using GZIP compression.")
+        ->isRequired(false)->withDefaultValue<int>(1)->build());
+core::Property CompressContent::CompressMode(
+    core::PropertyBuilder::createProperty("Mode")->withDescription("Indicates whether the processor should compress content or decompress content.")
+        ->isRequired(false)->withAllowableValues<std::string>({MODE_COMPRESS, MODE_DECOMPRESS})->withDefaultValue(MODE_COMPRESS)->build());
+core::Property CompressContent::CompressFormat(
+    core::PropertyBuilder::createProperty("Compression Format")->withDescription("The compression format to use.")
+        ->isRequired(false)
+        ->withAllowableValues<std::string>({
+          COMPRESSION_FORMAT_ATTRIBUTE,
+          COMPRESSION_FORMAT_GZIP,
+          COMPRESSION_FORMAT_BZIP2,
+          COMPRESSION_FORMAT_XZ_LZMA2,
 
 Review comment:
   This restriction will break some config YAMLs that don't make much sense to begin with. I'm fine with it, but want to raise some attention to this fact.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r404155996
 
 

 ##########
 File path: extensions/libarchive/CompressContent.cpp
 ##########
 @@ -74,7 +80,13 @@ void CompressContent::onSchedule(core::ProcessContext *context, core::ProcessSes
   if (context->getProperty(UpdateFileName.getName(), value) && !value.empty()) {
     org::apache::nifi::minifi::utils::StringUtils::StringToBool(value, updateFileName_);
   }
-  logger_->log_info("Compress Content: Mode [%s] Format [%s] Level [%d] UpdateFileName [%d]", compressMode_, compressFormat_, compressLevel_, updateFileName_);
+  value = "";
+  if (context->getProperty(EncapsulateInTar.getName(), value) && !value.empty()) {
 
 Review comment:
   Done.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements

Posted by GitBox <gi...@apache.org>.
bakaid commented on a change in pull request #747: MINIFICPP-1036 - libarchive extension improvements
URL: https://github.com/apache/nifi-minifi-cpp/pull/747#discussion_r403999685
 
 

 ##########
 File path: libminifi/src/io/ZlibStream.cpp
 ##########
 @@ -0,0 +1,202 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "io/ZlibStream.h"
+#include "Exception.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace io {
+
+/* ZlibBaseStream */
+
+ZlibBaseStream::ZlibBaseStream()
+    : ZlibBaseStream(this) {
+}
+
+ZlibBaseStream::ZlibBaseStream(DataStream* other)
+    : BaseStream(other)
+    , outputBuffer_(16384U) {
+  strm_.zalloc = Z_NULL;
+  strm_.zfree = Z_NULL;
+  strm_.opaque = Z_NULL;
+}
+
+bool ZlibBaseStream::isFinished() const {
+  return state_ == ZlibStreamState::FINISHED;
+}
+
+/* ZlibCompressStream */
+
+ZlibCompressStream::ZlibCompressStream(bool gzip, int level)
+  : ZlibCompressStream(this, gzip, level) {
+}
+
+ZlibCompressStream::ZlibCompressStream(DataStream* other, bool gzip, int level)
+  : ZlibBaseStream(other)
+  , logger_(logging::LoggerFactory<ZlibCompressStream>::getLogger()) {
+  int ret = deflateInit2(
+      &strm_,
+      level,
+      Z_DEFLATED /* method */,
+      15 + (gzip ? 16 : 0) /* windowBits */,
+      8 /* memLevel */,
+      Z_DEFAULT_STRATEGY /* strategy */);
+  if (ret != Z_OK) {
+    logger_->log_error("Failed to initialize z_stream with deflateInit2, error code: %d", ret);
+    throw Exception(ExceptionType::GENERAL_EXCEPTION, "zlib deflateInit2 failed");
+  }
+
+  state_ = ZlibStreamState::INITIALIZED;
+}
+
+ZlibCompressStream::~ZlibCompressStream() {
+  if (state_ != ZlibStreamState::UNINITIALIZED) {
+    deflateEnd(&strm_);
+  }
+}
+
+int ZlibCompressStream::writeData(uint8_t* value, int size) {
+  if (state_ != ZlibStreamState::INITIALIZED) {
+    logger_->log_error("writeData called in invalid ZlibCompressStream state, state is %hhu", state_);
+    return -1;
+  }
+
+  strm_.next_in = value;
+  strm_.avail_in = size;
+
+  /*
+   * deflate consumes all input data it can (i.e. if it has enough output buffer it never leaves input data unconsumed)
+   * and fills the output buffer to the brim every time it can. This means that the proper way to use deflate is to
+   * watch avail_out: once deflate does not have to fill it to the brim, then it has consumed all data we have provided
+   * to it, and does not need more output buffer for the time being.
+   * When we have no more input data, we must call deflate with Z_FINISH to make it empty its internal buffers and
+   * close the compressed stream.
+   */
+  do {
+    logger_->log_trace("writeData has %u B of input data left", strm_.avail_in);
+
+    strm_.next_out = outputBuffer_.data();
+    strm_.avail_out = outputBuffer_.size();
+
+    int flush = value == nullptr ? Z_FINISH : Z_NO_FLUSH;
+    logger_->log_trace("calling deflate with flush %d", flush);
+
+    int ret = deflate(&strm_, flush);
+    if (ret == Z_STREAM_ERROR) {
+      logger_->log_error("deflate failed, error code: %d", ret);
+      state_ = ZlibStreamState::ERRORED;
+      return -1;
+    }
+    int output_size = outputBuffer_.size() - strm_.avail_out;
+    logger_->log_trace("deflate produced %d B of output data", output_size);
+    if (BaseStream::writeData(outputBuffer_.data(), output_size) != output_size) {
+      logger_->log_error("Failed to write to underlying stream");
+      state_ = ZlibStreamState::ERRORED;
+      return -1;
+    }
+  } while (strm_.avail_out == 0);
+
+  return size;
+}
+
+void ZlibCompressStream::closeStream() {
+  if (state_ == ZlibStreamState::INITIALIZED) {
+    if (writeData(nullptr, 0U) == 0) {
+      state_ = ZlibStreamState::FINISHED;
+    }
+  }
+}
+
+/* ZlibDecompressStream */
+
+ZlibDecompressStream::ZlibDecompressStream(bool gzip)
+  : ZlibDecompressStream(this, gzip) {
+}
+
+ZlibDecompressStream::ZlibDecompressStream(DataStream* other, bool gzip)
+    : ZlibBaseStream(other)
+    , logger_(logging::LoggerFactory<ZlibDecompressStream>::getLogger()) {
+  int ret = inflateInit2(&strm_, 15 + (gzip ? 16 : 0) /* windowBits */);
+  if (ret != Z_OK) {
+    logger_->log_error("Failed to initialize z_stream with inflateInit2, error code: %d", ret);
+    throw Exception(ExceptionType::GENERAL_EXCEPTION, "zlib inflateInit2 failed");
+  }
+
+  state_ = ZlibStreamState::INITIALIZED;
+}
+
+ZlibDecompressStream::~ZlibDecompressStream() {
+  if (state_ != ZlibStreamState::UNINITIALIZED) {
 
 Review comment:
   Not if you ever want to make this object movable: in that case you would have to set the moved-from object to UNINITIALIZED to prevent it from freeing the z_stream.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services