You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by phrocker <gi...@git.apache.org> on 2017/11/09 15:30:57 UTC
[GitHub] nifi-minifi-cpp pull request #181: MINIFICPP-297: Remove Boost dependencies ...
GitHub user phrocker opened a pull request:
https://github.com/apache/nifi-minifi-cpp/pull/181
MINIFICPP-297: Remove Boost dependencies since we have very little re…
…liance on it
MINIFICPP-297: Ensure posix compliance
Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with MINIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-297
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi-minifi-cpp/pull/181.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #181
----
commit 1e4fb83511482e8048989adc9ab6e3d5e8d74d66
Author: Marc Parisi <ph...@apache.org>
Date: 2017-11-08T19:03:22Z
MINIFICPP-297: Remove Boost dependencies since we have very little reliance on it
MINIFICPP-297: Ensure posix compliance
----
---
[GitHub] nifi-minifi-cpp pull request #181: MINIFICPP-297: Remove Boost dependencies ...
Posted by achristianson <gi...@git.apache.org>.
Github user achristianson commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/181#discussion_r150386373
--- Diff: libminifi/include/utils/FileSystemUtils.h ---
@@ -0,0 +1,122 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef LIBMINIFI_INCLUDE_UTILS_FILESYSTEMUTILS_H_
+#define LIBMINIFI_INCLUDE_UTILS_FILESYSTEMUTILS_H_
+
+#ifdef BOOST_VERSION
+#include <boost/filesystem.hpp>
+#else
+#include <cstdlib>
+#endif
+#include <cstdio>
+#include <unistd.h>
+#include <fcntl.h>
+#include "io/validation.h"
+#include "Id.h"
+#include "StringUtils.h"
+#ifdef WIN32
+#define stat _stat
+#endif
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+
+/**
+ * Simple implementation of some file system utilities.
+ *
+ * unique_file is not a static implementation so that we can support scope driven temporary files.
+ */
+class FileSystemUtils {
+ public:
+
+ FileSystemUtils() {
+ }
+
+ ~FileSystemUtils() {
+ for (auto file : unique_files_) {
+ unlink(file.c_str());
+ }
+ }
+ std::string unique_file(const std::string &location, bool keep = false) {
+
+ if (!IsNullOrEmpty(location)) {
+ std::string file_name = location + "/" + non_repeating_string_generator_.generate();
+ while (!verify_not_exist(file_name)) {
+ file_name = location + "/" + non_repeating_string_generator_.generate();
+ }
+ if (!keep)
+ unique_files_.push_back(file_name);
+ return file_name;
--- End diff --
Return std::move()?
---
[GitHub] nifi-minifi-cpp issue #181: MINIFICPP-297: Remove Boost dependencies since w...
Posted by achristianson <gi...@git.apache.org>.
Github user achristianson commented on the issue:
https://github.com/apache/nifi-minifi-cpp/pull/181
@phrocker my thought is to definitely keep the RAII to have simpler/safer resource management. I see the unlink in the destructor. I would just tweak the naming & organization a bit. Now I see this is actually a unique *temporary* file session, so I would put that in the name. I wouldn't get rid of FileSystemUtils, but would create a nested class i.e. FileSystemUtils::TempFileManager. Usage like this:
```c++
{
FileSystemUtils::TempFileManager mgr;
auto tmp_file = mgr.unique_file("/tmp", false);
// Do something with tmp_file
} // All tmp files created with mgr are cleaned up (RAII)
```
---
[GitHub] nifi-minifi-cpp pull request #181: MINIFICPP-297: Remove Boost dependencies ...
Posted by achristianson <gi...@git.apache.org>.
Github user achristianson commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/181#discussion_r150386380
--- Diff: libminifi/include/utils/FileSystemUtils.h ---
@@ -0,0 +1,122 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef LIBMINIFI_INCLUDE_UTILS_FILESYSTEMUTILS_H_
+#define LIBMINIFI_INCLUDE_UTILS_FILESYSTEMUTILS_H_
+
+#ifdef BOOST_VERSION
+#include <boost/filesystem.hpp>
+#else
+#include <cstdlib>
+#endif
+#include <cstdio>
+#include <unistd.h>
+#include <fcntl.h>
+#include "io/validation.h"
+#include "Id.h"
+#include "StringUtils.h"
+#ifdef WIN32
+#define stat _stat
+#endif
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+
+/**
+ * Simple implementation of some file system utilities.
+ *
+ * unique_file is not a static implementation so that we can support scope driven temporary files.
+ */
+class FileSystemUtils {
+ public:
+
+ FileSystemUtils() {
+ }
+
+ ~FileSystemUtils() {
+ for (auto file : unique_files_) {
+ unlink(file.c_str());
+ }
+ }
+ std::string unique_file(const std::string &location, bool keep = false) {
+
+ if (!IsNullOrEmpty(location)) {
+ std::string file_name = location + "/" + non_repeating_string_generator_.generate();
+ while (!verify_not_exist(file_name)) {
+ file_name = location + "/" + non_repeating_string_generator_.generate();
+ }
+ if (!keep)
+ unique_files_.push_back(file_name);
+ return file_name;
+ } else {
+ std::string file_name = "/tmp/" + non_repeating_string_generator_.generate();
+ while (!verify_not_exist(file_name)) {
+ file_name = "/tmp/" + non_repeating_string_generator_.generate();
+ }
+ if (!keep)
+ unique_files_.push_back(file_name);
+ return file_name;
--- End diff --
Return std::move(file_name)?
---
[GitHub] nifi-minifi-cpp issue #181: MINIFICPP-297: Remove Boost dependencies since w...
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on the issue:
https://github.com/apache/nifi-minifi-cpp/pull/181
@achristianson statefulness helps use RAII to eliminate temp files, whereas other techniques don't do this. It's a balance between cleaning them up automatically or requiring the user to. As I saw previously with the code I removed, this was not done, and I fear most users won't properly do this. I'm with you. My initial implementation was just a simple create file without tracking. I'm still on the fence about that. We should require users to remove files, but that's not an expectation from creating temp files.
---
[GitHub] nifi-minifi-cpp pull request #181: MINIFICPP-297: Remove Boost dependencies ...
Posted by minifirocks <gi...@git.apache.org>.
Github user minifirocks commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/181#discussion_r150865974
--- Diff: libminifi/include/utils/file/FileManager.h ---
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef LIBMINIFI_INCLUDE_UTILS_FILEMANAGER_H_
+#define LIBMINIFI_INCLUDE_UTILS_FILEMANAGER_H_
+
+#ifdef BOOST_VERSION
+#include <boost/filesystem.hpp>
+#else
+#include <cstdlib>
+#endif
+#include <cstdio>
+#include <unistd.h>
+#include <fcntl.h>
+#include "io/validation.h"
+#include "utils/Id.h"
+#include "utils/StringUtils.h"
+#ifdef WIN32
+#define stat _stat
+#endif
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+/**
+ * Simple implementation of simple file manager utilities.
+ *
+ * unique_file is not a static implementation so that we can support scope driven temporary files.
+ */
+class FileManager {
+ public:
+
+ FileManager() {
+ }
+
+ ~FileManager() {
+ for (auto file : unique_files_) {
+ unlink(file.c_str());
+ }
+ }
+ std::string unique_file(const std::string &location, bool keep = false) {
+
+ if (!IsNullOrEmpty(location)) {
+ std::string file_name = location + "/" + non_repeating_string_generator_.generate();
+ while (!verify_not_exist(file_name)) {
+ file_name = location + "/" + non_repeating_string_generator_.generate();
+ }
+ if (!keep)
+ unique_files_.push_back(file_name);
--- End diff --
why we need to do unlink later in the destructor
---
[GitHub] nifi-minifi-cpp issue #181: MINIFICPP-297: Remove Boost dependencies since w...
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on the issue:
https://github.com/apache/nifi-minifi-cpp/pull/181
Still doing some validation on older *nix releases to validate that the stat structure is the same everywhere. In some cases it will not be, so need to finish validation against all releases.
---
[GitHub] nifi-minifi-cpp pull request #181: MINIFICPP-297: Remove Boost dependencies ...
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/181#discussion_r150867403
--- Diff: libminifi/include/utils/file/FileManager.h ---
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef LIBMINIFI_INCLUDE_UTILS_FILEMANAGER_H_
+#define LIBMINIFI_INCLUDE_UTILS_FILEMANAGER_H_
+
+#ifdef BOOST_VERSION
+#include <boost/filesystem.hpp>
+#else
+#include <cstdlib>
+#endif
+#include <cstdio>
+#include <unistd.h>
+#include <fcntl.h>
+#include "io/validation.h"
+#include "utils/Id.h"
+#include "utils/StringUtils.h"
+#ifdef WIN32
+#define stat _stat
+#endif
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+/**
+ * Simple implementation of simple file manager utilities.
+ *
+ * unique_file is not a static implementation so that we can support scope driven temporary files.
+ */
+class FileManager {
+ public:
+
+ FileManager() {
+ }
+
+ ~FileManager() {
+ for (auto file : unique_files_) {
+ unlink(file.c_str());
+ }
+ }
+ std::string unique_file(const std::string &location, bool keep = false) {
+
+ if (!IsNullOrEmpty(location)) {
+ std::string file_name = location + "/" + non_repeating_string_generator_.generate();
+ while (!verify_not_exist(file_name)) {
+ file_name = location + "/" + non_repeating_string_generator_.generate();
+ }
+ if (!keep)
+ unique_files_.push_back(file_name);
--- End diff --
These are temporary files. If we don't specify keep it will be a temp file.
---
[GitHub] nifi-minifi-cpp pull request #181: MINIFICPP-297: Remove Boost dependencies ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi-minifi-cpp/pull/181
---
[GitHub] nifi-minifi-cpp issue #181: MINIFICPP-297: Remove Boost dependencies since w...
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on the issue:
https://github.com/apache/nifi-minifi-cpp/pull/181
@achristianson what are your thoughts on that? I'm so on the fence I need a second opinion about that.
---
[GitHub] nifi-minifi-cpp pull request #181: MINIFICPP-297: Remove Boost dependencies ...
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/181#discussion_r150574090
--- Diff: libminifi/include/utils/FileSystemUtils.h ---
@@ -0,0 +1,122 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef LIBMINIFI_INCLUDE_UTILS_FILESYSTEMUTILS_H_
+#define LIBMINIFI_INCLUDE_UTILS_FILESYSTEMUTILS_H_
+
+#ifdef BOOST_VERSION
+#include <boost/filesystem.hpp>
+#else
+#include <cstdlib>
+#endif
+#include <cstdio>
+#include <unistd.h>
+#include <fcntl.h>
+#include "io/validation.h"
+#include "Id.h"
+#include "StringUtils.h"
+#ifdef WIN32
+#define stat _stat
+#endif
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+
+/**
+ * Simple implementation of some file system utilities.
+ *
+ * unique_file is not a static implementation so that we can support scope driven temporary files.
+ */
+class FileSystemUtils {
+ public:
+
+ FileSystemUtils() {
+ }
+
+ ~FileSystemUtils() {
+ for (auto file : unique_files_) {
+ unlink(file.c_str());
+ }
+ }
+ std::string unique_file(const std::string &location, bool keep = false) {
+
+ if (!IsNullOrEmpty(location)) {
+ std::string file_name = location + "/" + non_repeating_string_generator_.generate();
+ while (!verify_not_exist(file_name)) {
+ file_name = location + "/" + non_repeating_string_generator_.generate();
+ }
+ if (!keep)
+ unique_files_.push_back(file_name);
+ return file_name;
--- End diff --
Sorry a better way to put my previous statement is that I'm relying on CE and RVO, especially with the push_back. It is an optimization, but one the compiler can choose.
---
[GitHub] nifi-minifi-cpp pull request #181: MINIFICPP-297: Remove Boost dependencies ...
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/181#discussion_r150546489
--- Diff: libminifi/include/utils/FileSystemUtils.h ---
@@ -0,0 +1,122 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef LIBMINIFI_INCLUDE_UTILS_FILESYSTEMUTILS_H_
+#define LIBMINIFI_INCLUDE_UTILS_FILESYSTEMUTILS_H_
+
+#ifdef BOOST_VERSION
+#include <boost/filesystem.hpp>
+#else
+#include <cstdlib>
+#endif
+#include <cstdio>
+#include <unistd.h>
+#include <fcntl.h>
+#include "io/validation.h"
+#include "Id.h"
+#include "StringUtils.h"
+#ifdef WIN32
+#define stat _stat
+#endif
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+
+/**
+ * Simple implementation of some file system utilities.
+ *
+ * unique_file is not a static implementation so that we can support scope driven temporary files.
+ */
+class FileSystemUtils {
+ public:
+
+ FileSystemUtils() {
+ }
+
+ ~FileSystemUtils() {
+ for (auto file : unique_files_) {
+ unlink(file.c_str());
+ }
+ }
+ std::string unique_file(const std::string &location, bool keep = false) {
+
+ if (!IsNullOrEmpty(location)) {
+ std::string file_name = location + "/" + non_repeating_string_generator_.generate();
+ while (!verify_not_exist(file_name)) {
+ file_name = location + "/" + non_repeating_string_generator_.generate();
+ }
+ if (!keep)
+ unique_files_.push_back(file_name);
+ return file_name;
--- End diff --
We cannot since it may go into a map. Then we move conditionally and that's generally not an accepted move.
---
[GitHub] nifi-minifi-cpp issue #181: MINIFICPP-297: Remove Boost dependencies since w...
Posted by achristianson <gi...@git.apache.org>.
Github user achristianson commented on the issue:
https://github.com/apache/nifi-minifi-cpp/pull/181
This looks great. My only critique is that FilesystemUtils is used in a stateful way (presumably just to keep a cache of previous unique files?) I think keeping it entirely stateless would be preferable. Looking at the same issue, keeping a stateful instance of "utilities" isn't very meaningful. If part of it really needs to be stateful, I would consider making a nested class in FilesystemUtils i.e. FilesystemUtils::UniqueFileGenerator.
Overall the change is great and removes a huge dependency without adding a lot of complexity.
---