You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2021/05/13 15:49:47 UTC
[GitHub] [nifi-minifi-cpp] lordgamez opened a new pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
lordgamez opened a new pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073
- Refactor SQL processors to remove explicit `soci` library dependancy
- Create MockDB for unit tests
- Add option to use real SQLite database in unit tests
- Add integration tests to test SQL processors with PostgreSQL database
Jira ticket: https://issues.apache.org/jira/browse/MINIFICPP-1541
-------------------------------------------------------------------
Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically main)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r644912446
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.cpp
##########
@@ -0,0 +1,386 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "MockConnectors.h"
+
+#include <fstream>
+#include <algorithm>
+#include <utility>
+#include <string>
+#include <memory>
+
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+std::size_t MockRow::size() const {
+ return column_names_.size();
+}
+
+std::string MockRow::getColumnName(std::size_t index) const {
+ return column_names_.at(index);
+}
+
+bool MockRow::isNull(std::size_t index) const {
+ return column_values_.at(index) == "NULL";
+}
+
+DataType MockRow::getDataType(std::size_t index) const {
+ return column_types_.at(index);
+}
+
+std::string MockRow::getString(std::size_t index) const {
+ return column_values_.at(index);
+}
+
+double MockRow::getDouble(std::size_t index) const {
+ return std::atof(column_values_.at(index).c_str());
+}
+
+int MockRow::getInteger(std::size_t index) const {
+ return std::atoi(column_values_.at(index).c_str());
+}
+
+long long MockRow::getLongLong(std::size_t index) const { // NOLINT
+ return std::atoll(column_values_.at(index).c_str());
+}
+
+unsigned long long MockRow::getUnsignedLongLong(std::size_t index) const { // NOLINT
+ return static_cast<unsigned long long>(std::atoll(column_values_.at(index).c_str())); // NOLINT
+}
+
+std::tm MockRow::getDate(std::size_t /*index*/) const {
+ throw std::runtime_error("date not implemented");
+}
+
+std::vector<std::string> MockRow::getValues() const {
+ return column_values_;
+}
+
+std::string MockRow::getValue(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return column_values_.at(it-column_names_.begin());
+ }
+ throw std::runtime_error("Unknown column name for getting value");
+}
+
+void MockRowset::addRow(const std::vector<std::string>& column_values) {
+ rows_.emplace_back(column_names_, column_types_, column_values);
+}
+
+void MockRowset::reset() {
+ current_row_ = rows_.begin();
+}
+
+bool MockRowset::is_done() {
+ return current_row_ == rows_.end();
+}
+
+Row& MockRowset::getCurrent() {
+ return *current_row_;
+}
+
+void MockRowset::next() {
+ ++current_row_;
+}
+
+std::vector<std::string> MockRowset::getColumnNames() const {
+ return column_names_;
+}
+
+std::vector<DataType> MockRowset::getColumnTypes() const {
+ return column_types_;
+}
+
+std::vector<MockRow> MockRowset::getRows() const {
+ return rows_;
+}
+
+std::size_t MockRowset::getColumnIndex(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return it-column_names_.begin();
+ }
+ throw std::runtime_error("Unknown column name for getting index");
+}
+
+void MockRowset::sort(const std::string& order_by_col, bool order_ascending) {
+ std::sort(rows_.begin(), rows_.end(), [&](const MockRow& first, const MockRow& second) {
+ if (order_ascending) {
+ return first.getValue(order_by_col) < second.getValue(order_by_col);
+ } else {
+ return first.getValue(order_by_col) > second.getValue(order_by_col);
+ }
+ });
+}
+
+std::unique_ptr<MockRowset> MockRowset::select(const std::vector<std::string>& cols, const std::function<bool(const MockRow&)>& condition, const std::string& order_by_col, bool order_ascending) {
+ if (!order_by_col.empty()) {
+ sort(order_by_col, order_ascending);
+ }
+
+ std::unique_ptr<MockRowset> rowset;
+ if (cols.empty()) {
+ rowset = utils::make_unique<MockRowset>(column_names_, column_types_);
+ } else {
+ std::vector<DataType> col_types;
+ for (const auto& col : cols) {
+ col_types.push_back(column_types_.at(getColumnIndex(col)));
+ }
+ rowset = utils::make_unique<MockRowset>(cols, col_types);
+ }
+
+ std::vector<std::string> used_cols = cols.empty() ? column_names_ : cols;
+ for (const auto& row : rows_) {
+ if (condition(row)) {
+ std::vector<std::string> values;
+ for (const auto& col : used_cols) {
+ values.push_back(row.getValue(col));
+ }
+ rowset->addRow(values);
+ }
+ }
+
+ return rowset;
+}
+
+std::unique_ptr<Rowset> MockDB::execute(const std::string& query, const std::vector<std::string>& args) {
+ if (minifi::utils::StringUtils::startsWith(query, "create table")) {
+ createTable(query);
+ } else if (minifi::utils::StringUtils::startsWith(query, "insert into")) {
+ insertInto(query, args);
+ } else if (minifi::utils::StringUtils::startsWith(query, "select")) {
+ return select(query, args);
+ } else {
+ throw std::runtime_error("Unknown query type");
+ }
+
+ return nullptr;
+}
+
+void MockDB::createTable(const std::string& query) {
+ std::smatch match;
+ std::regex expr("create table (\\w+)\\s*\\((.*)\\);");
+ std::regex_search(query, match, expr);
+ std::string table_name = match[1];
+ auto columns_with_type = minifi::utils::StringUtils::splitAndTrim(match[2], ",");
+ std::vector<std::string> col_names;
+ std::vector<DataType> col_types;
+ for (const auto& col_with_type : columns_with_type) {
+ auto splitted = minifi::utils::StringUtils::splitAndTrim(col_with_type, " ");
+ col_names.push_back(splitted[0]);
+ col_types.push_back(stringToDataType(splitted[1]));
+ }
+ tables_.emplace(table_name, MockRowset{col_names, col_types});
+ storeDb();
+}
+
+void MockDB::insertInto(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("insert into (\\w+)\\s*(\\((.*)\\))*\\s*values\\s*\\((.+)\\)");
+ std::regex_search(replaced_query, match, expr);
+ std::string table_name = match[1];
+ std::vector<std::string> values = minifi::utils::StringUtils::splitAndTrim(match[4], ",");
+ for (auto& value : values) {
+ value = minifi::utils::StringUtils::removeFramingCharacters(value, '\'');
+ }
+ auto insert_col_names = minifi::utils::StringUtils::splitAndTrim(match[3], ",");
+ if (!insert_col_names.empty()) {
+ auto col_names = tables_.at(table_name).getColumnNames();
+ std::vector<std::string> row;
+ for (const auto& col_name : col_names) {
+ auto it = std::find(insert_col_names.begin(), insert_col_names.end(), col_name);
+ if (it != insert_col_names.end()) {
+ row.push_back(values.at(it-insert_col_names.begin()));
+ } else {
+ row.push_back("NULL");
+ }
+ }
+ tables_.at(table_name).addRow(row);
+ } else {
+ tables_.at(table_name).addRow(values);
+ }
+
+ storeDb();
+}
+
+std::unique_ptr<Rowset> MockDB::select(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("select\\s+(.+)\\s+from\\s+(\\w+)\\s*(where ((.+(?= order by))|.+$))*\\s*(order by (.+))*");
+ std::regex_search(replaced_query, match, expr);
+ auto cols = minifi::utils::StringUtils::splitAndTrim(match[1], ",");
+ if (cols[0] == "*") {
+ cols = {};
+ }
+ std::string table_name = match[2];
+ std::string condition_str = match[4];
+ std::function<bool(const MockRow&)> condition;
+ if (!condition_str.empty()) {
+ if (condition_str == "int_col > 103") {
Review comment:
Added a parser for simple where conditions in 96c3a9dea8e0d9c0492ccf5850184755651418e9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r644583752
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.cpp
##########
@@ -0,0 +1,386 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "MockConnectors.h"
+
+#include <fstream>
+#include <algorithm>
+#include <utility>
+#include <string>
+#include <memory>
+
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+std::size_t MockRow::size() const {
+ return column_names_.size();
+}
+
+std::string MockRow::getColumnName(std::size_t index) const {
+ return column_names_.at(index);
+}
+
+bool MockRow::isNull(std::size_t index) const {
+ return column_values_.at(index) == "NULL";
+}
+
+DataType MockRow::getDataType(std::size_t index) const {
+ return column_types_.at(index);
+}
+
+std::string MockRow::getString(std::size_t index) const {
+ return column_values_.at(index);
+}
+
+double MockRow::getDouble(std::size_t index) const {
+ return std::atof(column_values_.at(index).c_str());
+}
+
+int MockRow::getInteger(std::size_t index) const {
+ return std::atoi(column_values_.at(index).c_str());
Review comment:
should we use the `std::string` accepting version? (`std::stoi` and others)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r644737739
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.h
##########
@@ -0,0 +1,175 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <regex>
+#include <map>
+#include <vector>
+#include <algorithm>
+#include <memory>
+#include <string>
+
+#include "data/DatabaseConnectors.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+class MockRow : public Row {
+ public:
+ MockRow(std::vector<std::string>& column_names, std::vector<DataType>& column_types, const std::vector<std::string>& column_values)
+ : column_names_(column_names), column_types_(column_types), column_values_(column_values) {
+ }
+
+ MockRow(MockRow&& other) = default;
+ MockRow(const MockRow& other) = default;
+ MockRow& operator=(MockRow&& other) {
+ column_names_ = other.column_names_;
+ column_types_ = other.column_types_;
Review comment:
I'd say just use pointers. Whether you write `*column_names_` or `column_names_.get()` doesn't really matter. Raw pointers are not evil as observer pointers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r644729942
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.h
##########
@@ -0,0 +1,175 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <regex>
+#include <map>
+#include <vector>
+#include <algorithm>
+#include <memory>
+#include <string>
+
+#include "data/DatabaseConnectors.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+class MockRow : public Row {
+ public:
+ MockRow(std::vector<std::string>& column_names, std::vector<DataType>& column_types, const std::vector<std::string>& column_values)
+ : column_names_(column_names), column_types_(column_types), column_values_(column_values) {
+ }
+
+ MockRow(MockRow&& other) = default;
+ MockRow(const MockRow& other) = default;
+ MockRow& operator=(MockRow&& other) {
+ column_names_ = other.column_names_;
+ column_types_ = other.column_types_;
Review comment:
You right this was wrong. The original goal was just to make the class copyable, and `std::reference_wrapper` could have been a good solution here. Finally I decided it's simpler just to use a raw pointer in this case. Updated in ee9452ebd591ff22139606ba2cd682df912d14f4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r648097041
##########
File path: libminifi/include/utils/StringUtils.h
##########
@@ -137,6 +137,7 @@ class StringUtils {
static std::vector<std::string> split(const std::string &str, const std::string &delimiter);
static std::vector<std::string> splitAndTrim(const std::string &str, const std::string &delimiter);
+ static std::vector<std::string> splitAndTrimOnString(std::string str, const std::string& delimiter);
Review comment:
Because of the new semantics of `split` there are still some issues that need to be resolved, putting it to draft until it's fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r644737739
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.h
##########
@@ -0,0 +1,175 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <regex>
+#include <map>
+#include <vector>
+#include <algorithm>
+#include <memory>
+#include <string>
+
+#include "data/DatabaseConnectors.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+class MockRow : public Row {
+ public:
+ MockRow(std::vector<std::string>& column_names, std::vector<DataType>& column_types, const std::vector<std::string>& column_values)
+ : column_names_(column_names), column_types_(column_types), column_values_(column_values) {
+ }
+
+ MockRow(MockRow&& other) = default;
+ MockRow(const MockRow& other) = default;
+ MockRow& operator=(MockRow&& other) {
+ column_names_ = other.column_names_;
+ column_types_ = other.column_types_;
Review comment:
I'd say just use pointers. Whether you write `*column_names_` or `column_names_.get()` doesn't really matter. Raw pointers are not evil as observer pointers.
edit: looks like I was late with this reply :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r644604564
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.h
##########
@@ -0,0 +1,175 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <regex>
+#include <map>
+#include <vector>
+#include <algorithm>
+#include <memory>
+#include <string>
+
+#include "data/DatabaseConnectors.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+class MockRow : public Row {
+ public:
+ MockRow(std::vector<std::string>& column_names, std::vector<DataType>& column_types, const std::vector<std::string>& column_values)
+ : column_names_(column_names), column_types_(column_types), column_values_(column_values) {
+ }
+
+ MockRow(MockRow&& other) = default;
+ MockRow(const MockRow& other) = default;
+ MockRow& operator=(MockRow&& other) {
+ column_names_ = other.column_names_;
+ column_types_ = other.column_types_;
Review comment:
using `std::reference_wrapper` would most likely offer the semantics we are looking for here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r676613868
##########
File path: .github/workflows/ci.yml
##########
@@ -36,7 +36,7 @@ jobs:
export LDFLAGS="-L/usr/local/opt/flex/lib"
export CPPFLAGS="-I/usr/local/opt/flex/include"
# CPPFLAGS are not recognized by cmake, so we have to force them to CFLAGS and CXXFLAGS to have flex 2.6 working
- ./bootstrap.sh -e -t && cd build && cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_FLAGS="${CPPFLAGS} ${CFLAGS}" -DCMAKE_CXX_FLAGS="${CPPFLAGS} ${CXXFLAGS}" -DENABLE_LUA_SCRIPTING=1 -DENABLE_SQL=ON -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_RULE_MESSAGES=OFF -DSTRICT_GSL_CHECKS=AUDIT -DFAIL_ON_WARNINGS=OFF .. && cmake --build . --parallel 4
+ ./bootstrap.sh -e -t && cd build && cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_FLAGS="${CPPFLAGS} ${CFLAGS}" -DCMAKE_CXX_FLAGS="${CPPFLAGS} ${CXXFLAGS}" -DENABLE_LUA_SCRIPTING=1 -DENABLE_SQL=ON -DUSE_REAL_ODBC_TEST_DRIVER=ON -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_RULE_MESSAGES=OFF -DSTRICT_GSL_CHECKS=AUDIT -DFAIL_ON_WARNINGS=ON .. && cmake --build . --parallel 4
Review comment:
Had some conflicts in the ci.yml while rebasing and I changed the `FAIL_ON_WARNINGS` flag to `ON` in the mac build and added the `ENABLE_AZURE=ON` option in the clang build. These options were included in the previous base and changed on the main branch, but I think both should be included in the CI jobs. If it's not a problem I would like to include these in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r644746011
##########
File path: extensions/sql/data/DatabaseConnectors.h
##########
@@ -20,72 +20,68 @@
#include <memory>
#include <string>
-
-#include <soci/soci.h>
-
-#include "Utils.h"
+#include <vector>
+#include <ctime>
namespace org {
namespace apache {
namespace nifi {
namespace minifi {
namespace sql {
-/**
- * We do not intend to create an abstract facade here. We know that SOCI is the underlying
- * SQL library. We only wish to abstract ODBC specific information
- */
+enum class DataType {
+ STRING,
+ DOUBLE,
+ INTEGER,
+ LONG_LONG,
+ UNSIGNED_LONG_LONG,
+ DATE
+};
-class Statement {
+class Row {
public:
+ virtual ~Row() = default;
+ virtual std::size_t size() const = 0;
+ virtual std::string getColumnName(std::size_t index) const = 0;
+ virtual bool isNull(std::size_t index) const = 0;
+ virtual DataType getDataType(std::size_t index) const = 0;
+ virtual std::string getString(std::size_t index) const = 0;
+ virtual double getDouble(std::size_t index) const = 0;
+ virtual int getInteger(std::size_t index) const = 0;
+ virtual long long getLongLong(std::size_t index) const = 0;
+ virtual unsigned long long getUnsignedLongLong(std::size_t index) const = 0;
+ virtual std::tm getDate(std::size_t index) const = 0;
Review comment:
This could be a `std::chrono::time_point` or if you only need the date part, then `date::year_month_day`. These types are easier to work with.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r644601240
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.h
##########
@@ -0,0 +1,175 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <regex>
+#include <map>
+#include <vector>
+#include <algorithm>
+#include <memory>
+#include <string>
+
+#include "data/DatabaseConnectors.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+class MockRow : public Row {
+ public:
+ MockRow(std::vector<std::string>& column_names, std::vector<DataType>& column_types, const std::vector<std::string>& column_values)
+ : column_names_(column_names), column_types_(column_types), column_values_(column_values) {
+ }
+
+ MockRow(MockRow&& other) = default;
+ MockRow(const MockRow& other) = default;
+ MockRow& operator=(MockRow&& other) {
+ column_names_ = other.column_names_;
+ column_types_ = other.column_types_;
Review comment:
I don't know the exact intention here, but I wager it won't be doing that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r645489479
##########
File path: libminifi/include/utils/StringUtils.h
##########
@@ -137,6 +137,7 @@ class StringUtils {
static std::vector<std::string> split(const std::string &str, const std::string &delimiter);
static std::vector<std::string> splitAndTrim(const std::string &str, const std::string &delimiter);
+ static std::vector<std::string> splitAndTrimOnString(std::string str, const std::string& delimiter);
Review comment:
`splitAndTrim` only uses the first character of the given string as a delimiter, so it actually is using a character as a delimiter not a string. I'm not sure if that should be updated to have a character as an input delimiter and keep 2 overloads or the functions should be merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r670367872
##########
File path: extensions/sql/data/SociConnectors.cpp
##########
@@ -0,0 +1,181 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "SociConnectors.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+void SociRow::setIterator(const soci::rowset<soci::row>::iterator& iter) {
+ current_ = iter;
+}
+
+soci::rowset<soci::row>::iterator SociRow::getIterator() const {
+ return current_;
+}
+
+void SociRow::next() {
+ ++current_;
+}
+
+std::size_t SociRow::size() const {
+ return current_->size();
+}
+
+std::string SociRow::getColumnName(std::size_t index) const {
+ return current_->get_properties(index).get_name();
+}
+
+bool SociRow::isNull(std::size_t index) const {
+ return current_->get_indicator(index) == soci::i_null;
+}
+
+DataType SociRow::getDataType(std::size_t index) const {
+ switch (const auto dataType = current_->get_properties(index).get_data_type()) {
+ case soci::data_type::dt_string: {
+ return DataType::STRING;
+ }
+ case soci::data_type::dt_double: {
+ return DataType::DOUBLE;
+ }
+ case soci::data_type::dt_integer: {
+ return DataType::INTEGER;
+ }
+ case soci::data_type::dt_long_long: {
+ return DataType::LONG_LONG;
+ }
+ case soci::data_type::dt_unsigned_long_long: {
+ return DataType::UNSIGNED_LONG_LONG;
+ }
+ case soci::data_type::dt_date: {
+ return DataType::DATE;
+ }
+ default: {
+ throw minifi::Exception(PROCESSOR_EXCEPTION, "SQLRowsetProcessor: Unsupported data type " + std::to_string(dataType));
+ }
+ }
+}
+
+std::string SociRow::getString(std::size_t index) const {
+ return current_->get<std::string>(index);
+}
+
+double SociRow::getDouble(std::size_t index) const {
+ return current_->get<double>(index);
+}
+
+int SociRow::getInteger(std::size_t index) const {
+ return current_->get<int>(index);
+}
+
+long long SociRow::getLongLong(std::size_t index) const {
+ return current_->get<long long>(index);
+}
+
+unsigned long long SociRow::getUnsignedLongLong(std::size_t index) const {
+ return current_->get<unsigned long long>(index);
+}
+
+std::tm SociRow::getDate(std::size_t index) const {
+ return current_->get<std::tm>(index);
+}
Review comment:
I considered it, but as these functions needed to be generalized in a common base class (to be later implemented in the mock and the soci implementation) I went with the pure virtual functions for each getter in the base `Row` class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r644915776
##########
File path: extensions/sql/data/DatabaseConnectors.h
##########
@@ -20,72 +20,68 @@
#include <memory>
#include <string>
-
-#include <soci/soci.h>
-
-#include "Utils.h"
+#include <vector>
+#include <ctime>
namespace org {
namespace apache {
namespace nifi {
namespace minifi {
namespace sql {
-/**
- * We do not intend to create an abstract facade here. We know that SOCI is the underlying
- * SQL library. We only wish to abstract ODBC specific information
- */
+enum class DataType {
+ STRING,
+ DOUBLE,
+ INTEGER,
+ LONG_LONG,
+ UNSIGNED_LONG_LONG,
+ DATE
+};
-class Statement {
+class Row {
public:
+ virtual ~Row() = default;
+ virtual std::size_t size() const = 0;
+ virtual std::string getColumnName(std::size_t index) const = 0;
+ virtual bool isNull(std::size_t index) const = 0;
+ virtual DataType getDataType(std::size_t index) const = 0;
+ virtual std::string getString(std::size_t index) const = 0;
+ virtual double getDouble(std::size_t index) const = 0;
+ virtual int getInteger(std::size_t index) const = 0;
+ virtual long long getLongLong(std::size_t index) const = 0;
+ virtual unsigned long long getUnsignedLongLong(std::size_t index) const = 0;
+ virtual std::tm getDate(std::size_t index) const = 0;
Review comment:
The specialization for `row::get` for the date type is `std::tm` in the implementing [SOCI library](http://soci.sourceforge.net/doc/master/types/) that's why I stuck with that type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r668063753
##########
File path: libminifi/test/sql-tests/mocks/MockODBCService.h
##########
@@ -0,0 +1,66 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <memory>
+#include <string>
+
+#include "MockConnectors.h"
+#include "services/DatabaseService.h"
+#include "core/Resource.h"
+#include "data/DatabaseConnectors.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+namespace controllers {
+
+class MockODBCService : public DatabaseService {
+ public:
+ explicit MockODBCService(const std::string &name, utils::Identifier uuid = utils::Identifier())
+ : DatabaseService(name, uuid),
+ logger_(logging::LoggerFactory<MockODBCService>::getLogger()) {
+ initialize();
+ }
+
+ explicit MockODBCService(const std::string &name, const std::shared_ptr<Configure> &configuration)
+ : DatabaseService(name),
+ logger_(logging::LoggerFactory<MockODBCService>::getLogger()) {
+ setConfiguration(configuration);
+ initialize();
+ }
+
+ std::unique_ptr<sql::Connection> getConnection() const {
+ return std::unique_ptr<sql::Connection>(new MockODBCConnection(connection_string_));
Review comment:
You can use utils::make_unique here, there an implicit converting move constructor from `std::unique_ptr<Derived>&&` to `std::unique_ptr<Base>`
##########
File path: extensions/sql/data/SociConnectors.cpp
##########
@@ -0,0 +1,181 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "SociConnectors.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+void SociRow::setIterator(const soci::rowset<soci::row>::iterator& iter) {
+ current_ = iter;
+}
+
+soci::rowset<soci::row>::iterator SociRow::getIterator() const {
+ return current_;
+}
+
+void SociRow::next() {
+ ++current_;
+}
+
+std::size_t SociRow::size() const {
+ return current_->size();
+}
+
+std::string SociRow::getColumnName(std::size_t index) const {
+ return current_->get_properties(index).get_name();
+}
+
+bool SociRow::isNull(std::size_t index) const {
+ return current_->get_indicator(index) == soci::i_null;
+}
+
+DataType SociRow::getDataType(std::size_t index) const {
+ switch (const auto dataType = current_->get_properties(index).get_data_type()) {
+ case soci::data_type::dt_string: {
+ return DataType::STRING;
+ }
+ case soci::data_type::dt_double: {
+ return DataType::DOUBLE;
+ }
+ case soci::data_type::dt_integer: {
+ return DataType::INTEGER;
+ }
+ case soci::data_type::dt_long_long: {
+ return DataType::LONG_LONG;
+ }
+ case soci::data_type::dt_unsigned_long_long: {
+ return DataType::UNSIGNED_LONG_LONG;
+ }
+ case soci::data_type::dt_date: {
+ return DataType::DATE;
+ }
+ default: {
+ throw minifi::Exception(PROCESSOR_EXCEPTION, "SQLRowsetProcessor: Unsupported data type " + std::to_string(dataType));
+ }
+ }
+}
+
+std::string SociRow::getString(std::size_t index) const {
+ return current_->get<std::string>(index);
+}
+
+double SociRow::getDouble(std::size_t index) const {
+ return current_->get<double>(index);
+}
+
+int SociRow::getInteger(std::size_t index) const {
+ return current_->get<int>(index);
+}
+
+long long SociRow::getLongLong(std::size_t index) const {
+ return current_->get<long long>(index);
+}
+
+unsigned long long SociRow::getUnsignedLongLong(std::size_t index) const {
+ return current_->get<unsigned long long>(index);
+}
+
+std::tm SociRow::getDate(std::size_t index) const {
+ return current_->get<std::tm>(index);
+}
Review comment:
Did you consider making a `get<T>` member function template with no generic implementation and 6 full specializations? I fancy `get<int>` better than `getInt`, but this is really minor, so feel free to ignore if you don't like the idea.
```suggestion
template<> std::string SociRow::get<std::string>(std::size_t index) const {
return current_->get<std::string>(index);
}
template<> double SociRow::get<double>(std::size_t index) const {
return current_->get<double>(index);
}
template<> int SociRow::get<int>(std::size_t index) const {
return current_->get<int>(index);
}
template<> long long SociRow::get<long long>(std::size_t index) const {
return current_->get<long long>(index);
}
template<> unsigned long long SociRow::get<unsigned long long>(std::size_t index) const {
return current_->get<unsigned long long>(index);
}
template<> std::tm SociRow::get<std::tm>(std::size_t index) const {
return current_->get<std::tm>(index);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r644548803
##########
File path: win_build_vs.bat
##########
@@ -65,7 +65,7 @@ for %%x in (%*) do (
mkdir %builddir%
pushd %builddir%\
-cmake -G %generator% -A %build_platform% -DINSTALLER_MERGE_MODULES=%installer_merge_modules% -DTEST_CUSTOM_WEL_PROVIDER=%test_custom_wel_provider% -DENABLE_SQL=%build_SQL% -DCMAKE_BUILD_TYPE_INIT=%cmake_build_type% -DCMAKE_BUILD_TYPE=%cmake_build_type% -DWIN32=WIN32 -DENABLE_LIBRDKAFKA=%build_kafka% -DENABLE_JNI=%build_jni% -DOPENSSL_OFF=OFF -DENABLE_COAP=%build_coap% -DENABLE_AWS=%build_AWS% -DENABLE_PDH=%build_PDH% -DENABLE_AZURE=%build_azure% -DENABLE_SFTP=%build_SFTP% -DUSE_SHARED_LIBS=OFF -DDISABLE_CONTROLLER=ON -DBUILD_ROCKSDB=ON -DFORCE_WINDOWS=ON -DUSE_SYSTEM_UUID=OFF -DDISABLE_LIBARCHIVE=OFF -DDISABLE_SCRIPTING=ON -DEXCLUDE_BOOST=ON -DENABLE_WEL=TRUE -DFAIL_ON_WARNINGS=OFF -DSKIP_TESTS=%skiptests% %strict_gsl_checks% %redist% -DENABLE_LINTER=%build_linter% .. && msbuild /m nifi-minifi-cpp.sln /property:Configuration=%cmake_build_type% /property:Platform=%build_platform% && copy main\%cmake_build_type%\minifi.exe main\
+cmake -G %generator% -A %build_platform% -DINSTALLER_MERGE_MODULES=%installer_merge_modules% -DTEST_CUSTOM_WEL_PROVIDER=%test_custom_wel_provider% -DENABLE_SQL=%build_SQL% -DUSE_REAL_ODBC_TEST_DRIVER=ON -DCMAKE_BUILD_TYPE_INIT=%cmake_build_type% -DCMAKE_BUILD_TYPE=%cmake_build_type% -DWIN32=WIN32 -DENABLE_LIBRDKAFKA=%build_kafka% -DENABLE_JNI=%build_jni% -DOPENSSL_OFF=OFF -DENABLE_COAP=%build_coap% -DENABLE_AWS=%build_AWS% -DENABLE_PDH=%build_PDH% -DENABLE_AZURE=%build_azure% -DENABLE_SFTP=%build_SFTP% -DUSE_SHARED_LIBS=OFF -DDISABLE_CONTROLLER=ON -DBUILD_ROCKSDB=ON -DFORCE_WINDOWS=ON -DUSE_SYSTEM_UUID=OFF -DDISABLE_LIBARCHIVE=OFF -DDISABLE_SCRIPTING=ON -DEXCLUDE_BOOST=ON -DENABLE_WEL=TRUE -DFAIL_ON_WARNINGS=OFF -DSKIP_TESTS=%skiptests% %strict_gsl_checks% %redist% -DENABLE_LINTER=%build_linter% .. && msbuild /m nifi-minifi-cpp.sln /property:Configuration=%cmake_build_type% /property:Platform=%build_platform% && copy main\%cmake_build_type%\minifi.exe main\
Review comment:
as I understand this will always make the tests try to use the real odbc driver on windows, shouldn't we introduce some flag which is only default-enabled in the ci?
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.cpp
##########
@@ -0,0 +1,386 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "MockConnectors.h"
+
+#include <fstream>
+#include <algorithm>
+#include <utility>
+#include <string>
+#include <memory>
+
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+std::size_t MockRow::size() const {
+ return column_names_.size();
+}
+
+std::string MockRow::getColumnName(std::size_t index) const {
+ return column_names_.at(index);
+}
+
+bool MockRow::isNull(std::size_t index) const {
+ return column_values_.at(index) == "NULL";
+}
+
+DataType MockRow::getDataType(std::size_t index) const {
+ return column_types_.at(index);
+}
+
+std::string MockRow::getString(std::size_t index) const {
+ return column_values_.at(index);
+}
+
+double MockRow::getDouble(std::size_t index) const {
+ return std::atof(column_values_.at(index).c_str());
+}
+
+int MockRow::getInteger(std::size_t index) const {
+ return std::atoi(column_values_.at(index).c_str());
+}
+
+long long MockRow::getLongLong(std::size_t index) const { // NOLINT
+ return std::atoll(column_values_.at(index).c_str());
+}
+
+unsigned long long MockRow::getUnsignedLongLong(std::size_t index) const { // NOLINT
+ return static_cast<unsigned long long>(std::atoll(column_values_.at(index).c_str())); // NOLINT
+}
+
+std::tm MockRow::getDate(std::size_t /*index*/) const {
+ throw std::runtime_error("date not implemented");
+}
+
+std::vector<std::string> MockRow::getValues() const {
+ return column_values_;
+}
+
+std::string MockRow::getValue(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return column_values_.at(it-column_names_.begin());
+ }
+ throw std::runtime_error("Unknown column name for getting value");
+}
+
+void MockRowset::addRow(const std::vector<std::string>& column_values) {
+ rows_.emplace_back(column_names_, column_types_, column_values);
+}
+
+void MockRowset::reset() {
+ current_row_ = rows_.begin();
+}
+
+bool MockRowset::is_done() {
+ return current_row_ == rows_.end();
+}
+
+Row& MockRowset::getCurrent() {
+ return *current_row_;
+}
+
+void MockRowset::next() {
+ ++current_row_;
+}
+
+std::vector<std::string> MockRowset::getColumnNames() const {
+ return column_names_;
+}
+
+std::vector<DataType> MockRowset::getColumnTypes() const {
+ return column_types_;
+}
+
+std::vector<MockRow> MockRowset::getRows() const {
+ return rows_;
+}
+
+std::size_t MockRowset::getColumnIndex(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return it-column_names_.begin();
+ }
+ throw std::runtime_error("Unknown column name for getting index");
+}
+
+void MockRowset::sort(const std::string& order_by_col, bool order_ascending) {
+ std::sort(rows_.begin(), rows_.end(), [&](const MockRow& first, const MockRow& second) {
+ if (order_ascending) {
+ return first.getValue(order_by_col) < second.getValue(order_by_col);
+ } else {
+ return first.getValue(order_by_col) > second.getValue(order_by_col);
+ }
+ });
+}
+
+std::unique_ptr<MockRowset> MockRowset::select(const std::vector<std::string>& cols, const std::function<bool(const MockRow&)>& condition, const std::string& order_by_col, bool order_ascending) {
+ if (!order_by_col.empty()) {
+ sort(order_by_col, order_ascending);
+ }
+
+ std::unique_ptr<MockRowset> rowset;
+ if (cols.empty()) {
+ rowset = utils::make_unique<MockRowset>(column_names_, column_types_);
+ } else {
+ std::vector<DataType> col_types;
+ for (const auto& col : cols) {
+ col_types.push_back(column_types_.at(getColumnIndex(col)));
+ }
+ rowset = utils::make_unique<MockRowset>(cols, col_types);
+ }
+
+ std::vector<std::string> used_cols = cols.empty() ? column_names_ : cols;
+ for (const auto& row : rows_) {
+ if (condition(row)) {
+ std::vector<std::string> values;
+ for (const auto& col : used_cols) {
+ values.push_back(row.getValue(col));
+ }
+ rowset->addRow(values);
+ }
+ }
+
+ return rowset;
+}
+
+std::unique_ptr<Rowset> MockDB::execute(const std::string& query, const std::vector<std::string>& args) {
+ if (minifi::utils::StringUtils::startsWith(query, "create table")) {
+ createTable(query);
+ } else if (minifi::utils::StringUtils::startsWith(query, "insert into")) {
+ insertInto(query, args);
+ } else if (minifi::utils::StringUtils::startsWith(query, "select")) {
+ return select(query, args);
+ } else {
+ throw std::runtime_error("Unknown query type");
+ }
+
+ return nullptr;
+}
+
+void MockDB::createTable(const std::string& query) {
+ std::smatch match;
+ std::regex expr("create table (\\w+)\\s*\\((.*)\\);");
+ std::regex_search(query, match, expr);
+ std::string table_name = match[1];
+ auto columns_with_type = minifi::utils::StringUtils::splitAndTrim(match[2], ",");
+ std::vector<std::string> col_names;
+ std::vector<DataType> col_types;
+ for (const auto& col_with_type : columns_with_type) {
+ auto splitted = minifi::utils::StringUtils::splitAndTrim(col_with_type, " ");
+ col_names.push_back(splitted[0]);
+ col_types.push_back(stringToDataType(splitted[1]));
+ }
+ tables_.emplace(table_name, MockRowset{col_names, col_types});
+ storeDb();
+}
+
+void MockDB::insertInto(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("insert into (\\w+)\\s*(\\((.*)\\))*\\s*values\\s*\\((.+)\\)");
+ std::regex_search(replaced_query, match, expr);
+ std::string table_name = match[1];
+ std::vector<std::string> values = minifi::utils::StringUtils::splitAndTrim(match[4], ",");
+ for (auto& value : values) {
+ value = minifi::utils::StringUtils::removeFramingCharacters(value, '\'');
+ }
+ auto insert_col_names = minifi::utils::StringUtils::splitAndTrim(match[3], ",");
+ if (!insert_col_names.empty()) {
+ auto col_names = tables_.at(table_name).getColumnNames();
+ std::vector<std::string> row;
+ for (const auto& col_name : col_names) {
+ auto it = std::find(insert_col_names.begin(), insert_col_names.end(), col_name);
+ if (it != insert_col_names.end()) {
+ row.push_back(values.at(it-insert_col_names.begin()));
+ } else {
+ row.push_back("NULL");
+ }
+ }
+ tables_.at(table_name).addRow(row);
+ } else {
+ tables_.at(table_name).addRow(values);
+ }
+
+ storeDb();
+}
+
+std::unique_ptr<Rowset> MockDB::select(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("select\\s+(.+)\\s+from\\s+(\\w+)\\s*(where ((.+(?= order by))|.+$))*\\s*(order by (.+))*");
+ std::regex_search(replaced_query, match, expr);
+ auto cols = minifi::utils::StringUtils::splitAndTrim(match[1], ",");
+ if (cols[0] == "*") {
+ cols = {};
+ }
+ std::string table_name = match[2];
+ std::string condition_str = match[4];
+ std::function<bool(const MockRow&)> condition;
+ if (!condition_str.empty()) {
+ if (condition_str == "int_col > 103") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) > 103; };
+ } else if (condition_str == "int_col > 102") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) > 102; };
+ } else if (condition_str == "int_col = 11") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) == 11; };
+ } else if (condition_str == "int_col = 11 and text_col = banana") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) == 11 && row.getValue("text_col") == "banana"; };
+ } else {
+ throw std::runtime_error("Unimplemented WHERE condition");
+ }
+ } else {
+ condition = [](const MockRow&){ return true; };
+ }
+ std::string order = match[7];
+ std::string order_col;
+ bool descending = false;
+ if (!order.empty()) {
+ auto order_col_and_sort = minifi::utils::StringUtils::splitAndTrim(order, " ");
+ order_col = order_col_and_sort[0];
+ descending = order_col_and_sort[1] == "desc";
+ }
+ return tables_.at(table_name).select(cols, condition, order_col, !descending);
+}
+
+void MockDB::readDb() {
+ tables_.clear();
+ std::ifstream file(file_path_);
+ std::string line;
+ ParsePhase phase = ParsePhase::NEW_TABLE;
+ std::string table_name;
+ std::vector<std::string> column_names;
+ std::vector<DataType> column_types;
+ while (std::getline(file, line)) {
+ switch (phase) {
+ case ParsePhase::NEW_TABLE: {
+ table_name = minifi::utils::StringUtils::trim(line);
+ phase = ParsePhase::COLUMN_NAMES;
+ break;
+ }
+ case ParsePhase::COLUMN_NAMES: {
+ column_names = minifi::utils::StringUtils::splitAndTrim(line, "|");
+ phase = ParsePhase::COLUMN_TYPES;
+ break;
+ }
+ case ParsePhase::COLUMN_TYPES: {
+ column_types.clear();
+ auto type_strs = minifi::utils::StringUtils::splitAndTrim(line, "|");
+ for (const auto& type : type_strs) {
+ column_types.push_back(stringToDataType(type));
+ }
+ tables_.emplace(table_name, MockRowset(column_names, column_types));
+ phase = ParsePhase::ROW_VALUES;
+ break;
+ }
+ case ParsePhase::ROW_VALUES: {
+ line = minifi::utils::StringUtils::trim(line);
+ if (line.empty()) {
+ phase = ParsePhase::NEW_TABLE;
+ break;
+ }
+ auto cells = minifi::utils::StringUtils::splitAndTrim(line, "|");
+ tables_.at(table_name).addRow(cells);
+ break;
+ }
+ }
+ }
+}
+
+void MockDB::storeDb() {
+ std::ofstream file(file_path_);
+ for (const auto& table : tables_) {
+ file << table.first << std::endl;
+ auto& rowset = table.second;
+ for (const auto& col_name : rowset.getColumnNames()) {
+ file << col_name << "|";
+ }
+ file << std::endl;
+ for (const auto& col_type : rowset.getColumnTypes()) {
+ file << dataTypeToString(col_type) << "|";
+ }
+ file << std::endl;
+ for (const auto& row : rowset.getRows()) {
+ for (const auto& cell : row.getValues()) {
+ file << cell << "|";
+ }
+ file << std::endl;
+ }
+ file << std::endl;
+ }
+}
+
+DataType MockDB::stringToDataType(const std::string& type_str) {
+ if (type_str == "integer") return DataType::INTEGER;
+ if (type_str == "text") return DataType::STRING;
+ if (type_str == "real") return DataType::DOUBLE;
+ throw std::runtime_error("Unimplemented data type");
+}
+
+std::string MockDB::dataTypeToString(DataType data_type) {
+ switch (data_type) {
+ case DataType::INTEGER: return "integer";
+ case DataType::STRING: return "text";
+ case DataType::DOUBLE: return "real";
+ default: throw std::runtime_error("Unimplemented data type");
+ }
+}
+
+std::unique_ptr<Rowset> MockStatement::execute(const std::vector<std::string>& args) {
+ MockDB db(file_path_);
+ return db.execute(query_, args);
+}
+
+MockODBCConnection::MockODBCConnection(std::string connectionString)
+ : connection_string_(std::move(connectionString)) {
+ std::smatch match;
+ std::regex expr("Database=(.*)(;|$)");
Review comment:
not clear whether we need it or not, but there is a `Regex` class in `RegexUtils.h`, aiming to address GCC<4.9 compilers, should we use that here as well?
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.cpp
##########
@@ -0,0 +1,386 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "MockConnectors.h"
+
+#include <fstream>
+#include <algorithm>
+#include <utility>
+#include <string>
+#include <memory>
+
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+std::size_t MockRow::size() const {
+ return column_names_.size();
+}
+
+std::string MockRow::getColumnName(std::size_t index) const {
+ return column_names_.at(index);
+}
+
+bool MockRow::isNull(std::size_t index) const {
+ return column_values_.at(index) == "NULL";
+}
+
+DataType MockRow::getDataType(std::size_t index) const {
+ return column_types_.at(index);
+}
+
+std::string MockRow::getString(std::size_t index) const {
+ return column_values_.at(index);
+}
+
+double MockRow::getDouble(std::size_t index) const {
+ return std::atof(column_values_.at(index).c_str());
+}
+
+int MockRow::getInteger(std::size_t index) const {
+ return std::atoi(column_values_.at(index).c_str());
+}
+
+long long MockRow::getLongLong(std::size_t index) const { // NOLINT
+ return std::atoll(column_values_.at(index).c_str());
+}
+
+unsigned long long MockRow::getUnsignedLongLong(std::size_t index) const { // NOLINT
+ return static_cast<unsigned long long>(std::atoll(column_values_.at(index).c_str())); // NOLINT
+}
+
+std::tm MockRow::getDate(std::size_t /*index*/) const {
+ throw std::runtime_error("date not implemented");
+}
+
+std::vector<std::string> MockRow::getValues() const {
+ return column_values_;
+}
+
+std::string MockRow::getValue(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return column_values_.at(it-column_names_.begin());
+ }
+ throw std::runtime_error("Unknown column name for getting value");
+}
+
+void MockRowset::addRow(const std::vector<std::string>& column_values) {
+ rows_.emplace_back(column_names_, column_types_, column_values);
+}
+
+void MockRowset::reset() {
+ current_row_ = rows_.begin();
+}
+
+bool MockRowset::is_done() {
+ return current_row_ == rows_.end();
+}
+
+Row& MockRowset::getCurrent() {
+ return *current_row_;
+}
+
+void MockRowset::next() {
+ ++current_row_;
+}
+
+std::vector<std::string> MockRowset::getColumnNames() const {
+ return column_names_;
+}
+
+std::vector<DataType> MockRowset::getColumnTypes() const {
+ return column_types_;
+}
+
+std::vector<MockRow> MockRowset::getRows() const {
+ return rows_;
+}
+
+std::size_t MockRowset::getColumnIndex(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return it-column_names_.begin();
+ }
+ throw std::runtime_error("Unknown column name for getting index");
+}
+
+void MockRowset::sort(const std::string& order_by_col, bool order_ascending) {
+ std::sort(rows_.begin(), rows_.end(), [&](const MockRow& first, const MockRow& second) {
+ if (order_ascending) {
+ return first.getValue(order_by_col) < second.getValue(order_by_col);
+ } else {
+ return first.getValue(order_by_col) > second.getValue(order_by_col);
+ }
+ });
+}
+
+std::unique_ptr<MockRowset> MockRowset::select(const std::vector<std::string>& cols, const std::function<bool(const MockRow&)>& condition, const std::string& order_by_col, bool order_ascending) {
+ if (!order_by_col.empty()) {
+ sort(order_by_col, order_ascending);
+ }
+
+ std::unique_ptr<MockRowset> rowset;
+ if (cols.empty()) {
+ rowset = utils::make_unique<MockRowset>(column_names_, column_types_);
+ } else {
+ std::vector<DataType> col_types;
+ for (const auto& col : cols) {
+ col_types.push_back(column_types_.at(getColumnIndex(col)));
+ }
+ rowset = utils::make_unique<MockRowset>(cols, col_types);
+ }
+
+ std::vector<std::string> used_cols = cols.empty() ? column_names_ : cols;
+ for (const auto& row : rows_) {
+ if (condition(row)) {
+ std::vector<std::string> values;
+ for (const auto& col : used_cols) {
+ values.push_back(row.getValue(col));
+ }
+ rowset->addRow(values);
+ }
+ }
+
+ return rowset;
+}
+
+std::unique_ptr<Rowset> MockDB::execute(const std::string& query, const std::vector<std::string>& args) {
+ if (minifi::utils::StringUtils::startsWith(query, "create table")) {
+ createTable(query);
+ } else if (minifi::utils::StringUtils::startsWith(query, "insert into")) {
+ insertInto(query, args);
+ } else if (minifi::utils::StringUtils::startsWith(query, "select")) {
+ return select(query, args);
+ } else {
+ throw std::runtime_error("Unknown query type");
+ }
+
+ return nullptr;
+}
+
+void MockDB::createTable(const std::string& query) {
+ std::smatch match;
+ std::regex expr("create table (\\w+)\\s*\\((.*)\\);");
+ std::regex_search(query, match, expr);
+ std::string table_name = match[1];
+ auto columns_with_type = minifi::utils::StringUtils::splitAndTrim(match[2], ",");
+ std::vector<std::string> col_names;
+ std::vector<DataType> col_types;
+ for (const auto& col_with_type : columns_with_type) {
+ auto splitted = minifi::utils::StringUtils::splitAndTrim(col_with_type, " ");
+ col_names.push_back(splitted[0]);
+ col_types.push_back(stringToDataType(splitted[1]));
+ }
+ tables_.emplace(table_name, MockRowset{col_names, col_types});
+ storeDb();
+}
+
+void MockDB::insertInto(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("insert into (\\w+)\\s*(\\((.*)\\))*\\s*values\\s*\\((.+)\\)");
+ std::regex_search(replaced_query, match, expr);
+ std::string table_name = match[1];
+ std::vector<std::string> values = minifi::utils::StringUtils::splitAndTrim(match[4], ",");
+ for (auto& value : values) {
+ value = minifi::utils::StringUtils::removeFramingCharacters(value, '\'');
+ }
+ auto insert_col_names = minifi::utils::StringUtils::splitAndTrim(match[3], ",");
+ if (!insert_col_names.empty()) {
+ auto col_names = tables_.at(table_name).getColumnNames();
+ std::vector<std::string> row;
+ for (const auto& col_name : col_names) {
+ auto it = std::find(insert_col_names.begin(), insert_col_names.end(), col_name);
+ if (it != insert_col_names.end()) {
+ row.push_back(values.at(it-insert_col_names.begin()));
+ } else {
+ row.push_back("NULL");
+ }
+ }
+ tables_.at(table_name).addRow(row);
+ } else {
+ tables_.at(table_name).addRow(values);
+ }
+
+ storeDb();
+}
+
+std::unique_ptr<Rowset> MockDB::select(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("select\\s+(.+)\\s+from\\s+(\\w+)\\s*(where ((.+(?= order by))|.+$))*\\s*(order by (.+))*");
+ std::regex_search(replaced_query, match, expr);
+ auto cols = minifi::utils::StringUtils::splitAndTrim(match[1], ",");
+ if (cols[0] == "*") {
+ cols = {};
+ }
+ std::string table_name = match[2];
+ std::string condition_str = match[4];
+ std::function<bool(const MockRow&)> condition;
+ if (!condition_str.empty()) {
+ if (condition_str == "int_col > 103") {
Review comment:
I feel like we should go for a different solution than hard-coded conditions, I agree that writing an expression parser would complicate things a lot, could we reuse some expression parsing utility from our codebase? (e.g. from Expression.cpp)
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.cpp
##########
@@ -0,0 +1,386 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "MockConnectors.h"
+
+#include <fstream>
+#include <algorithm>
+#include <utility>
+#include <string>
+#include <memory>
+
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+std::size_t MockRow::size() const {
+ return column_names_.size();
+}
+
+std::string MockRow::getColumnName(std::size_t index) const {
+ return column_names_.at(index);
+}
+
+bool MockRow::isNull(std::size_t index) const {
+ return column_values_.at(index) == "NULL";
+}
+
+DataType MockRow::getDataType(std::size_t index) const {
+ return column_types_.at(index);
+}
+
+std::string MockRow::getString(std::size_t index) const {
+ return column_values_.at(index);
+}
+
+double MockRow::getDouble(std::size_t index) const {
+ return std::atof(column_values_.at(index).c_str());
+}
+
+int MockRow::getInteger(std::size_t index) const {
+ return std::atoi(column_values_.at(index).c_str());
Review comment:
should we use the `std::string` accepting version? (`std::stoi` and others)
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.cpp
##########
@@ -0,0 +1,386 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "MockConnectors.h"
+
+#include <fstream>
+#include <algorithm>
+#include <utility>
+#include <string>
+#include <memory>
+
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+std::size_t MockRow::size() const {
+ return column_names_.size();
+}
+
+std::string MockRow::getColumnName(std::size_t index) const {
+ return column_names_.at(index);
+}
+
+bool MockRow::isNull(std::size_t index) const {
+ return column_values_.at(index) == "NULL";
+}
+
+DataType MockRow::getDataType(std::size_t index) const {
+ return column_types_.at(index);
+}
+
+std::string MockRow::getString(std::size_t index) const {
+ return column_values_.at(index);
+}
+
+double MockRow::getDouble(std::size_t index) const {
+ return std::atof(column_values_.at(index).c_str());
+}
+
+int MockRow::getInteger(std::size_t index) const {
+ return std::atoi(column_values_.at(index).c_str());
+}
+
+long long MockRow::getLongLong(std::size_t index) const { // NOLINT
+ return std::atoll(column_values_.at(index).c_str());
+}
+
+unsigned long long MockRow::getUnsignedLongLong(std::size_t index) const { // NOLINT
+ return static_cast<unsigned long long>(std::atoll(column_values_.at(index).c_str())); // NOLINT
+}
+
+std::tm MockRow::getDate(std::size_t /*index*/) const {
+ throw std::runtime_error("date not implemented");
+}
+
+std::vector<std::string> MockRow::getValues() const {
+ return column_values_;
+}
+
+std::string MockRow::getValue(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return column_values_.at(it-column_names_.begin());
+ }
+ throw std::runtime_error("Unknown column name for getting value");
+}
+
+void MockRowset::addRow(const std::vector<std::string>& column_values) {
+ rows_.emplace_back(column_names_, column_types_, column_values);
+}
+
+void MockRowset::reset() {
+ current_row_ = rows_.begin();
+}
+
+bool MockRowset::is_done() {
+ return current_row_ == rows_.end();
+}
+
+Row& MockRowset::getCurrent() {
+ return *current_row_;
+}
+
+void MockRowset::next() {
+ ++current_row_;
+}
+
+std::vector<std::string> MockRowset::getColumnNames() const {
+ return column_names_;
+}
+
+std::vector<DataType> MockRowset::getColumnTypes() const {
+ return column_types_;
+}
+
+std::vector<MockRow> MockRowset::getRows() const {
+ return rows_;
+}
+
+std::size_t MockRowset::getColumnIndex(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return it-column_names_.begin();
+ }
+ throw std::runtime_error("Unknown column name for getting index");
+}
+
+void MockRowset::sort(const std::string& order_by_col, bool order_ascending) {
+ std::sort(rows_.begin(), rows_.end(), [&](const MockRow& first, const MockRow& second) {
+ if (order_ascending) {
+ return first.getValue(order_by_col) < second.getValue(order_by_col);
+ } else {
+ return first.getValue(order_by_col) > second.getValue(order_by_col);
+ }
+ });
+}
+
+std::unique_ptr<MockRowset> MockRowset::select(const std::vector<std::string>& cols, const std::function<bool(const MockRow&)>& condition, const std::string& order_by_col, bool order_ascending) {
+ if (!order_by_col.empty()) {
+ sort(order_by_col, order_ascending);
+ }
+
+ std::unique_ptr<MockRowset> rowset;
+ if (cols.empty()) {
+ rowset = utils::make_unique<MockRowset>(column_names_, column_types_);
+ } else {
+ std::vector<DataType> col_types;
+ for (const auto& col : cols) {
+ col_types.push_back(column_types_.at(getColumnIndex(col)));
+ }
+ rowset = utils::make_unique<MockRowset>(cols, col_types);
+ }
+
+ std::vector<std::string> used_cols = cols.empty() ? column_names_ : cols;
+ for (const auto& row : rows_) {
+ if (condition(row)) {
+ std::vector<std::string> values;
+ for (const auto& col : used_cols) {
+ values.push_back(row.getValue(col));
+ }
+ rowset->addRow(values);
+ }
+ }
+
+ return rowset;
+}
+
+std::unique_ptr<Rowset> MockDB::execute(const std::string& query, const std::vector<std::string>& args) {
+ if (minifi::utils::StringUtils::startsWith(query, "create table")) {
+ createTable(query);
+ } else if (minifi::utils::StringUtils::startsWith(query, "insert into")) {
+ insertInto(query, args);
+ } else if (minifi::utils::StringUtils::startsWith(query, "select")) {
+ return select(query, args);
+ } else {
+ throw std::runtime_error("Unknown query type");
+ }
+
+ return nullptr;
+}
+
+void MockDB::createTable(const std::string& query) {
+ std::smatch match;
+ std::regex expr("create table (\\w+)\\s*\\((.*)\\);");
+ std::regex_search(query, match, expr);
+ std::string table_name = match[1];
+ auto columns_with_type = minifi::utils::StringUtils::splitAndTrim(match[2], ",");
+ std::vector<std::string> col_names;
+ std::vector<DataType> col_types;
+ for (const auto& col_with_type : columns_with_type) {
+ auto splitted = minifi::utils::StringUtils::splitAndTrim(col_with_type, " ");
+ col_names.push_back(splitted[0]);
+ col_types.push_back(stringToDataType(splitted[1]));
+ }
+ tables_.emplace(table_name, MockRowset{col_names, col_types});
+ storeDb();
+}
+
+void MockDB::insertInto(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("insert into (\\w+)\\s*(\\((.*)\\))*\\s*values\\s*\\((.+)\\)");
+ std::regex_search(replaced_query, match, expr);
+ std::string table_name = match[1];
+ std::vector<std::string> values = minifi::utils::StringUtils::splitAndTrim(match[4], ",");
+ for (auto& value : values) {
+ value = minifi::utils::StringUtils::removeFramingCharacters(value, '\'');
+ }
+ auto insert_col_names = minifi::utils::StringUtils::splitAndTrim(match[3], ",");
+ if (!insert_col_names.empty()) {
+ auto col_names = tables_.at(table_name).getColumnNames();
+ std::vector<std::string> row;
+ for (const auto& col_name : col_names) {
+ auto it = std::find(insert_col_names.begin(), insert_col_names.end(), col_name);
+ if (it != insert_col_names.end()) {
+ row.push_back(values.at(it-insert_col_names.begin()));
+ } else {
+ row.push_back("NULL");
+ }
+ }
+ tables_.at(table_name).addRow(row);
+ } else {
+ tables_.at(table_name).addRow(values);
+ }
+
+ storeDb();
+}
+
+std::unique_ptr<Rowset> MockDB::select(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("select\\s+(.+)\\s+from\\s+(\\w+)\\s*(where ((.+(?= order by))|.+$))*\\s*(order by (.+))*");
+ std::regex_search(replaced_query, match, expr);
+ auto cols = minifi::utils::StringUtils::splitAndTrim(match[1], ",");
+ if (cols[0] == "*") {
+ cols = {};
+ }
+ std::string table_name = match[2];
+ std::string condition_str = match[4];
+ std::function<bool(const MockRow&)> condition;
+ if (!condition_str.empty()) {
+ if (condition_str == "int_col > 103") {
Review comment:
maybe we could cover our test use-cases with some well-crafted regex-es (and maybe calling further regex-es on submatches if need be)
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.h
##########
@@ -0,0 +1,175 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <regex>
+#include <map>
+#include <vector>
+#include <algorithm>
+#include <memory>
+#include <string>
+
+#include "data/DatabaseConnectors.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+class MockRow : public Row {
+ public:
+ MockRow(std::vector<std::string>& column_names, std::vector<DataType>& column_types, const std::vector<std::string>& column_values)
+ : column_names_(column_names), column_types_(column_types), column_values_(column_values) {
+ }
+
+ MockRow(MockRow&& other) = default;
+ MockRow(const MockRow& other) = default;
+ MockRow& operator=(MockRow&& other) {
+ column_names_ = other.column_names_;
+ column_types_ = other.column_types_;
Review comment:
I don't know the exact intention here, but I wager it won't be doing that
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.h
##########
@@ -0,0 +1,175 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <regex>
+#include <map>
+#include <vector>
+#include <algorithm>
+#include <memory>
+#include <string>
+
+#include "data/DatabaseConnectors.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+class MockRow : public Row {
+ public:
+ MockRow(std::vector<std::string>& column_names, std::vector<DataType>& column_types, const std::vector<std::string>& column_values)
+ : column_names_(column_names), column_types_(column_types), column_values_(column_values) {
+ }
+
+ MockRow(MockRow&& other) = default;
+ MockRow(const MockRow& other) = default;
+ MockRow& operator=(MockRow&& other) {
+ column_names_ = other.column_names_;
+ column_types_ = other.column_types_;
Review comment:
using `std::reference_wrapper` would most likely offer the semantics we are looking for here
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.h
##########
@@ -0,0 +1,175 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <regex>
+#include <map>
+#include <vector>
+#include <algorithm>
+#include <memory>
+#include <string>
+
+#include "data/DatabaseConnectors.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+class MockRow : public Row {
+ public:
+ MockRow(std::vector<std::string>& column_names, std::vector<DataType>& column_types, const std::vector<std::string>& column_values)
+ : column_names_(column_names), column_types_(column_types), column_values_(column_values) {
+ }
+
+ MockRow(MockRow&& other) = default;
+ MockRow(const MockRow& other) = default;
+ MockRow& operator=(MockRow&& other) {
+ column_names_ = other.column_names_;
+ column_types_ = other.column_types_;
Review comment:
also won't the lack of `noexcept` make it unavailable for `std::vector`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r644579725
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.cpp
##########
@@ -0,0 +1,386 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "MockConnectors.h"
+
+#include <fstream>
+#include <algorithm>
+#include <utility>
+#include <string>
+#include <memory>
+
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+std::size_t MockRow::size() const {
+ return column_names_.size();
+}
+
+std::string MockRow::getColumnName(std::size_t index) const {
+ return column_names_.at(index);
+}
+
+bool MockRow::isNull(std::size_t index) const {
+ return column_values_.at(index) == "NULL";
+}
+
+DataType MockRow::getDataType(std::size_t index) const {
+ return column_types_.at(index);
+}
+
+std::string MockRow::getString(std::size_t index) const {
+ return column_values_.at(index);
+}
+
+double MockRow::getDouble(std::size_t index) const {
+ return std::atof(column_values_.at(index).c_str());
+}
+
+int MockRow::getInteger(std::size_t index) const {
+ return std::atoi(column_values_.at(index).c_str());
+}
+
+long long MockRow::getLongLong(std::size_t index) const { // NOLINT
+ return std::atoll(column_values_.at(index).c_str());
+}
+
+unsigned long long MockRow::getUnsignedLongLong(std::size_t index) const { // NOLINT
+ return static_cast<unsigned long long>(std::atoll(column_values_.at(index).c_str())); // NOLINT
+}
+
+std::tm MockRow::getDate(std::size_t /*index*/) const {
+ throw std::runtime_error("date not implemented");
+}
+
+std::vector<std::string> MockRow::getValues() const {
+ return column_values_;
+}
+
+std::string MockRow::getValue(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return column_values_.at(it-column_names_.begin());
+ }
+ throw std::runtime_error("Unknown column name for getting value");
+}
+
+void MockRowset::addRow(const std::vector<std::string>& column_values) {
+ rows_.emplace_back(column_names_, column_types_, column_values);
+}
+
+void MockRowset::reset() {
+ current_row_ = rows_.begin();
+}
+
+bool MockRowset::is_done() {
+ return current_row_ == rows_.end();
+}
+
+Row& MockRowset::getCurrent() {
+ return *current_row_;
+}
+
+void MockRowset::next() {
+ ++current_row_;
+}
+
+std::vector<std::string> MockRowset::getColumnNames() const {
+ return column_names_;
+}
+
+std::vector<DataType> MockRowset::getColumnTypes() const {
+ return column_types_;
+}
+
+std::vector<MockRow> MockRowset::getRows() const {
+ return rows_;
+}
+
+std::size_t MockRowset::getColumnIndex(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return it-column_names_.begin();
+ }
+ throw std::runtime_error("Unknown column name for getting index");
+}
+
+void MockRowset::sort(const std::string& order_by_col, bool order_ascending) {
+ std::sort(rows_.begin(), rows_.end(), [&](const MockRow& first, const MockRow& second) {
+ if (order_ascending) {
+ return first.getValue(order_by_col) < second.getValue(order_by_col);
+ } else {
+ return first.getValue(order_by_col) > second.getValue(order_by_col);
+ }
+ });
+}
+
+std::unique_ptr<MockRowset> MockRowset::select(const std::vector<std::string>& cols, const std::function<bool(const MockRow&)>& condition, const std::string& order_by_col, bool order_ascending) {
+ if (!order_by_col.empty()) {
+ sort(order_by_col, order_ascending);
+ }
+
+ std::unique_ptr<MockRowset> rowset;
+ if (cols.empty()) {
+ rowset = utils::make_unique<MockRowset>(column_names_, column_types_);
+ } else {
+ std::vector<DataType> col_types;
+ for (const auto& col : cols) {
+ col_types.push_back(column_types_.at(getColumnIndex(col)));
+ }
+ rowset = utils::make_unique<MockRowset>(cols, col_types);
+ }
+
+ std::vector<std::string> used_cols = cols.empty() ? column_names_ : cols;
+ for (const auto& row : rows_) {
+ if (condition(row)) {
+ std::vector<std::string> values;
+ for (const auto& col : used_cols) {
+ values.push_back(row.getValue(col));
+ }
+ rowset->addRow(values);
+ }
+ }
+
+ return rowset;
+}
+
+std::unique_ptr<Rowset> MockDB::execute(const std::string& query, const std::vector<std::string>& args) {
+ if (minifi::utils::StringUtils::startsWith(query, "create table")) {
+ createTable(query);
+ } else if (minifi::utils::StringUtils::startsWith(query, "insert into")) {
+ insertInto(query, args);
+ } else if (minifi::utils::StringUtils::startsWith(query, "select")) {
+ return select(query, args);
+ } else {
+ throw std::runtime_error("Unknown query type");
+ }
+
+ return nullptr;
+}
+
+void MockDB::createTable(const std::string& query) {
+ std::smatch match;
+ std::regex expr("create table (\\w+)\\s*\\((.*)\\);");
+ std::regex_search(query, match, expr);
+ std::string table_name = match[1];
+ auto columns_with_type = minifi::utils::StringUtils::splitAndTrim(match[2], ",");
+ std::vector<std::string> col_names;
+ std::vector<DataType> col_types;
+ for (const auto& col_with_type : columns_with_type) {
+ auto splitted = minifi::utils::StringUtils::splitAndTrim(col_with_type, " ");
+ col_names.push_back(splitted[0]);
+ col_types.push_back(stringToDataType(splitted[1]));
+ }
+ tables_.emplace(table_name, MockRowset{col_names, col_types});
+ storeDb();
+}
+
+void MockDB::insertInto(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("insert into (\\w+)\\s*(\\((.*)\\))*\\s*values\\s*\\((.+)\\)");
+ std::regex_search(replaced_query, match, expr);
+ std::string table_name = match[1];
+ std::vector<std::string> values = minifi::utils::StringUtils::splitAndTrim(match[4], ",");
+ for (auto& value : values) {
+ value = minifi::utils::StringUtils::removeFramingCharacters(value, '\'');
+ }
+ auto insert_col_names = minifi::utils::StringUtils::splitAndTrim(match[3], ",");
+ if (!insert_col_names.empty()) {
+ auto col_names = tables_.at(table_name).getColumnNames();
+ std::vector<std::string> row;
+ for (const auto& col_name : col_names) {
+ auto it = std::find(insert_col_names.begin(), insert_col_names.end(), col_name);
+ if (it != insert_col_names.end()) {
+ row.push_back(values.at(it-insert_col_names.begin()));
+ } else {
+ row.push_back("NULL");
+ }
+ }
+ tables_.at(table_name).addRow(row);
+ } else {
+ tables_.at(table_name).addRow(values);
+ }
+
+ storeDb();
+}
+
+std::unique_ptr<Rowset> MockDB::select(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("select\\s+(.+)\\s+from\\s+(\\w+)\\s*(where ((.+(?= order by))|.+$))*\\s*(order by (.+))*");
+ std::regex_search(replaced_query, match, expr);
+ auto cols = minifi::utils::StringUtils::splitAndTrim(match[1], ",");
+ if (cols[0] == "*") {
+ cols = {};
+ }
+ std::string table_name = match[2];
+ std::string condition_str = match[4];
+ std::function<bool(const MockRow&)> condition;
+ if (!condition_str.empty()) {
+ if (condition_str == "int_col > 103") {
Review comment:
I feel like we should go for a different solution than hard-coded conditions, I agree that writing an expression parser would complicate things a lot, could we reuse some expression parsing utility from our codebase? (e.g. from Expression.cpp)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r644555388
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.cpp
##########
@@ -0,0 +1,386 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "MockConnectors.h"
+
+#include <fstream>
+#include <algorithm>
+#include <utility>
+#include <string>
+#include <memory>
+
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+std::size_t MockRow::size() const {
+ return column_names_.size();
+}
+
+std::string MockRow::getColumnName(std::size_t index) const {
+ return column_names_.at(index);
+}
+
+bool MockRow::isNull(std::size_t index) const {
+ return column_values_.at(index) == "NULL";
+}
+
+DataType MockRow::getDataType(std::size_t index) const {
+ return column_types_.at(index);
+}
+
+std::string MockRow::getString(std::size_t index) const {
+ return column_values_.at(index);
+}
+
+double MockRow::getDouble(std::size_t index) const {
+ return std::atof(column_values_.at(index).c_str());
+}
+
+int MockRow::getInteger(std::size_t index) const {
+ return std::atoi(column_values_.at(index).c_str());
+}
+
+long long MockRow::getLongLong(std::size_t index) const { // NOLINT
+ return std::atoll(column_values_.at(index).c_str());
+}
+
+unsigned long long MockRow::getUnsignedLongLong(std::size_t index) const { // NOLINT
+ return static_cast<unsigned long long>(std::atoll(column_values_.at(index).c_str())); // NOLINT
+}
+
+std::tm MockRow::getDate(std::size_t /*index*/) const {
+ throw std::runtime_error("date not implemented");
+}
+
+std::vector<std::string> MockRow::getValues() const {
+ return column_values_;
+}
+
+std::string MockRow::getValue(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return column_values_.at(it-column_names_.begin());
+ }
+ throw std::runtime_error("Unknown column name for getting value");
+}
+
+void MockRowset::addRow(const std::vector<std::string>& column_values) {
+ rows_.emplace_back(column_names_, column_types_, column_values);
+}
+
+void MockRowset::reset() {
+ current_row_ = rows_.begin();
+}
+
+bool MockRowset::is_done() {
+ return current_row_ == rows_.end();
+}
+
+Row& MockRowset::getCurrent() {
+ return *current_row_;
+}
+
+void MockRowset::next() {
+ ++current_row_;
+}
+
+std::vector<std::string> MockRowset::getColumnNames() const {
+ return column_names_;
+}
+
+std::vector<DataType> MockRowset::getColumnTypes() const {
+ return column_types_;
+}
+
+std::vector<MockRow> MockRowset::getRows() const {
+ return rows_;
+}
+
+std::size_t MockRowset::getColumnIndex(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return it-column_names_.begin();
+ }
+ throw std::runtime_error("Unknown column name for getting index");
+}
+
+void MockRowset::sort(const std::string& order_by_col, bool order_ascending) {
+ std::sort(rows_.begin(), rows_.end(), [&](const MockRow& first, const MockRow& second) {
+ if (order_ascending) {
+ return first.getValue(order_by_col) < second.getValue(order_by_col);
+ } else {
+ return first.getValue(order_by_col) > second.getValue(order_by_col);
+ }
+ });
+}
+
+std::unique_ptr<MockRowset> MockRowset::select(const std::vector<std::string>& cols, const std::function<bool(const MockRow&)>& condition, const std::string& order_by_col, bool order_ascending) {
+ if (!order_by_col.empty()) {
+ sort(order_by_col, order_ascending);
+ }
+
+ std::unique_ptr<MockRowset> rowset;
+ if (cols.empty()) {
+ rowset = utils::make_unique<MockRowset>(column_names_, column_types_);
+ } else {
+ std::vector<DataType> col_types;
+ for (const auto& col : cols) {
+ col_types.push_back(column_types_.at(getColumnIndex(col)));
+ }
+ rowset = utils::make_unique<MockRowset>(cols, col_types);
+ }
+
+ std::vector<std::string> used_cols = cols.empty() ? column_names_ : cols;
+ for (const auto& row : rows_) {
+ if (condition(row)) {
+ std::vector<std::string> values;
+ for (const auto& col : used_cols) {
+ values.push_back(row.getValue(col));
+ }
+ rowset->addRow(values);
+ }
+ }
+
+ return rowset;
+}
+
+std::unique_ptr<Rowset> MockDB::execute(const std::string& query, const std::vector<std::string>& args) {
+ if (minifi::utils::StringUtils::startsWith(query, "create table")) {
+ createTable(query);
+ } else if (minifi::utils::StringUtils::startsWith(query, "insert into")) {
+ insertInto(query, args);
+ } else if (minifi::utils::StringUtils::startsWith(query, "select")) {
+ return select(query, args);
+ } else {
+ throw std::runtime_error("Unknown query type");
+ }
+
+ return nullptr;
+}
+
+void MockDB::createTable(const std::string& query) {
+ std::smatch match;
+ std::regex expr("create table (\\w+)\\s*\\((.*)\\);");
+ std::regex_search(query, match, expr);
+ std::string table_name = match[1];
+ auto columns_with_type = minifi::utils::StringUtils::splitAndTrim(match[2], ",");
+ std::vector<std::string> col_names;
+ std::vector<DataType> col_types;
+ for (const auto& col_with_type : columns_with_type) {
+ auto splitted = minifi::utils::StringUtils::splitAndTrim(col_with_type, " ");
+ col_names.push_back(splitted[0]);
+ col_types.push_back(stringToDataType(splitted[1]));
+ }
+ tables_.emplace(table_name, MockRowset{col_names, col_types});
+ storeDb();
+}
+
+void MockDB::insertInto(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("insert into (\\w+)\\s*(\\((.*)\\))*\\s*values\\s*\\((.+)\\)");
+ std::regex_search(replaced_query, match, expr);
+ std::string table_name = match[1];
+ std::vector<std::string> values = minifi::utils::StringUtils::splitAndTrim(match[4], ",");
+ for (auto& value : values) {
+ value = minifi::utils::StringUtils::removeFramingCharacters(value, '\'');
+ }
+ auto insert_col_names = minifi::utils::StringUtils::splitAndTrim(match[3], ",");
+ if (!insert_col_names.empty()) {
+ auto col_names = tables_.at(table_name).getColumnNames();
+ std::vector<std::string> row;
+ for (const auto& col_name : col_names) {
+ auto it = std::find(insert_col_names.begin(), insert_col_names.end(), col_name);
+ if (it != insert_col_names.end()) {
+ row.push_back(values.at(it-insert_col_names.begin()));
+ } else {
+ row.push_back("NULL");
+ }
+ }
+ tables_.at(table_name).addRow(row);
+ } else {
+ tables_.at(table_name).addRow(values);
+ }
+
+ storeDb();
+}
+
+std::unique_ptr<Rowset> MockDB::select(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("select\\s+(.+)\\s+from\\s+(\\w+)\\s*(where ((.+(?= order by))|.+$))*\\s*(order by (.+))*");
+ std::regex_search(replaced_query, match, expr);
+ auto cols = minifi::utils::StringUtils::splitAndTrim(match[1], ",");
+ if (cols[0] == "*") {
+ cols = {};
+ }
+ std::string table_name = match[2];
+ std::string condition_str = match[4];
+ std::function<bool(const MockRow&)> condition;
+ if (!condition_str.empty()) {
+ if (condition_str == "int_col > 103") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) > 103; };
+ } else if (condition_str == "int_col > 102") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) > 102; };
+ } else if (condition_str == "int_col = 11") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) == 11; };
+ } else if (condition_str == "int_col = 11 and text_col = banana") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) == 11 && row.getValue("text_col") == "banana"; };
+ } else {
+ throw std::runtime_error("Unimplemented WHERE condition");
+ }
+ } else {
+ condition = [](const MockRow&){ return true; };
+ }
+ std::string order = match[7];
+ std::string order_col;
+ bool descending = false;
+ if (!order.empty()) {
+ auto order_col_and_sort = minifi::utils::StringUtils::splitAndTrim(order, " ");
+ order_col = order_col_and_sort[0];
+ descending = order_col_and_sort[1] == "desc";
+ }
+ return tables_.at(table_name).select(cols, condition, order_col, !descending);
+}
+
+void MockDB::readDb() {
+ tables_.clear();
+ std::ifstream file(file_path_);
+ std::string line;
+ ParsePhase phase = ParsePhase::NEW_TABLE;
+ std::string table_name;
+ std::vector<std::string> column_names;
+ std::vector<DataType> column_types;
+ while (std::getline(file, line)) {
+ switch (phase) {
+ case ParsePhase::NEW_TABLE: {
+ table_name = minifi::utils::StringUtils::trim(line);
+ phase = ParsePhase::COLUMN_NAMES;
+ break;
+ }
+ case ParsePhase::COLUMN_NAMES: {
+ column_names = minifi::utils::StringUtils::splitAndTrim(line, "|");
+ phase = ParsePhase::COLUMN_TYPES;
+ break;
+ }
+ case ParsePhase::COLUMN_TYPES: {
+ column_types.clear();
+ auto type_strs = minifi::utils::StringUtils::splitAndTrim(line, "|");
+ for (const auto& type : type_strs) {
+ column_types.push_back(stringToDataType(type));
+ }
+ tables_.emplace(table_name, MockRowset(column_names, column_types));
+ phase = ParsePhase::ROW_VALUES;
+ break;
+ }
+ case ParsePhase::ROW_VALUES: {
+ line = minifi::utils::StringUtils::trim(line);
+ if (line.empty()) {
+ phase = ParsePhase::NEW_TABLE;
+ break;
+ }
+ auto cells = minifi::utils::StringUtils::splitAndTrim(line, "|");
+ tables_.at(table_name).addRow(cells);
+ break;
+ }
+ }
+ }
+}
+
+void MockDB::storeDb() {
+ std::ofstream file(file_path_);
+ for (const auto& table : tables_) {
+ file << table.first << std::endl;
+ auto& rowset = table.second;
+ for (const auto& col_name : rowset.getColumnNames()) {
+ file << col_name << "|";
+ }
+ file << std::endl;
+ for (const auto& col_type : rowset.getColumnTypes()) {
+ file << dataTypeToString(col_type) << "|";
+ }
+ file << std::endl;
+ for (const auto& row : rowset.getRows()) {
+ for (const auto& cell : row.getValues()) {
+ file << cell << "|";
+ }
+ file << std::endl;
+ }
+ file << std::endl;
+ }
+}
+
+DataType MockDB::stringToDataType(const std::string& type_str) {
+ if (type_str == "integer") return DataType::INTEGER;
+ if (type_str == "text") return DataType::STRING;
+ if (type_str == "real") return DataType::DOUBLE;
+ throw std::runtime_error("Unimplemented data type");
+}
+
+std::string MockDB::dataTypeToString(DataType data_type) {
+ switch (data_type) {
+ case DataType::INTEGER: return "integer";
+ case DataType::STRING: return "text";
+ case DataType::DOUBLE: return "real";
+ default: throw std::runtime_error("Unimplemented data type");
+ }
+}
+
+std::unique_ptr<Rowset> MockStatement::execute(const std::vector<std::string>& args) {
+ MockDB db(file_path_);
+ return db.execute(query_, args);
+}
+
+MockODBCConnection::MockODBCConnection(std::string connectionString)
+ : connection_string_(std::move(connectionString)) {
+ std::smatch match;
+ std::regex expr("Database=(.*)(;|$)");
Review comment:
not clear whether we need it or not, but there is a `Regex` class in `RegexUtils.h`, aiming to address GCC<4.9 compilers, should we use that here as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r644668234
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.cpp
##########
@@ -0,0 +1,386 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "MockConnectors.h"
+
+#include <fstream>
+#include <algorithm>
+#include <utility>
+#include <string>
+#include <memory>
+
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+std::size_t MockRow::size() const {
+ return column_names_.size();
+}
+
+std::string MockRow::getColumnName(std::size_t index) const {
+ return column_names_.at(index);
+}
+
+bool MockRow::isNull(std::size_t index) const {
+ return column_values_.at(index) == "NULL";
+}
+
+DataType MockRow::getDataType(std::size_t index) const {
+ return column_types_.at(index);
+}
+
+std::string MockRow::getString(std::size_t index) const {
+ return column_values_.at(index);
+}
+
+double MockRow::getDouble(std::size_t index) const {
+ return std::atof(column_values_.at(index).c_str());
+}
+
+int MockRow::getInteger(std::size_t index) const {
+ return std::atoi(column_values_.at(index).c_str());
+}
+
+long long MockRow::getLongLong(std::size_t index) const { // NOLINT
+ return std::atoll(column_values_.at(index).c_str());
+}
+
+unsigned long long MockRow::getUnsignedLongLong(std::size_t index) const { // NOLINT
+ return static_cast<unsigned long long>(std::atoll(column_values_.at(index).c_str())); // NOLINT
+}
+
+std::tm MockRow::getDate(std::size_t /*index*/) const {
+ throw std::runtime_error("date not implemented");
+}
+
+std::vector<std::string> MockRow::getValues() const {
+ return column_values_;
+}
+
+std::string MockRow::getValue(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return column_values_.at(it-column_names_.begin());
+ }
+ throw std::runtime_error("Unknown column name for getting value");
+}
+
+void MockRowset::addRow(const std::vector<std::string>& column_values) {
+ rows_.emplace_back(column_names_, column_types_, column_values);
+}
+
+void MockRowset::reset() {
+ current_row_ = rows_.begin();
+}
+
+bool MockRowset::is_done() {
+ return current_row_ == rows_.end();
+}
+
+Row& MockRowset::getCurrent() {
+ return *current_row_;
+}
+
+void MockRowset::next() {
+ ++current_row_;
+}
+
+std::vector<std::string> MockRowset::getColumnNames() const {
+ return column_names_;
+}
+
+std::vector<DataType> MockRowset::getColumnTypes() const {
+ return column_types_;
+}
+
+std::vector<MockRow> MockRowset::getRows() const {
+ return rows_;
+}
+
+std::size_t MockRowset::getColumnIndex(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return it-column_names_.begin();
+ }
+ throw std::runtime_error("Unknown column name for getting index");
+}
+
+void MockRowset::sort(const std::string& order_by_col, bool order_ascending) {
+ std::sort(rows_.begin(), rows_.end(), [&](const MockRow& first, const MockRow& second) {
+ if (order_ascending) {
+ return first.getValue(order_by_col) < second.getValue(order_by_col);
+ } else {
+ return first.getValue(order_by_col) > second.getValue(order_by_col);
+ }
+ });
+}
+
+std::unique_ptr<MockRowset> MockRowset::select(const std::vector<std::string>& cols, const std::function<bool(const MockRow&)>& condition, const std::string& order_by_col, bool order_ascending) {
+ if (!order_by_col.empty()) {
+ sort(order_by_col, order_ascending);
+ }
+
+ std::unique_ptr<MockRowset> rowset;
+ if (cols.empty()) {
+ rowset = utils::make_unique<MockRowset>(column_names_, column_types_);
+ } else {
+ std::vector<DataType> col_types;
+ for (const auto& col : cols) {
+ col_types.push_back(column_types_.at(getColumnIndex(col)));
+ }
+ rowset = utils::make_unique<MockRowset>(cols, col_types);
+ }
+
+ std::vector<std::string> used_cols = cols.empty() ? column_names_ : cols;
+ for (const auto& row : rows_) {
+ if (condition(row)) {
+ std::vector<std::string> values;
+ for (const auto& col : used_cols) {
+ values.push_back(row.getValue(col));
+ }
+ rowset->addRow(values);
+ }
+ }
+
+ return rowset;
+}
+
+std::unique_ptr<Rowset> MockDB::execute(const std::string& query, const std::vector<std::string>& args) {
+ if (minifi::utils::StringUtils::startsWith(query, "create table")) {
+ createTable(query);
+ } else if (minifi::utils::StringUtils::startsWith(query, "insert into")) {
+ insertInto(query, args);
+ } else if (minifi::utils::StringUtils::startsWith(query, "select")) {
+ return select(query, args);
+ } else {
+ throw std::runtime_error("Unknown query type");
+ }
+
+ return nullptr;
+}
+
+void MockDB::createTable(const std::string& query) {
+ std::smatch match;
+ std::regex expr("create table (\\w+)\\s*\\((.*)\\);");
+ std::regex_search(query, match, expr);
+ std::string table_name = match[1];
+ auto columns_with_type = minifi::utils::StringUtils::splitAndTrim(match[2], ",");
+ std::vector<std::string> col_names;
+ std::vector<DataType> col_types;
+ for (const auto& col_with_type : columns_with_type) {
+ auto splitted = minifi::utils::StringUtils::splitAndTrim(col_with_type, " ");
+ col_names.push_back(splitted[0]);
+ col_types.push_back(stringToDataType(splitted[1]));
+ }
+ tables_.emplace(table_name, MockRowset{col_names, col_types});
+ storeDb();
+}
+
+void MockDB::insertInto(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("insert into (\\w+)\\s*(\\((.*)\\))*\\s*values\\s*\\((.+)\\)");
+ std::regex_search(replaced_query, match, expr);
+ std::string table_name = match[1];
+ std::vector<std::string> values = minifi::utils::StringUtils::splitAndTrim(match[4], ",");
+ for (auto& value : values) {
+ value = minifi::utils::StringUtils::removeFramingCharacters(value, '\'');
+ }
+ auto insert_col_names = minifi::utils::StringUtils::splitAndTrim(match[3], ",");
+ if (!insert_col_names.empty()) {
+ auto col_names = tables_.at(table_name).getColumnNames();
+ std::vector<std::string> row;
+ for (const auto& col_name : col_names) {
+ auto it = std::find(insert_col_names.begin(), insert_col_names.end(), col_name);
+ if (it != insert_col_names.end()) {
+ row.push_back(values.at(it-insert_col_names.begin()));
+ } else {
+ row.push_back("NULL");
+ }
+ }
+ tables_.at(table_name).addRow(row);
+ } else {
+ tables_.at(table_name).addRow(values);
+ }
+
+ storeDb();
+}
+
+std::unique_ptr<Rowset> MockDB::select(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("select\\s+(.+)\\s+from\\s+(\\w+)\\s*(where ((.+(?= order by))|.+$))*\\s*(order by (.+))*");
+ std::regex_search(replaced_query, match, expr);
+ auto cols = minifi::utils::StringUtils::splitAndTrim(match[1], ",");
+ if (cols[0] == "*") {
+ cols = {};
+ }
+ std::string table_name = match[2];
+ std::string condition_str = match[4];
+ std::function<bool(const MockRow&)> condition;
+ if (!condition_str.empty()) {
+ if (condition_str == "int_col > 103") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) > 103; };
+ } else if (condition_str == "int_col > 102") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) > 102; };
+ } else if (condition_str == "int_col = 11") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) == 11; };
+ } else if (condition_str == "int_col = 11 and text_col = banana") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) == 11 && row.getValue("text_col") == "banana"; };
+ } else {
+ throw std::runtime_error("Unimplemented WHERE condition");
+ }
+ } else {
+ condition = [](const MockRow&){ return true; };
+ }
+ std::string order = match[7];
+ std::string order_col;
+ bool descending = false;
+ if (!order.empty()) {
+ auto order_col_and_sort = minifi::utils::StringUtils::splitAndTrim(order, " ");
+ order_col = order_col_and_sort[0];
+ descending = order_col_and_sort[1] == "desc";
+ }
+ return tables_.at(table_name).select(cols, condition, order_col, !descending);
+}
+
+void MockDB::readDb() {
+ tables_.clear();
+ std::ifstream file(file_path_);
+ std::string line;
+ ParsePhase phase = ParsePhase::NEW_TABLE;
+ std::string table_name;
+ std::vector<std::string> column_names;
+ std::vector<DataType> column_types;
+ while (std::getline(file, line)) {
+ switch (phase) {
+ case ParsePhase::NEW_TABLE: {
+ table_name = minifi::utils::StringUtils::trim(line);
+ phase = ParsePhase::COLUMN_NAMES;
+ break;
+ }
+ case ParsePhase::COLUMN_NAMES: {
+ column_names = minifi::utils::StringUtils::splitAndTrim(line, "|");
+ phase = ParsePhase::COLUMN_TYPES;
+ break;
+ }
+ case ParsePhase::COLUMN_TYPES: {
+ column_types.clear();
+ auto type_strs = minifi::utils::StringUtils::splitAndTrim(line, "|");
+ for (const auto& type : type_strs) {
+ column_types.push_back(stringToDataType(type));
+ }
+ tables_.emplace(table_name, MockRowset(column_names, column_types));
+ phase = ParsePhase::ROW_VALUES;
+ break;
+ }
+ case ParsePhase::ROW_VALUES: {
+ line = minifi::utils::StringUtils::trim(line);
+ if (line.empty()) {
+ phase = ParsePhase::NEW_TABLE;
+ break;
+ }
+ auto cells = minifi::utils::StringUtils::splitAndTrim(line, "|");
+ tables_.at(table_name).addRow(cells);
+ break;
+ }
+ }
+ }
+}
+
+void MockDB::storeDb() {
+ std::ofstream file(file_path_);
+ for (const auto& table : tables_) {
+ file << table.first << std::endl;
+ auto& rowset = table.second;
+ for (const auto& col_name : rowset.getColumnNames()) {
+ file << col_name << "|";
+ }
+ file << std::endl;
+ for (const auto& col_type : rowset.getColumnTypes()) {
+ file << dataTypeToString(col_type) << "|";
+ }
+ file << std::endl;
+ for (const auto& row : rowset.getRows()) {
+ for (const auto& cell : row.getValues()) {
+ file << cell << "|";
+ }
+ file << std::endl;
+ }
+ file << std::endl;
+ }
+}
+
+DataType MockDB::stringToDataType(const std::string& type_str) {
+ if (type_str == "integer") return DataType::INTEGER;
+ if (type_str == "text") return DataType::STRING;
+ if (type_str == "real") return DataType::DOUBLE;
+ throw std::runtime_error("Unimplemented data type");
+}
+
+std::string MockDB::dataTypeToString(DataType data_type) {
+ switch (data_type) {
+ case DataType::INTEGER: return "integer";
+ case DataType::STRING: return "text";
+ case DataType::DOUBLE: return "real";
+ default: throw std::runtime_error("Unimplemented data type");
+ }
+}
+
+std::unique_ptr<Rowset> MockStatement::execute(const std::vector<std::string>& args) {
+ MockDB db(file_path_);
+ return db.execute(query_, args);
+}
+
+MockODBCConnection::MockODBCConnection(std::string connectionString)
+ : connection_string_(std::move(connectionString)) {
+ std::smatch match;
+ std::regex expr("Database=(.*)(;|$)");
Review comment:
My thinking was that as we are already on the way of dropping support for GCC 4.8 and we already have a ticket [MINIFICPP-1508](https://issues.apache.org/jira/browse/MINIFICPP-1508) for removing the `Regex` utility class it wouldn't be ideal to add more functionality to it.
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.cpp
##########
@@ -0,0 +1,386 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "MockConnectors.h"
+
+#include <fstream>
+#include <algorithm>
+#include <utility>
+#include <string>
+#include <memory>
+
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+std::size_t MockRow::size() const {
+ return column_names_.size();
+}
+
+std::string MockRow::getColumnName(std::size_t index) const {
+ return column_names_.at(index);
+}
+
+bool MockRow::isNull(std::size_t index) const {
+ return column_values_.at(index) == "NULL";
+}
+
+DataType MockRow::getDataType(std::size_t index) const {
+ return column_types_.at(index);
+}
+
+std::string MockRow::getString(std::size_t index) const {
+ return column_values_.at(index);
+}
+
+double MockRow::getDouble(std::size_t index) const {
+ return std::atof(column_values_.at(index).c_str());
+}
+
+int MockRow::getInteger(std::size_t index) const {
+ return std::atoi(column_values_.at(index).c_str());
+}
+
+long long MockRow::getLongLong(std::size_t index) const { // NOLINT
+ return std::atoll(column_values_.at(index).c_str());
+}
+
+unsigned long long MockRow::getUnsignedLongLong(std::size_t index) const { // NOLINT
+ return static_cast<unsigned long long>(std::atoll(column_values_.at(index).c_str())); // NOLINT
+}
+
+std::tm MockRow::getDate(std::size_t /*index*/) const {
+ throw std::runtime_error("date not implemented");
+}
+
+std::vector<std::string> MockRow::getValues() const {
+ return column_values_;
+}
+
+std::string MockRow::getValue(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return column_values_.at(it-column_names_.begin());
+ }
+ throw std::runtime_error("Unknown column name for getting value");
+}
+
+void MockRowset::addRow(const std::vector<std::string>& column_values) {
+ rows_.emplace_back(column_names_, column_types_, column_values);
+}
+
+void MockRowset::reset() {
+ current_row_ = rows_.begin();
+}
+
+bool MockRowset::is_done() {
+ return current_row_ == rows_.end();
+}
+
+Row& MockRowset::getCurrent() {
+ return *current_row_;
+}
+
+void MockRowset::next() {
+ ++current_row_;
+}
+
+std::vector<std::string> MockRowset::getColumnNames() const {
+ return column_names_;
+}
+
+std::vector<DataType> MockRowset::getColumnTypes() const {
+ return column_types_;
+}
+
+std::vector<MockRow> MockRowset::getRows() const {
+ return rows_;
+}
+
+std::size_t MockRowset::getColumnIndex(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return it-column_names_.begin();
+ }
+ throw std::runtime_error("Unknown column name for getting index");
+}
+
+void MockRowset::sort(const std::string& order_by_col, bool order_ascending) {
+ std::sort(rows_.begin(), rows_.end(), [&](const MockRow& first, const MockRow& second) {
+ if (order_ascending) {
+ return first.getValue(order_by_col) < second.getValue(order_by_col);
+ } else {
+ return first.getValue(order_by_col) > second.getValue(order_by_col);
+ }
+ });
+}
+
+std::unique_ptr<MockRowset> MockRowset::select(const std::vector<std::string>& cols, const std::function<bool(const MockRow&)>& condition, const std::string& order_by_col, bool order_ascending) {
+ if (!order_by_col.empty()) {
+ sort(order_by_col, order_ascending);
+ }
+
+ std::unique_ptr<MockRowset> rowset;
+ if (cols.empty()) {
+ rowset = utils::make_unique<MockRowset>(column_names_, column_types_);
+ } else {
+ std::vector<DataType> col_types;
+ for (const auto& col : cols) {
+ col_types.push_back(column_types_.at(getColumnIndex(col)));
+ }
+ rowset = utils::make_unique<MockRowset>(cols, col_types);
+ }
+
+ std::vector<std::string> used_cols = cols.empty() ? column_names_ : cols;
+ for (const auto& row : rows_) {
+ if (condition(row)) {
+ std::vector<std::string> values;
+ for (const auto& col : used_cols) {
+ values.push_back(row.getValue(col));
+ }
+ rowset->addRow(values);
+ }
+ }
+
+ return rowset;
+}
+
+std::unique_ptr<Rowset> MockDB::execute(const std::string& query, const std::vector<std::string>& args) {
+ if (minifi::utils::StringUtils::startsWith(query, "create table")) {
+ createTable(query);
+ } else if (minifi::utils::StringUtils::startsWith(query, "insert into")) {
+ insertInto(query, args);
+ } else if (minifi::utils::StringUtils::startsWith(query, "select")) {
+ return select(query, args);
+ } else {
+ throw std::runtime_error("Unknown query type");
+ }
+
+ return nullptr;
+}
+
+void MockDB::createTable(const std::string& query) {
+ std::smatch match;
+ std::regex expr("create table (\\w+)\\s*\\((.*)\\);");
+ std::regex_search(query, match, expr);
+ std::string table_name = match[1];
+ auto columns_with_type = minifi::utils::StringUtils::splitAndTrim(match[2], ",");
+ std::vector<std::string> col_names;
+ std::vector<DataType> col_types;
+ for (const auto& col_with_type : columns_with_type) {
+ auto splitted = minifi::utils::StringUtils::splitAndTrim(col_with_type, " ");
+ col_names.push_back(splitted[0]);
+ col_types.push_back(stringToDataType(splitted[1]));
+ }
+ tables_.emplace(table_name, MockRowset{col_names, col_types});
+ storeDb();
+}
+
+void MockDB::insertInto(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("insert into (\\w+)\\s*(\\((.*)\\))*\\s*values\\s*\\((.+)\\)");
+ std::regex_search(replaced_query, match, expr);
+ std::string table_name = match[1];
+ std::vector<std::string> values = minifi::utils::StringUtils::splitAndTrim(match[4], ",");
+ for (auto& value : values) {
+ value = minifi::utils::StringUtils::removeFramingCharacters(value, '\'');
+ }
+ auto insert_col_names = minifi::utils::StringUtils::splitAndTrim(match[3], ",");
+ if (!insert_col_names.empty()) {
+ auto col_names = tables_.at(table_name).getColumnNames();
+ std::vector<std::string> row;
+ for (const auto& col_name : col_names) {
+ auto it = std::find(insert_col_names.begin(), insert_col_names.end(), col_name);
+ if (it != insert_col_names.end()) {
+ row.push_back(values.at(it-insert_col_names.begin()));
+ } else {
+ row.push_back("NULL");
+ }
+ }
+ tables_.at(table_name).addRow(row);
+ } else {
+ tables_.at(table_name).addRow(values);
+ }
+
+ storeDb();
+}
+
+std::unique_ptr<Rowset> MockDB::select(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("select\\s+(.+)\\s+from\\s+(\\w+)\\s*(where ((.+(?= order by))|.+$))*\\s*(order by (.+))*");
+ std::regex_search(replaced_query, match, expr);
+ auto cols = minifi::utils::StringUtils::splitAndTrim(match[1], ",");
+ if (cols[0] == "*") {
+ cols = {};
+ }
+ std::string table_name = match[2];
+ std::string condition_str = match[4];
+ std::function<bool(const MockRow&)> condition;
+ if (!condition_str.empty()) {
+ if (condition_str == "int_col > 103") {
Review comment:
I'll investigate some simple parsing solutions for handling our current cases. I think we need to find a good balance between maintainability and expandability. I used a hard coded condition as a start because we currently do not have too many conditions and it can be expanded easily, but if we add way more then it will become bloated. On the other side if we write a complex parser the maintainability would be way too much effort. I'll try to write a simple parser that would be sufficient for our cases without too much maintenance costs.
##########
File path: win_build_vs.bat
##########
@@ -65,7 +65,7 @@ for %%x in (%*) do (
mkdir %builddir%
pushd %builddir%\
-cmake -G %generator% -A %build_platform% -DINSTALLER_MERGE_MODULES=%installer_merge_modules% -DTEST_CUSTOM_WEL_PROVIDER=%test_custom_wel_provider% -DENABLE_SQL=%build_SQL% -DCMAKE_BUILD_TYPE_INIT=%cmake_build_type% -DCMAKE_BUILD_TYPE=%cmake_build_type% -DWIN32=WIN32 -DENABLE_LIBRDKAFKA=%build_kafka% -DENABLE_JNI=%build_jni% -DOPENSSL_OFF=OFF -DENABLE_COAP=%build_coap% -DENABLE_AWS=%build_AWS% -DENABLE_PDH=%build_PDH% -DENABLE_AZURE=%build_azure% -DENABLE_SFTP=%build_SFTP% -DUSE_SHARED_LIBS=OFF -DDISABLE_CONTROLLER=ON -DBUILD_ROCKSDB=ON -DFORCE_WINDOWS=ON -DUSE_SYSTEM_UUID=OFF -DDISABLE_LIBARCHIVE=OFF -DDISABLE_SCRIPTING=ON -DEXCLUDE_BOOST=ON -DENABLE_WEL=TRUE -DFAIL_ON_WARNINGS=OFF -DSKIP_TESTS=%skiptests% %strict_gsl_checks% %redist% -DENABLE_LINTER=%build_linter% .. && msbuild /m nifi-minifi-cpp.sln /property:Configuration=%cmake_build_type% /property:Platform=%build_platform% && copy main\%cmake_build_type%\minifi.exe main\
+cmake -G %generator% -A %build_platform% -DINSTALLER_MERGE_MODULES=%installer_merge_modules% -DTEST_CUSTOM_WEL_PROVIDER=%test_custom_wel_provider% -DENABLE_SQL=%build_SQL% -DUSE_REAL_ODBC_TEST_DRIVER=ON -DCMAKE_BUILD_TYPE_INIT=%cmake_build_type% -DCMAKE_BUILD_TYPE=%cmake_build_type% -DWIN32=WIN32 -DENABLE_LIBRDKAFKA=%build_kafka% -DENABLE_JNI=%build_jni% -DOPENSSL_OFF=OFF -DENABLE_COAP=%build_coap% -DENABLE_AWS=%build_AWS% -DENABLE_PDH=%build_PDH% -DENABLE_AZURE=%build_azure% -DENABLE_SFTP=%build_SFTP% -DUSE_SHARED_LIBS=OFF -DDISABLE_CONTROLLER=ON -DBUILD_ROCKSDB=ON -DFORCE_WINDOWS=ON -DUSE_SYSTEM_UUID=OFF -DDISABLE_LIBARCHIVE=OFF -DDISABLE_SCRIPTING=ON -DEXCLUDE_BOOST=ON -DENABLE_WEL=TRUE -DFAIL_ON_WARNINGS=OFF -DSKIP_TESTS=%skiptests% %strict_gsl_checks% %redist% -DENABLE_LINTER=%build_linter% .. && msbuild /m nifi-minifi-cpp.sln /property:Configuration=%cmake_build_type% /property:Platform=%build_platform% && copy main\%cmake_build_type%\minifi.exe main\
Review comment:
You are right, that's a better solution, fixed in 856a4349dda249088cc9fa104646cd4843f28780
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.cpp
##########
@@ -0,0 +1,386 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "MockConnectors.h"
+
+#include <fstream>
+#include <algorithm>
+#include <utility>
+#include <string>
+#include <memory>
+
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+std::size_t MockRow::size() const {
+ return column_names_.size();
+}
+
+std::string MockRow::getColumnName(std::size_t index) const {
+ return column_names_.at(index);
+}
+
+bool MockRow::isNull(std::size_t index) const {
+ return column_values_.at(index) == "NULL";
+}
+
+DataType MockRow::getDataType(std::size_t index) const {
+ return column_types_.at(index);
+}
+
+std::string MockRow::getString(std::size_t index) const {
+ return column_values_.at(index);
+}
+
+double MockRow::getDouble(std::size_t index) const {
+ return std::atof(column_values_.at(index).c_str());
+}
+
+int MockRow::getInteger(std::size_t index) const {
+ return std::atoi(column_values_.at(index).c_str());
Review comment:
Updated in e905b1377f3689f285913e3886deab32a06cea4d
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.h
##########
@@ -0,0 +1,175 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <regex>
+#include <map>
+#include <vector>
+#include <algorithm>
+#include <memory>
+#include <string>
+
+#include "data/DatabaseConnectors.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+class MockRow : public Row {
+ public:
+ MockRow(std::vector<std::string>& column_names, std::vector<DataType>& column_types, const std::vector<std::string>& column_values)
+ : column_names_(column_names), column_types_(column_types), column_values_(column_values) {
+ }
+
+ MockRow(MockRow&& other) = default;
+ MockRow(const MockRow& other) = default;
+ MockRow& operator=(MockRow&& other) {
+ column_names_ = other.column_names_;
+ column_types_ = other.column_types_;
Review comment:
You right this was wrong. The original goal was just to make the class copyable, and `std::reference_wrapper` could have been a good solution here. Finally I decided it's simpler just to use a raw pointer in this case. Updated in ee9452ebd591ff22139606ba2cd682df912d14f4
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.cpp
##########
@@ -0,0 +1,386 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "MockConnectors.h"
+
+#include <fstream>
+#include <algorithm>
+#include <utility>
+#include <string>
+#include <memory>
+
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+std::size_t MockRow::size() const {
+ return column_names_.size();
+}
+
+std::string MockRow::getColumnName(std::size_t index) const {
+ return column_names_.at(index);
+}
+
+bool MockRow::isNull(std::size_t index) const {
+ return column_values_.at(index) == "NULL";
+}
+
+DataType MockRow::getDataType(std::size_t index) const {
+ return column_types_.at(index);
+}
+
+std::string MockRow::getString(std::size_t index) const {
+ return column_values_.at(index);
+}
+
+double MockRow::getDouble(std::size_t index) const {
+ return std::atof(column_values_.at(index).c_str());
+}
+
+int MockRow::getInteger(std::size_t index) const {
+ return std::atoi(column_values_.at(index).c_str());
+}
+
+long long MockRow::getLongLong(std::size_t index) const { // NOLINT
+ return std::atoll(column_values_.at(index).c_str());
+}
+
+unsigned long long MockRow::getUnsignedLongLong(std::size_t index) const { // NOLINT
+ return static_cast<unsigned long long>(std::atoll(column_values_.at(index).c_str())); // NOLINT
+}
+
+std::tm MockRow::getDate(std::size_t /*index*/) const {
+ throw std::runtime_error("date not implemented");
+}
+
+std::vector<std::string> MockRow::getValues() const {
+ return column_values_;
+}
+
+std::string MockRow::getValue(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return column_values_.at(it-column_names_.begin());
+ }
+ throw std::runtime_error("Unknown column name for getting value");
+}
+
+void MockRowset::addRow(const std::vector<std::string>& column_values) {
+ rows_.emplace_back(column_names_, column_types_, column_values);
+}
+
+void MockRowset::reset() {
+ current_row_ = rows_.begin();
+}
+
+bool MockRowset::is_done() {
+ return current_row_ == rows_.end();
+}
+
+Row& MockRowset::getCurrent() {
+ return *current_row_;
+}
+
+void MockRowset::next() {
+ ++current_row_;
+}
+
+std::vector<std::string> MockRowset::getColumnNames() const {
+ return column_names_;
+}
+
+std::vector<DataType> MockRowset::getColumnTypes() const {
+ return column_types_;
+}
+
+std::vector<MockRow> MockRowset::getRows() const {
+ return rows_;
+}
+
+std::size_t MockRowset::getColumnIndex(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return it-column_names_.begin();
+ }
+ throw std::runtime_error("Unknown column name for getting index");
+}
+
+void MockRowset::sort(const std::string& order_by_col, bool order_ascending) {
+ std::sort(rows_.begin(), rows_.end(), [&](const MockRow& first, const MockRow& second) {
+ if (order_ascending) {
+ return first.getValue(order_by_col) < second.getValue(order_by_col);
+ } else {
+ return first.getValue(order_by_col) > second.getValue(order_by_col);
+ }
+ });
+}
+
+std::unique_ptr<MockRowset> MockRowset::select(const std::vector<std::string>& cols, const std::function<bool(const MockRow&)>& condition, const std::string& order_by_col, bool order_ascending) {
+ if (!order_by_col.empty()) {
+ sort(order_by_col, order_ascending);
+ }
+
+ std::unique_ptr<MockRowset> rowset;
+ if (cols.empty()) {
+ rowset = utils::make_unique<MockRowset>(column_names_, column_types_);
+ } else {
+ std::vector<DataType> col_types;
+ for (const auto& col : cols) {
+ col_types.push_back(column_types_.at(getColumnIndex(col)));
+ }
+ rowset = utils::make_unique<MockRowset>(cols, col_types);
+ }
+
+ std::vector<std::string> used_cols = cols.empty() ? column_names_ : cols;
+ for (const auto& row : rows_) {
+ if (condition(row)) {
+ std::vector<std::string> values;
+ for (const auto& col : used_cols) {
+ values.push_back(row.getValue(col));
+ }
+ rowset->addRow(values);
+ }
+ }
+
+ return rowset;
+}
+
+std::unique_ptr<Rowset> MockDB::execute(const std::string& query, const std::vector<std::string>& args) {
+ if (minifi::utils::StringUtils::startsWith(query, "create table")) {
+ createTable(query);
+ } else if (minifi::utils::StringUtils::startsWith(query, "insert into")) {
+ insertInto(query, args);
+ } else if (minifi::utils::StringUtils::startsWith(query, "select")) {
+ return select(query, args);
+ } else {
+ throw std::runtime_error("Unknown query type");
+ }
+
+ return nullptr;
+}
+
+void MockDB::createTable(const std::string& query) {
+ std::smatch match;
+ std::regex expr("create table (\\w+)\\s*\\((.*)\\);");
+ std::regex_search(query, match, expr);
+ std::string table_name = match[1];
+ auto columns_with_type = minifi::utils::StringUtils::splitAndTrim(match[2], ",");
+ std::vector<std::string> col_names;
+ std::vector<DataType> col_types;
+ for (const auto& col_with_type : columns_with_type) {
+ auto splitted = minifi::utils::StringUtils::splitAndTrim(col_with_type, " ");
+ col_names.push_back(splitted[0]);
+ col_types.push_back(stringToDataType(splitted[1]));
+ }
+ tables_.emplace(table_name, MockRowset{col_names, col_types});
+ storeDb();
+}
+
+void MockDB::insertInto(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("insert into (\\w+)\\s*(\\((.*)\\))*\\s*values\\s*\\((.+)\\)");
+ std::regex_search(replaced_query, match, expr);
+ std::string table_name = match[1];
+ std::vector<std::string> values = minifi::utils::StringUtils::splitAndTrim(match[4], ",");
+ for (auto& value : values) {
+ value = minifi::utils::StringUtils::removeFramingCharacters(value, '\'');
+ }
+ auto insert_col_names = minifi::utils::StringUtils::splitAndTrim(match[3], ",");
+ if (!insert_col_names.empty()) {
+ auto col_names = tables_.at(table_name).getColumnNames();
+ std::vector<std::string> row;
+ for (const auto& col_name : col_names) {
+ auto it = std::find(insert_col_names.begin(), insert_col_names.end(), col_name);
+ if (it != insert_col_names.end()) {
+ row.push_back(values.at(it-insert_col_names.begin()));
+ } else {
+ row.push_back("NULL");
+ }
+ }
+ tables_.at(table_name).addRow(row);
+ } else {
+ tables_.at(table_name).addRow(values);
+ }
+
+ storeDb();
+}
+
+std::unique_ptr<Rowset> MockDB::select(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("select\\s+(.+)\\s+from\\s+(\\w+)\\s*(where ((.+(?= order by))|.+$))*\\s*(order by (.+))*");
+ std::regex_search(replaced_query, match, expr);
+ auto cols = minifi::utils::StringUtils::splitAndTrim(match[1], ",");
+ if (cols[0] == "*") {
+ cols = {};
+ }
+ std::string table_name = match[2];
+ std::string condition_str = match[4];
+ std::function<bool(const MockRow&)> condition;
+ if (!condition_str.empty()) {
+ if (condition_str == "int_col > 103") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) > 103; };
+ } else if (condition_str == "int_col > 102") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) > 102; };
+ } else if (condition_str == "int_col = 11") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) == 11; };
+ } else if (condition_str == "int_col = 11 and text_col = banana") {
Review comment:
When replacing arguments I did not add apostrophes so it worked like this. The code was removed in favor of 96c3a9dea8e0d9c0492ccf5850184755651418e9
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.cpp
##########
@@ -0,0 +1,386 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "MockConnectors.h"
+
+#include <fstream>
+#include <algorithm>
+#include <utility>
+#include <string>
+#include <memory>
+
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+std::size_t MockRow::size() const {
+ return column_names_.size();
+}
+
+std::string MockRow::getColumnName(std::size_t index) const {
+ return column_names_.at(index);
+}
+
+bool MockRow::isNull(std::size_t index) const {
+ return column_values_.at(index) == "NULL";
+}
+
+DataType MockRow::getDataType(std::size_t index) const {
+ return column_types_.at(index);
+}
+
+std::string MockRow::getString(std::size_t index) const {
+ return column_values_.at(index);
+}
+
+double MockRow::getDouble(std::size_t index) const {
+ return std::atof(column_values_.at(index).c_str());
+}
+
+int MockRow::getInteger(std::size_t index) const {
+ return std::atoi(column_values_.at(index).c_str());
+}
+
+long long MockRow::getLongLong(std::size_t index) const { // NOLINT
+ return std::atoll(column_values_.at(index).c_str());
+}
+
+unsigned long long MockRow::getUnsignedLongLong(std::size_t index) const { // NOLINT
+ return static_cast<unsigned long long>(std::atoll(column_values_.at(index).c_str())); // NOLINT
+}
+
+std::tm MockRow::getDate(std::size_t /*index*/) const {
+ throw std::runtime_error("date not implemented");
+}
+
+std::vector<std::string> MockRow::getValues() const {
+ return column_values_;
+}
+
+std::string MockRow::getValue(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return column_values_.at(it-column_names_.begin());
+ }
+ throw std::runtime_error("Unknown column name for getting value");
+}
+
+void MockRowset::addRow(const std::vector<std::string>& column_values) {
+ rows_.emplace_back(column_names_, column_types_, column_values);
+}
+
+void MockRowset::reset() {
+ current_row_ = rows_.begin();
+}
+
+bool MockRowset::is_done() {
+ return current_row_ == rows_.end();
+}
+
+Row& MockRowset::getCurrent() {
+ return *current_row_;
+}
+
+void MockRowset::next() {
+ ++current_row_;
+}
+
+std::vector<std::string> MockRowset::getColumnNames() const {
+ return column_names_;
+}
+
+std::vector<DataType> MockRowset::getColumnTypes() const {
+ return column_types_;
+}
+
+std::vector<MockRow> MockRowset::getRows() const {
+ return rows_;
+}
+
+std::size_t MockRowset::getColumnIndex(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return it-column_names_.begin();
+ }
+ throw std::runtime_error("Unknown column name for getting index");
+}
+
+void MockRowset::sort(const std::string& order_by_col, bool order_ascending) {
+ std::sort(rows_.begin(), rows_.end(), [&](const MockRow& first, const MockRow& second) {
+ if (order_ascending) {
+ return first.getValue(order_by_col) < second.getValue(order_by_col);
+ } else {
+ return first.getValue(order_by_col) > second.getValue(order_by_col);
+ }
+ });
+}
+
+std::unique_ptr<MockRowset> MockRowset::select(const std::vector<std::string>& cols, const std::function<bool(const MockRow&)>& condition, const std::string& order_by_col, bool order_ascending) {
+ if (!order_by_col.empty()) {
+ sort(order_by_col, order_ascending);
+ }
+
+ std::unique_ptr<MockRowset> rowset;
+ if (cols.empty()) {
+ rowset = utils::make_unique<MockRowset>(column_names_, column_types_);
+ } else {
+ std::vector<DataType> col_types;
+ for (const auto& col : cols) {
+ col_types.push_back(column_types_.at(getColumnIndex(col)));
+ }
+ rowset = utils::make_unique<MockRowset>(cols, col_types);
+ }
+
+ std::vector<std::string> used_cols = cols.empty() ? column_names_ : cols;
+ for (const auto& row : rows_) {
+ if (condition(row)) {
+ std::vector<std::string> values;
+ for (const auto& col : used_cols) {
+ values.push_back(row.getValue(col));
+ }
+ rowset->addRow(values);
+ }
+ }
+
+ return rowset;
+}
+
+std::unique_ptr<Rowset> MockDB::execute(const std::string& query, const std::vector<std::string>& args) {
+ if (minifi::utils::StringUtils::startsWith(query, "create table")) {
+ createTable(query);
+ } else if (minifi::utils::StringUtils::startsWith(query, "insert into")) {
+ insertInto(query, args);
+ } else if (minifi::utils::StringUtils::startsWith(query, "select")) {
+ return select(query, args);
+ } else {
+ throw std::runtime_error("Unknown query type");
+ }
+
+ return nullptr;
+}
+
+void MockDB::createTable(const std::string& query) {
+ std::smatch match;
+ std::regex expr("create table (\\w+)\\s*\\((.*)\\);");
+ std::regex_search(query, match, expr);
+ std::string table_name = match[1];
+ auto columns_with_type = minifi::utils::StringUtils::splitAndTrim(match[2], ",");
+ std::vector<std::string> col_names;
+ std::vector<DataType> col_types;
+ for (const auto& col_with_type : columns_with_type) {
+ auto splitted = minifi::utils::StringUtils::splitAndTrim(col_with_type, " ");
+ col_names.push_back(splitted[0]);
+ col_types.push_back(stringToDataType(splitted[1]));
+ }
+ tables_.emplace(table_name, MockRowset{col_names, col_types});
+ storeDb();
+}
+
+void MockDB::insertInto(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("insert into (\\w+)\\s*(\\((.*)\\))*\\s*values\\s*\\((.+)\\)");
+ std::regex_search(replaced_query, match, expr);
+ std::string table_name = match[1];
+ std::vector<std::string> values = minifi::utils::StringUtils::splitAndTrim(match[4], ",");
+ for (auto& value : values) {
+ value = minifi::utils::StringUtils::removeFramingCharacters(value, '\'');
+ }
+ auto insert_col_names = minifi::utils::StringUtils::splitAndTrim(match[3], ",");
+ if (!insert_col_names.empty()) {
+ auto col_names = tables_.at(table_name).getColumnNames();
+ std::vector<std::string> row;
+ for (const auto& col_name : col_names) {
+ auto it = std::find(insert_col_names.begin(), insert_col_names.end(), col_name);
+ if (it != insert_col_names.end()) {
+ row.push_back(values.at(it-insert_col_names.begin()));
+ } else {
+ row.push_back("NULL");
+ }
+ }
+ tables_.at(table_name).addRow(row);
+ } else {
+ tables_.at(table_name).addRow(values);
+ }
+
+ storeDb();
+}
+
+std::unique_ptr<Rowset> MockDB::select(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("select\\s+(.+)\\s+from\\s+(\\w+)\\s*(where ((.+(?= order by))|.+$))*\\s*(order by (.+))*");
+ std::regex_search(replaced_query, match, expr);
+ auto cols = minifi::utils::StringUtils::splitAndTrim(match[1], ",");
+ if (cols[0] == "*") {
+ cols = {};
+ }
+ std::string table_name = match[2];
+ std::string condition_str = match[4];
+ std::function<bool(const MockRow&)> condition;
+ if (!condition_str.empty()) {
+ if (condition_str == "int_col > 103") {
Review comment:
Added a parser for simple where conditions in 96c3a9dea8e0d9c0492ccf5850184755651418e9
##########
File path: extensions/sql/data/DatabaseConnectors.h
##########
@@ -20,72 +20,68 @@
#include <memory>
#include <string>
-
-#include <soci/soci.h>
-
-#include "Utils.h"
+#include <vector>
+#include <ctime>
namespace org {
namespace apache {
namespace nifi {
namespace minifi {
namespace sql {
-/**
- * We do not intend to create an abstract facade here. We know that SOCI is the underlying
- * SQL library. We only wish to abstract ODBC specific information
- */
+enum class DataType {
+ STRING,
+ DOUBLE,
+ INTEGER,
+ LONG_LONG,
+ UNSIGNED_LONG_LONG,
+ DATE
+};
-class Statement {
+class Row {
public:
+ virtual ~Row() = default;
+ virtual std::size_t size() const = 0;
+ virtual std::string getColumnName(std::size_t index) const = 0;
+ virtual bool isNull(std::size_t index) const = 0;
+ virtual DataType getDataType(std::size_t index) const = 0;
+ virtual std::string getString(std::size_t index) const = 0;
+ virtual double getDouble(std::size_t index) const = 0;
+ virtual int getInteger(std::size_t index) const = 0;
+ virtual long long getLongLong(std::size_t index) const = 0;
+ virtual unsigned long long getUnsignedLongLong(std::size_t index) const = 0;
+ virtual std::tm getDate(std::size_t index) const = 0;
Review comment:
The specialization for `row::get` for the date type is `std::tm` in the implementing [SOCI library](http://soci.sourceforge.net/doc/master/types/) that's why I sticked with that type.
##########
File path: extensions/sql/data/DatabaseConnectors.h
##########
@@ -20,72 +20,68 @@
#include <memory>
#include <string>
-
-#include <soci/soci.h>
-
-#include "Utils.h"
+#include <vector>
+#include <ctime>
namespace org {
namespace apache {
namespace nifi {
namespace minifi {
namespace sql {
-/**
- * We do not intend to create an abstract facade here. We know that SOCI is the underlying
- * SQL library. We only wish to abstract ODBC specific information
- */
+enum class DataType {
+ STRING,
+ DOUBLE,
+ INTEGER,
+ LONG_LONG,
+ UNSIGNED_LONG_LONG,
+ DATE
+};
-class Statement {
+class Row {
public:
+ virtual ~Row() = default;
+ virtual std::size_t size() const = 0;
+ virtual std::string getColumnName(std::size_t index) const = 0;
+ virtual bool isNull(std::size_t index) const = 0;
+ virtual DataType getDataType(std::size_t index) const = 0;
+ virtual std::string getString(std::size_t index) const = 0;
+ virtual double getDouble(std::size_t index) const = 0;
+ virtual int getInteger(std::size_t index) const = 0;
+ virtual long long getLongLong(std::size_t index) const = 0;
+ virtual unsigned long long getUnsignedLongLong(std::size_t index) const = 0;
+ virtual std::tm getDate(std::size_t index) const = 0;
Review comment:
The specialization for `row::get` for the date type is `std::tm` in the implementing [SOCI library](http://soci.sourceforge.net/doc/master/types/) that's why I stuck with that type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r644728831
##########
File path: win_build_vs.bat
##########
@@ -65,7 +65,7 @@ for %%x in (%*) do (
mkdir %builddir%
pushd %builddir%\
-cmake -G %generator% -A %build_platform% -DINSTALLER_MERGE_MODULES=%installer_merge_modules% -DTEST_CUSTOM_WEL_PROVIDER=%test_custom_wel_provider% -DENABLE_SQL=%build_SQL% -DCMAKE_BUILD_TYPE_INIT=%cmake_build_type% -DCMAKE_BUILD_TYPE=%cmake_build_type% -DWIN32=WIN32 -DENABLE_LIBRDKAFKA=%build_kafka% -DENABLE_JNI=%build_jni% -DOPENSSL_OFF=OFF -DENABLE_COAP=%build_coap% -DENABLE_AWS=%build_AWS% -DENABLE_PDH=%build_PDH% -DENABLE_AZURE=%build_azure% -DENABLE_SFTP=%build_SFTP% -DUSE_SHARED_LIBS=OFF -DDISABLE_CONTROLLER=ON -DBUILD_ROCKSDB=ON -DFORCE_WINDOWS=ON -DUSE_SYSTEM_UUID=OFF -DDISABLE_LIBARCHIVE=OFF -DDISABLE_SCRIPTING=ON -DEXCLUDE_BOOST=ON -DENABLE_WEL=TRUE -DFAIL_ON_WARNINGS=OFF -DSKIP_TESTS=%skiptests% %strict_gsl_checks% %redist% -DENABLE_LINTER=%build_linter% .. && msbuild /m nifi-minifi-cpp.sln /property:Configuration=%cmake_build_type% /property:Platform=%build_platform% && copy main\%cmake_build_type%\minifi.exe main\
+cmake -G %generator% -A %build_platform% -DINSTALLER_MERGE_MODULES=%installer_merge_modules% -DTEST_CUSTOM_WEL_PROVIDER=%test_custom_wel_provider% -DENABLE_SQL=%build_SQL% -DUSE_REAL_ODBC_TEST_DRIVER=ON -DCMAKE_BUILD_TYPE_INIT=%cmake_build_type% -DCMAKE_BUILD_TYPE=%cmake_build_type% -DWIN32=WIN32 -DENABLE_LIBRDKAFKA=%build_kafka% -DENABLE_JNI=%build_jni% -DOPENSSL_OFF=OFF -DENABLE_COAP=%build_coap% -DENABLE_AWS=%build_AWS% -DENABLE_PDH=%build_PDH% -DENABLE_AZURE=%build_azure% -DENABLE_SFTP=%build_SFTP% -DUSE_SHARED_LIBS=OFF -DDISABLE_CONTROLLER=ON -DBUILD_ROCKSDB=ON -DFORCE_WINDOWS=ON -DUSE_SYSTEM_UUID=OFF -DDISABLE_LIBARCHIVE=OFF -DDISABLE_SCRIPTING=ON -DEXCLUDE_BOOST=ON -DENABLE_WEL=TRUE -DFAIL_ON_WARNINGS=OFF -DSKIP_TESTS=%skiptests% %strict_gsl_checks% %redist% -DENABLE_LINTER=%build_linter% .. && msbuild /m nifi-minifi-cpp.sln /property:Configuration=%cmake_build_type% /property:Platform=%build_platform% && copy main\%cmake_build_type%\minifi.exe main\
Review comment:
You are right, that's a better solution, fixed in 856a4349dda249088cc9fa104646cd4843f28780
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni closed pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
adamdebreceni closed pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r678899606
##########
File path: win_build_vs.bat
##########
@@ -65,7 +65,7 @@ for %%x in (%*) do (
mkdir %builddir%
pushd %builddir%\
-cmake -G %generator% -A %build_platform% -DINSTALLER_MERGE_MODULES=%installer_merge_modules% -DTEST_CUSTOM_WEL_PROVIDER=%test_custom_wel_provider% -DENABLE_SQL=%build_SQL% -DCMAKE_BUILD_TYPE_INIT=%cmake_build_type% -DCMAKE_BUILD_TYPE=%cmake_build_type% -DWIN32=WIN32 -DENABLE_LIBRDKAFKA=%build_kafka% -DENABLE_JNI=%build_jni% -DOPENSSL_OFF=OFF -DENABLE_COAP=%build_coap% -DENABLE_AWS=%build_AWS% -DENABLE_PDH=%build_PDH% -DENABLE_AZURE=%build_azure% -DENABLE_SFTP=%build_SFTP% -DUSE_SHARED_LIBS=OFF -DDISABLE_CONTROLLER=ON -DBUILD_ROCKSDB=ON -DFORCE_WINDOWS=ON -DUSE_SYSTEM_UUID=OFF -DDISABLE_LIBARCHIVE=OFF -DDISABLE_SCRIPTING=ON -DEXCLUDE_BOOST=ON -DENABLE_WEL=TRUE -DFAIL_ON_WARNINGS=OFF -DSKIP_TESTS=%skiptests% %strict_gsl_checks% %redist% -DENABLE_LINTER=%build_linter% .. && msbuild /m nifi-minifi-cpp.sln /property:Configuration=%cmake_build_type% /property:Platform=%build_platform% && copy main\%cmake_build_type%\minifi.exe main\
+cmake -G %generator% -A %build_platform% -DINSTALLER_MERGE_MODULES=%installer_merge_modules% -DTEST_CUSTOM_WEL_PROVIDER=%test_custom_wel_provider% -DENABLE_SQL=%build_SQL% -DUSE_REAL_ODBC_TEST_DRIVER=ON -DCMAKE_BUILD_TYPE_INIT=%cmake_build_type% -DCMAKE_BUILD_TYPE=%cmake_build_type% -DWIN32=WIN32 -DENABLE_LIBRDKAFKA=%build_kafka% -DENABLE_JNI=%build_jni% -DOPENSSL_OFF=OFF -DENABLE_COAP=%build_coap% -DENABLE_AWS=%build_AWS% -DENABLE_PDH=%build_PDH% -DENABLE_AZURE=%build_azure% -DENABLE_SFTP=%build_SFTP% -DUSE_SHARED_LIBS=OFF -DDISABLE_CONTROLLER=ON -DBUILD_ROCKSDB=ON -DFORCE_WINDOWS=ON -DUSE_SYSTEM_UUID=OFF -DDISABLE_LIBARCHIVE=OFF -DDISABLE_SCRIPTING=ON -DEXCLUDE_BOOST=ON -DENABLE_WEL=TRUE -DFAIL_ON_WARNINGS=OFF -DSKIP_TESTS=%skiptests% %strict_gsl_checks% %redist% -DENABLE_LINTER=%build_linter% .. && msbuild /m nifi-minifi-cpp.sln /property:Configuration=%cmake_build_type% /property:Platform=%build_platform% && copy main\%cmake_build_type%\minifi.exe main\
Review comment:
The flag we use in the bat file is actually `/RO`
```
if [%%~x] EQU [/RO] set real_odbc=ON
```
```
| /RO | Use real ODBC driver in tests instead of mock SQL driver |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r644618079
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.h
##########
@@ -0,0 +1,175 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <regex>
+#include <map>
+#include <vector>
+#include <algorithm>
+#include <memory>
+#include <string>
+
+#include "data/DatabaseConnectors.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+class MockRow : public Row {
+ public:
+ MockRow(std::vector<std::string>& column_names, std::vector<DataType>& column_types, const std::vector<std::string>& column_values)
+ : column_names_(column_names), column_types_(column_types), column_values_(column_values) {
+ }
+
+ MockRow(MockRow&& other) = default;
+ MockRow(const MockRow& other) = default;
+ MockRow& operator=(MockRow&& other) {
+ column_names_ = other.column_names_;
+ column_types_ = other.column_types_;
Review comment:
also won't the lack of `noexcept` make it unavailable for `std::vector`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r644911926
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.cpp
##########
@@ -0,0 +1,386 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "MockConnectors.h"
+
+#include <fstream>
+#include <algorithm>
+#include <utility>
+#include <string>
+#include <memory>
+
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+std::size_t MockRow::size() const {
+ return column_names_.size();
+}
+
+std::string MockRow::getColumnName(std::size_t index) const {
+ return column_names_.at(index);
+}
+
+bool MockRow::isNull(std::size_t index) const {
+ return column_values_.at(index) == "NULL";
+}
+
+DataType MockRow::getDataType(std::size_t index) const {
+ return column_types_.at(index);
+}
+
+std::string MockRow::getString(std::size_t index) const {
+ return column_values_.at(index);
+}
+
+double MockRow::getDouble(std::size_t index) const {
+ return std::atof(column_values_.at(index).c_str());
+}
+
+int MockRow::getInteger(std::size_t index) const {
+ return std::atoi(column_values_.at(index).c_str());
+}
+
+long long MockRow::getLongLong(std::size_t index) const { // NOLINT
+ return std::atoll(column_values_.at(index).c_str());
+}
+
+unsigned long long MockRow::getUnsignedLongLong(std::size_t index) const { // NOLINT
+ return static_cast<unsigned long long>(std::atoll(column_values_.at(index).c_str())); // NOLINT
+}
+
+std::tm MockRow::getDate(std::size_t /*index*/) const {
+ throw std::runtime_error("date not implemented");
+}
+
+std::vector<std::string> MockRow::getValues() const {
+ return column_values_;
+}
+
+std::string MockRow::getValue(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return column_values_.at(it-column_names_.begin());
+ }
+ throw std::runtime_error("Unknown column name for getting value");
+}
+
+void MockRowset::addRow(const std::vector<std::string>& column_values) {
+ rows_.emplace_back(column_names_, column_types_, column_values);
+}
+
+void MockRowset::reset() {
+ current_row_ = rows_.begin();
+}
+
+bool MockRowset::is_done() {
+ return current_row_ == rows_.end();
+}
+
+Row& MockRowset::getCurrent() {
+ return *current_row_;
+}
+
+void MockRowset::next() {
+ ++current_row_;
+}
+
+std::vector<std::string> MockRowset::getColumnNames() const {
+ return column_names_;
+}
+
+std::vector<DataType> MockRowset::getColumnTypes() const {
+ return column_types_;
+}
+
+std::vector<MockRow> MockRowset::getRows() const {
+ return rows_;
+}
+
+std::size_t MockRowset::getColumnIndex(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return it-column_names_.begin();
+ }
+ throw std::runtime_error("Unknown column name for getting index");
+}
+
+void MockRowset::sort(const std::string& order_by_col, bool order_ascending) {
+ std::sort(rows_.begin(), rows_.end(), [&](const MockRow& first, const MockRow& second) {
+ if (order_ascending) {
+ return first.getValue(order_by_col) < second.getValue(order_by_col);
+ } else {
+ return first.getValue(order_by_col) > second.getValue(order_by_col);
+ }
+ });
+}
+
+std::unique_ptr<MockRowset> MockRowset::select(const std::vector<std::string>& cols, const std::function<bool(const MockRow&)>& condition, const std::string& order_by_col, bool order_ascending) {
+ if (!order_by_col.empty()) {
+ sort(order_by_col, order_ascending);
+ }
+
+ std::unique_ptr<MockRowset> rowset;
+ if (cols.empty()) {
+ rowset = utils::make_unique<MockRowset>(column_names_, column_types_);
+ } else {
+ std::vector<DataType> col_types;
+ for (const auto& col : cols) {
+ col_types.push_back(column_types_.at(getColumnIndex(col)));
+ }
+ rowset = utils::make_unique<MockRowset>(cols, col_types);
+ }
+
+ std::vector<std::string> used_cols = cols.empty() ? column_names_ : cols;
+ for (const auto& row : rows_) {
+ if (condition(row)) {
+ std::vector<std::string> values;
+ for (const auto& col : used_cols) {
+ values.push_back(row.getValue(col));
+ }
+ rowset->addRow(values);
+ }
+ }
+
+ return rowset;
+}
+
+std::unique_ptr<Rowset> MockDB::execute(const std::string& query, const std::vector<std::string>& args) {
+ if (minifi::utils::StringUtils::startsWith(query, "create table")) {
+ createTable(query);
+ } else if (minifi::utils::StringUtils::startsWith(query, "insert into")) {
+ insertInto(query, args);
+ } else if (minifi::utils::StringUtils::startsWith(query, "select")) {
+ return select(query, args);
+ } else {
+ throw std::runtime_error("Unknown query type");
+ }
+
+ return nullptr;
+}
+
+void MockDB::createTable(const std::string& query) {
+ std::smatch match;
+ std::regex expr("create table (\\w+)\\s*\\((.*)\\);");
+ std::regex_search(query, match, expr);
+ std::string table_name = match[1];
+ auto columns_with_type = minifi::utils::StringUtils::splitAndTrim(match[2], ",");
+ std::vector<std::string> col_names;
+ std::vector<DataType> col_types;
+ for (const auto& col_with_type : columns_with_type) {
+ auto splitted = minifi::utils::StringUtils::splitAndTrim(col_with_type, " ");
+ col_names.push_back(splitted[0]);
+ col_types.push_back(stringToDataType(splitted[1]));
+ }
+ tables_.emplace(table_name, MockRowset{col_names, col_types});
+ storeDb();
+}
+
+void MockDB::insertInto(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("insert into (\\w+)\\s*(\\((.*)\\))*\\s*values\\s*\\((.+)\\)");
+ std::regex_search(replaced_query, match, expr);
+ std::string table_name = match[1];
+ std::vector<std::string> values = minifi::utils::StringUtils::splitAndTrim(match[4], ",");
+ for (auto& value : values) {
+ value = minifi::utils::StringUtils::removeFramingCharacters(value, '\'');
+ }
+ auto insert_col_names = minifi::utils::StringUtils::splitAndTrim(match[3], ",");
+ if (!insert_col_names.empty()) {
+ auto col_names = tables_.at(table_name).getColumnNames();
+ std::vector<std::string> row;
+ for (const auto& col_name : col_names) {
+ auto it = std::find(insert_col_names.begin(), insert_col_names.end(), col_name);
+ if (it != insert_col_names.end()) {
+ row.push_back(values.at(it-insert_col_names.begin()));
+ } else {
+ row.push_back("NULL");
+ }
+ }
+ tables_.at(table_name).addRow(row);
+ } else {
+ tables_.at(table_name).addRow(values);
+ }
+
+ storeDb();
+}
+
+std::unique_ptr<Rowset> MockDB::select(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("select\\s+(.+)\\s+from\\s+(\\w+)\\s*(where ((.+(?= order by))|.+$))*\\s*(order by (.+))*");
+ std::regex_search(replaced_query, match, expr);
+ auto cols = minifi::utils::StringUtils::splitAndTrim(match[1], ",");
+ if (cols[0] == "*") {
+ cols = {};
+ }
+ std::string table_name = match[2];
+ std::string condition_str = match[4];
+ std::function<bool(const MockRow&)> condition;
+ if (!condition_str.empty()) {
+ if (condition_str == "int_col > 103") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) > 103; };
+ } else if (condition_str == "int_col > 102") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) > 102; };
+ } else if (condition_str == "int_col = 11") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) == 11; };
+ } else if (condition_str == "int_col = 11 and text_col = banana") {
Review comment:
When replacing arguments I did not add apostrophes so it worked like this. The code was removed in favor of 96c3a9dea8e0d9c0492ccf5850184755651418e9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r647430794
##########
File path: libminifi/include/utils/StringUtils.h
##########
@@ -137,6 +137,7 @@ class StringUtils {
static std::vector<std::string> split(const std::string &str, const std::string &delimiter);
static std::vector<std::string> splitAndTrim(const std::string &str, const std::string &delimiter);
+ static std::vector<std::string> splitAndTrimOnString(std::string str, const std::string& delimiter);
Review comment:
Thanks for both of your inputs. I changed `split` and `splitAndTrim` to use full strings and changed the semantics to keep the zero length segments. I also added `*RemovingEmpty` versions for utility. Changes are incorporated in 3ee4f1357bf03ec5bed69a3ca63801d128e739f8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r678913235
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.cpp
##########
@@ -0,0 +1,386 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "MockConnectors.h"
+
+#include <fstream>
+#include <algorithm>
+#include <utility>
+#include <string>
+#include <memory>
+
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+std::size_t MockRow::size() const {
+ return column_names_.size();
+}
+
+std::string MockRow::getColumnName(std::size_t index) const {
+ return column_names_.at(index);
+}
+
+bool MockRow::isNull(std::size_t index) const {
+ return column_values_.at(index) == "NULL";
+}
+
+DataType MockRow::getDataType(std::size_t index) const {
+ return column_types_.at(index);
+}
+
+std::string MockRow::getString(std::size_t index) const {
+ return column_values_.at(index);
+}
+
+double MockRow::getDouble(std::size_t index) const {
+ return std::atof(column_values_.at(index).c_str());
+}
+
+int MockRow::getInteger(std::size_t index) const {
+ return std::atoi(column_values_.at(index).c_str());
+}
+
+long long MockRow::getLongLong(std::size_t index) const { // NOLINT
+ return std::atoll(column_values_.at(index).c_str());
+}
+
+unsigned long long MockRow::getUnsignedLongLong(std::size_t index) const { // NOLINT
+ return static_cast<unsigned long long>(std::atoll(column_values_.at(index).c_str())); // NOLINT
+}
+
+std::tm MockRow::getDate(std::size_t /*index*/) const {
+ throw std::runtime_error("date not implemented");
+}
+
+std::vector<std::string> MockRow::getValues() const {
+ return column_values_;
+}
+
+std::string MockRow::getValue(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return column_values_.at(it-column_names_.begin());
+ }
+ throw std::runtime_error("Unknown column name for getting value");
+}
+
+void MockRowset::addRow(const std::vector<std::string>& column_values) {
+ rows_.emplace_back(column_names_, column_types_, column_values);
+}
+
+void MockRowset::reset() {
+ current_row_ = rows_.begin();
+}
+
+bool MockRowset::is_done() {
+ return current_row_ == rows_.end();
+}
+
+Row& MockRowset::getCurrent() {
+ return *current_row_;
+}
+
+void MockRowset::next() {
+ ++current_row_;
+}
+
+std::vector<std::string> MockRowset::getColumnNames() const {
+ return column_names_;
+}
+
+std::vector<DataType> MockRowset::getColumnTypes() const {
+ return column_types_;
+}
+
+std::vector<MockRow> MockRowset::getRows() const {
+ return rows_;
+}
+
+std::size_t MockRowset::getColumnIndex(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return it-column_names_.begin();
+ }
+ throw std::runtime_error("Unknown column name for getting index");
+}
+
+void MockRowset::sort(const std::string& order_by_col, bool order_ascending) {
+ std::sort(rows_.begin(), rows_.end(), [&](const MockRow& first, const MockRow& second) {
+ if (order_ascending) {
+ return first.getValue(order_by_col) < second.getValue(order_by_col);
+ } else {
+ return first.getValue(order_by_col) > second.getValue(order_by_col);
+ }
+ });
+}
+
+std::unique_ptr<MockRowset> MockRowset::select(const std::vector<std::string>& cols, const std::function<bool(const MockRow&)>& condition, const std::string& order_by_col, bool order_ascending) {
+ if (!order_by_col.empty()) {
+ sort(order_by_col, order_ascending);
+ }
+
+ std::unique_ptr<MockRowset> rowset;
+ if (cols.empty()) {
+ rowset = utils::make_unique<MockRowset>(column_names_, column_types_);
+ } else {
+ std::vector<DataType> col_types;
+ for (const auto& col : cols) {
+ col_types.push_back(column_types_.at(getColumnIndex(col)));
+ }
+ rowset = utils::make_unique<MockRowset>(cols, col_types);
+ }
+
+ std::vector<std::string> used_cols = cols.empty() ? column_names_ : cols;
+ for (const auto& row : rows_) {
+ if (condition(row)) {
+ std::vector<std::string> values;
+ for (const auto& col : used_cols) {
+ values.push_back(row.getValue(col));
+ }
+ rowset->addRow(values);
+ }
+ }
+
+ return rowset;
+}
+
+std::unique_ptr<Rowset> MockDB::execute(const std::string& query, const std::vector<std::string>& args) {
+ if (minifi::utils::StringUtils::startsWith(query, "create table")) {
+ createTable(query);
+ } else if (minifi::utils::StringUtils::startsWith(query, "insert into")) {
+ insertInto(query, args);
+ } else if (minifi::utils::StringUtils::startsWith(query, "select")) {
+ return select(query, args);
+ } else {
+ throw std::runtime_error("Unknown query type");
+ }
+
+ return nullptr;
+}
+
+void MockDB::createTable(const std::string& query) {
+ std::smatch match;
+ std::regex expr("create table (\\w+)\\s*\\((.*)\\);");
+ std::regex_search(query, match, expr);
+ std::string table_name = match[1];
+ auto columns_with_type = minifi::utils::StringUtils::splitAndTrim(match[2], ",");
+ std::vector<std::string> col_names;
+ std::vector<DataType> col_types;
+ for (const auto& col_with_type : columns_with_type) {
+ auto splitted = minifi::utils::StringUtils::splitAndTrim(col_with_type, " ");
+ col_names.push_back(splitted[0]);
+ col_types.push_back(stringToDataType(splitted[1]));
+ }
+ tables_.emplace(table_name, MockRowset{col_names, col_types});
+ storeDb();
+}
+
+void MockDB::insertInto(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("insert into (\\w+)\\s*(\\((.*)\\))*\\s*values\\s*\\((.+)\\)");
+ std::regex_search(replaced_query, match, expr);
+ std::string table_name = match[1];
+ std::vector<std::string> values = minifi::utils::StringUtils::splitAndTrim(match[4], ",");
+ for (auto& value : values) {
+ value = minifi::utils::StringUtils::removeFramingCharacters(value, '\'');
+ }
+ auto insert_col_names = minifi::utils::StringUtils::splitAndTrim(match[3], ",");
+ if (!insert_col_names.empty()) {
+ auto col_names = tables_.at(table_name).getColumnNames();
+ std::vector<std::string> row;
+ for (const auto& col_name : col_names) {
+ auto it = std::find(insert_col_names.begin(), insert_col_names.end(), col_name);
+ if (it != insert_col_names.end()) {
+ row.push_back(values.at(it-insert_col_names.begin()));
+ } else {
+ row.push_back("NULL");
+ }
+ }
+ tables_.at(table_name).addRow(row);
+ } else {
+ tables_.at(table_name).addRow(values);
+ }
+
+ storeDb();
+}
+
+std::unique_ptr<Rowset> MockDB::select(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("select\\s+(.+)\\s+from\\s+(\\w+)\\s*(where ((.+(?= order by))|.+$))*\\s*(order by (.+))*");
+ std::regex_search(replaced_query, match, expr);
+ auto cols = minifi::utils::StringUtils::splitAndTrim(match[1], ",");
+ if (cols[0] == "*") {
+ cols = {};
+ }
+ std::string table_name = match[2];
+ std::string condition_str = match[4];
+ std::function<bool(const MockRow&)> condition;
+ if (!condition_str.empty()) {
+ if (condition_str == "int_col > 103") {
Review comment:
a comment might come in handy, that it parses an `and`-separated list of conditions in the format of `<column_name> [<>=] <value>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r644729031
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.cpp
##########
@@ -0,0 +1,386 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "MockConnectors.h"
+
+#include <fstream>
+#include <algorithm>
+#include <utility>
+#include <string>
+#include <memory>
+
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+std::size_t MockRow::size() const {
+ return column_names_.size();
+}
+
+std::string MockRow::getColumnName(std::size_t index) const {
+ return column_names_.at(index);
+}
+
+bool MockRow::isNull(std::size_t index) const {
+ return column_values_.at(index) == "NULL";
+}
+
+DataType MockRow::getDataType(std::size_t index) const {
+ return column_types_.at(index);
+}
+
+std::string MockRow::getString(std::size_t index) const {
+ return column_values_.at(index);
+}
+
+double MockRow::getDouble(std::size_t index) const {
+ return std::atof(column_values_.at(index).c_str());
+}
+
+int MockRow::getInteger(std::size_t index) const {
+ return std::atoi(column_values_.at(index).c_str());
Review comment:
Updated in e905b1377f3689f285913e3886deab32a06cea4d
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r645517495
##########
File path: libminifi/include/utils/StringUtils.h
##########
@@ -137,6 +137,7 @@ class StringUtils {
static std::vector<std::string> split(const std::string &str, const std::string &delimiter);
static std::vector<std::string> splitAndTrim(const std::string &str, const std::string &delimiter);
+ static std::vector<std::string> splitAndTrimOnString(std::string str, const std::string& delimiter);
Review comment:
3 reasonable ways of using the delimiter string if it's longer than a character is
1. Match any of the characters in the string. Strings like this would be reasonable: `" \t;:,"`. C# does this.
2. Match the whole delimiter string as a delimiter. This is the case with ...OnString now. JavaScript and Python does this.
3. Match the whole delimiter string as a regex. Java does this, but I wouln't go this route.
Taking a string and only using its first character is a bit strange to me, I would change that behavior. The fact that it's trimmed after splitting should be relatively separate, but convenience functions are OK.
I can't think of a good and simple solution that does it all and that I wouldn't want to change eventually, so I'm not making any suggestions, but I'm happy to provide my opinion if you want to change it. Please consider adding comments that clarify the behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r644587782
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.cpp
##########
@@ -0,0 +1,386 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "MockConnectors.h"
+
+#include <fstream>
+#include <algorithm>
+#include <utility>
+#include <string>
+#include <memory>
+
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+std::size_t MockRow::size() const {
+ return column_names_.size();
+}
+
+std::string MockRow::getColumnName(std::size_t index) const {
+ return column_names_.at(index);
+}
+
+bool MockRow::isNull(std::size_t index) const {
+ return column_values_.at(index) == "NULL";
+}
+
+DataType MockRow::getDataType(std::size_t index) const {
+ return column_types_.at(index);
+}
+
+std::string MockRow::getString(std::size_t index) const {
+ return column_values_.at(index);
+}
+
+double MockRow::getDouble(std::size_t index) const {
+ return std::atof(column_values_.at(index).c_str());
+}
+
+int MockRow::getInteger(std::size_t index) const {
+ return std::atoi(column_values_.at(index).c_str());
+}
+
+long long MockRow::getLongLong(std::size_t index) const { // NOLINT
+ return std::atoll(column_values_.at(index).c_str());
+}
+
+unsigned long long MockRow::getUnsignedLongLong(std::size_t index) const { // NOLINT
+ return static_cast<unsigned long long>(std::atoll(column_values_.at(index).c_str())); // NOLINT
+}
+
+std::tm MockRow::getDate(std::size_t /*index*/) const {
+ throw std::runtime_error("date not implemented");
+}
+
+std::vector<std::string> MockRow::getValues() const {
+ return column_values_;
+}
+
+std::string MockRow::getValue(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return column_values_.at(it-column_names_.begin());
+ }
+ throw std::runtime_error("Unknown column name for getting value");
+}
+
+void MockRowset::addRow(const std::vector<std::string>& column_values) {
+ rows_.emplace_back(column_names_, column_types_, column_values);
+}
+
+void MockRowset::reset() {
+ current_row_ = rows_.begin();
+}
+
+bool MockRowset::is_done() {
+ return current_row_ == rows_.end();
+}
+
+Row& MockRowset::getCurrent() {
+ return *current_row_;
+}
+
+void MockRowset::next() {
+ ++current_row_;
+}
+
+std::vector<std::string> MockRowset::getColumnNames() const {
+ return column_names_;
+}
+
+std::vector<DataType> MockRowset::getColumnTypes() const {
+ return column_types_;
+}
+
+std::vector<MockRow> MockRowset::getRows() const {
+ return rows_;
+}
+
+std::size_t MockRowset::getColumnIndex(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return it-column_names_.begin();
+ }
+ throw std::runtime_error("Unknown column name for getting index");
+}
+
+void MockRowset::sort(const std::string& order_by_col, bool order_ascending) {
+ std::sort(rows_.begin(), rows_.end(), [&](const MockRow& first, const MockRow& second) {
+ if (order_ascending) {
+ return first.getValue(order_by_col) < second.getValue(order_by_col);
+ } else {
+ return first.getValue(order_by_col) > second.getValue(order_by_col);
+ }
+ });
+}
+
+std::unique_ptr<MockRowset> MockRowset::select(const std::vector<std::string>& cols, const std::function<bool(const MockRow&)>& condition, const std::string& order_by_col, bool order_ascending) {
+ if (!order_by_col.empty()) {
+ sort(order_by_col, order_ascending);
+ }
+
+ std::unique_ptr<MockRowset> rowset;
+ if (cols.empty()) {
+ rowset = utils::make_unique<MockRowset>(column_names_, column_types_);
+ } else {
+ std::vector<DataType> col_types;
+ for (const auto& col : cols) {
+ col_types.push_back(column_types_.at(getColumnIndex(col)));
+ }
+ rowset = utils::make_unique<MockRowset>(cols, col_types);
+ }
+
+ std::vector<std::string> used_cols = cols.empty() ? column_names_ : cols;
+ for (const auto& row : rows_) {
+ if (condition(row)) {
+ std::vector<std::string> values;
+ for (const auto& col : used_cols) {
+ values.push_back(row.getValue(col));
+ }
+ rowset->addRow(values);
+ }
+ }
+
+ return rowset;
+}
+
+std::unique_ptr<Rowset> MockDB::execute(const std::string& query, const std::vector<std::string>& args) {
+ if (minifi::utils::StringUtils::startsWith(query, "create table")) {
+ createTable(query);
+ } else if (minifi::utils::StringUtils::startsWith(query, "insert into")) {
+ insertInto(query, args);
+ } else if (minifi::utils::StringUtils::startsWith(query, "select")) {
+ return select(query, args);
+ } else {
+ throw std::runtime_error("Unknown query type");
+ }
+
+ return nullptr;
+}
+
+void MockDB::createTable(const std::string& query) {
+ std::smatch match;
+ std::regex expr("create table (\\w+)\\s*\\((.*)\\);");
+ std::regex_search(query, match, expr);
+ std::string table_name = match[1];
+ auto columns_with_type = minifi::utils::StringUtils::splitAndTrim(match[2], ",");
+ std::vector<std::string> col_names;
+ std::vector<DataType> col_types;
+ for (const auto& col_with_type : columns_with_type) {
+ auto splitted = minifi::utils::StringUtils::splitAndTrim(col_with_type, " ");
+ col_names.push_back(splitted[0]);
+ col_types.push_back(stringToDataType(splitted[1]));
+ }
+ tables_.emplace(table_name, MockRowset{col_names, col_types});
+ storeDb();
+}
+
+void MockDB::insertInto(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("insert into (\\w+)\\s*(\\((.*)\\))*\\s*values\\s*\\((.+)\\)");
+ std::regex_search(replaced_query, match, expr);
+ std::string table_name = match[1];
+ std::vector<std::string> values = minifi::utils::StringUtils::splitAndTrim(match[4], ",");
+ for (auto& value : values) {
+ value = minifi::utils::StringUtils::removeFramingCharacters(value, '\'');
+ }
+ auto insert_col_names = minifi::utils::StringUtils::splitAndTrim(match[3], ",");
+ if (!insert_col_names.empty()) {
+ auto col_names = tables_.at(table_name).getColumnNames();
+ std::vector<std::string> row;
+ for (const auto& col_name : col_names) {
+ auto it = std::find(insert_col_names.begin(), insert_col_names.end(), col_name);
+ if (it != insert_col_names.end()) {
+ row.push_back(values.at(it-insert_col_names.begin()));
+ } else {
+ row.push_back("NULL");
+ }
+ }
+ tables_.at(table_name).addRow(row);
+ } else {
+ tables_.at(table_name).addRow(values);
+ }
+
+ storeDb();
+}
+
+std::unique_ptr<Rowset> MockDB::select(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("select\\s+(.+)\\s+from\\s+(\\w+)\\s*(where ((.+(?= order by))|.+$))*\\s*(order by (.+))*");
+ std::regex_search(replaced_query, match, expr);
+ auto cols = minifi::utils::StringUtils::splitAndTrim(match[1], ",");
+ if (cols[0] == "*") {
+ cols = {};
+ }
+ std::string table_name = match[2];
+ std::string condition_str = match[4];
+ std::function<bool(const MockRow&)> condition;
+ if (!condition_str.empty()) {
+ if (condition_str == "int_col > 103") {
Review comment:
Two problems with reusing Expression.cpp:
1. It's not available when compiling with expression language support disabled
2. It's parsing nifi expression language, but this looks like SQL to me
Not sure what would be the best way to solve this.
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.h
##########
@@ -0,0 +1,175 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <regex>
+#include <map>
+#include <vector>
+#include <algorithm>
+#include <memory>
+#include <string>
+
+#include "data/DatabaseConnectors.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+class MockRow : public Row {
+ public:
+ MockRow(std::vector<std::string>& column_names, std::vector<DataType>& column_types, const std::vector<std::string>& column_values)
+ : column_names_(column_names), column_types_(column_types), column_values_(column_values) {
+ }
+
+ MockRow(MockRow&& other) = default;
+ MockRow(const MockRow& other) = default;
+ MockRow& operator=(MockRow&& other) {
+ column_names_ = other.column_names_;
+ column_types_ = other.column_types_;
Review comment:
I'd say just use pointers. Whether you write `*column_names_` or `column_names_.get()` doesn't really matter. Raw pointers are not evil as observer pointers.
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.cpp
##########
@@ -0,0 +1,386 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "MockConnectors.h"
+
+#include <fstream>
+#include <algorithm>
+#include <utility>
+#include <string>
+#include <memory>
+
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+std::size_t MockRow::size() const {
+ return column_names_.size();
+}
+
+std::string MockRow::getColumnName(std::size_t index) const {
+ return column_names_.at(index);
+}
+
+bool MockRow::isNull(std::size_t index) const {
+ return column_values_.at(index) == "NULL";
+}
+
+DataType MockRow::getDataType(std::size_t index) const {
+ return column_types_.at(index);
+}
+
+std::string MockRow::getString(std::size_t index) const {
+ return column_values_.at(index);
+}
+
+double MockRow::getDouble(std::size_t index) const {
+ return std::atof(column_values_.at(index).c_str());
+}
+
+int MockRow::getInteger(std::size_t index) const {
+ return std::atoi(column_values_.at(index).c_str());
+}
+
+long long MockRow::getLongLong(std::size_t index) const { // NOLINT
+ return std::atoll(column_values_.at(index).c_str());
+}
+
+unsigned long long MockRow::getUnsignedLongLong(std::size_t index) const { // NOLINT
+ return static_cast<unsigned long long>(std::atoll(column_values_.at(index).c_str())); // NOLINT
+}
+
+std::tm MockRow::getDate(std::size_t /*index*/) const {
+ throw std::runtime_error("date not implemented");
+}
+
+std::vector<std::string> MockRow::getValues() const {
+ return column_values_;
+}
+
+std::string MockRow::getValue(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return column_values_.at(it-column_names_.begin());
+ }
+ throw std::runtime_error("Unknown column name for getting value");
+}
+
+void MockRowset::addRow(const std::vector<std::string>& column_values) {
+ rows_.emplace_back(column_names_, column_types_, column_values);
+}
+
+void MockRowset::reset() {
+ current_row_ = rows_.begin();
+}
+
+bool MockRowset::is_done() {
+ return current_row_ == rows_.end();
+}
+
+Row& MockRowset::getCurrent() {
+ return *current_row_;
+}
+
+void MockRowset::next() {
+ ++current_row_;
+}
+
+std::vector<std::string> MockRowset::getColumnNames() const {
+ return column_names_;
+}
+
+std::vector<DataType> MockRowset::getColumnTypes() const {
+ return column_types_;
+}
+
+std::vector<MockRow> MockRowset::getRows() const {
+ return rows_;
+}
+
+std::size_t MockRowset::getColumnIndex(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return it-column_names_.begin();
+ }
+ throw std::runtime_error("Unknown column name for getting index");
+}
+
+void MockRowset::sort(const std::string& order_by_col, bool order_ascending) {
+ std::sort(rows_.begin(), rows_.end(), [&](const MockRow& first, const MockRow& second) {
+ if (order_ascending) {
+ return first.getValue(order_by_col) < second.getValue(order_by_col);
+ } else {
+ return first.getValue(order_by_col) > second.getValue(order_by_col);
+ }
+ });
+}
+
+std::unique_ptr<MockRowset> MockRowset::select(const std::vector<std::string>& cols, const std::function<bool(const MockRow&)>& condition, const std::string& order_by_col, bool order_ascending) {
+ if (!order_by_col.empty()) {
+ sort(order_by_col, order_ascending);
+ }
+
+ std::unique_ptr<MockRowset> rowset;
+ if (cols.empty()) {
+ rowset = utils::make_unique<MockRowset>(column_names_, column_types_);
+ } else {
+ std::vector<DataType> col_types;
+ for (const auto& col : cols) {
+ col_types.push_back(column_types_.at(getColumnIndex(col)));
+ }
+ rowset = utils::make_unique<MockRowset>(cols, col_types);
+ }
+
+ std::vector<std::string> used_cols = cols.empty() ? column_names_ : cols;
+ for (const auto& row : rows_) {
+ if (condition(row)) {
+ std::vector<std::string> values;
+ for (const auto& col : used_cols) {
+ values.push_back(row.getValue(col));
+ }
+ rowset->addRow(values);
+ }
+ }
+
+ return rowset;
+}
+
+std::unique_ptr<Rowset> MockDB::execute(const std::string& query, const std::vector<std::string>& args) {
+ if (minifi::utils::StringUtils::startsWith(query, "create table")) {
+ createTable(query);
+ } else if (minifi::utils::StringUtils::startsWith(query, "insert into")) {
+ insertInto(query, args);
+ } else if (minifi::utils::StringUtils::startsWith(query, "select")) {
+ return select(query, args);
+ } else {
+ throw std::runtime_error("Unknown query type");
+ }
+
+ return nullptr;
+}
+
+void MockDB::createTable(const std::string& query) {
+ std::smatch match;
+ std::regex expr("create table (\\w+)\\s*\\((.*)\\);");
+ std::regex_search(query, match, expr);
+ std::string table_name = match[1];
+ auto columns_with_type = minifi::utils::StringUtils::splitAndTrim(match[2], ",");
+ std::vector<std::string> col_names;
+ std::vector<DataType> col_types;
+ for (const auto& col_with_type : columns_with_type) {
+ auto splitted = minifi::utils::StringUtils::splitAndTrim(col_with_type, " ");
+ col_names.push_back(splitted[0]);
+ col_types.push_back(stringToDataType(splitted[1]));
+ }
+ tables_.emplace(table_name, MockRowset{col_names, col_types});
+ storeDb();
+}
+
+void MockDB::insertInto(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("insert into (\\w+)\\s*(\\((.*)\\))*\\s*values\\s*\\((.+)\\)");
+ std::regex_search(replaced_query, match, expr);
+ std::string table_name = match[1];
+ std::vector<std::string> values = minifi::utils::StringUtils::splitAndTrim(match[4], ",");
+ for (auto& value : values) {
+ value = minifi::utils::StringUtils::removeFramingCharacters(value, '\'');
+ }
+ auto insert_col_names = minifi::utils::StringUtils::splitAndTrim(match[3], ",");
+ if (!insert_col_names.empty()) {
+ auto col_names = tables_.at(table_name).getColumnNames();
+ std::vector<std::string> row;
+ for (const auto& col_name : col_names) {
+ auto it = std::find(insert_col_names.begin(), insert_col_names.end(), col_name);
+ if (it != insert_col_names.end()) {
+ row.push_back(values.at(it-insert_col_names.begin()));
+ } else {
+ row.push_back("NULL");
+ }
+ }
+ tables_.at(table_name).addRow(row);
+ } else {
+ tables_.at(table_name).addRow(values);
+ }
+
+ storeDb();
+}
+
+std::unique_ptr<Rowset> MockDB::select(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("select\\s+(.+)\\s+from\\s+(\\w+)\\s*(where ((.+(?= order by))|.+$))*\\s*(order by (.+))*");
+ std::regex_search(replaced_query, match, expr);
+ auto cols = minifi::utils::StringUtils::splitAndTrim(match[1], ",");
+ if (cols[0] == "*") {
+ cols = {};
+ }
+ std::string table_name = match[2];
+ std::string condition_str = match[4];
+ std::function<bool(const MockRow&)> condition;
+ if (!condition_str.empty()) {
+ if (condition_str == "int_col > 103") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) > 103; };
+ } else if (condition_str == "int_col > 102") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) > 102; };
+ } else if (condition_str == "int_col = 11") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) == 11; };
+ } else if (condition_str == "int_col = 11 and text_col = banana") {
Review comment:
Shouldn't "banana" be in apostrophes here? Unless it's another column.
##########
File path: extensions/sql/data/DatabaseConnectors.h
##########
@@ -20,72 +20,68 @@
#include <memory>
#include <string>
-
-#include <soci/soci.h>
-
-#include "Utils.h"
+#include <vector>
+#include <ctime>
namespace org {
namespace apache {
namespace nifi {
namespace minifi {
namespace sql {
-/**
- * We do not intend to create an abstract facade here. We know that SOCI is the underlying
- * SQL library. We only wish to abstract ODBC specific information
- */
+enum class DataType {
+ STRING,
+ DOUBLE,
+ INTEGER,
+ LONG_LONG,
+ UNSIGNED_LONG_LONG,
+ DATE
+};
-class Statement {
+class Row {
public:
+ virtual ~Row() = default;
+ virtual std::size_t size() const = 0;
+ virtual std::string getColumnName(std::size_t index) const = 0;
+ virtual bool isNull(std::size_t index) const = 0;
+ virtual DataType getDataType(std::size_t index) const = 0;
+ virtual std::string getString(std::size_t index) const = 0;
+ virtual double getDouble(std::size_t index) const = 0;
+ virtual int getInteger(std::size_t index) const = 0;
+ virtual long long getLongLong(std::size_t index) const = 0;
+ virtual unsigned long long getUnsignedLongLong(std::size_t index) const = 0;
+ virtual std::tm getDate(std::size_t index) const = 0;
Review comment:
This could be a `std::chrono::time_point` or if you only need the date part, then `date::year_month_day`. These types are easier to work with.
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.h
##########
@@ -0,0 +1,175 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <regex>
+#include <map>
+#include <vector>
+#include <algorithm>
+#include <memory>
+#include <string>
+
+#include "data/DatabaseConnectors.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+class MockRow : public Row {
+ public:
+ MockRow(std::vector<std::string>& column_names, std::vector<DataType>& column_types, const std::vector<std::string>& column_values)
+ : column_names_(column_names), column_types_(column_types), column_values_(column_values) {
+ }
+
+ MockRow(MockRow&& other) = default;
+ MockRow(const MockRow& other) = default;
+ MockRow& operator=(MockRow&& other) {
+ column_names_ = other.column_names_;
+ column_types_ = other.column_types_;
Review comment:
I'd say just use pointers. Whether you write `*column_names_` or `column_names_.get()` doesn't really matter. Raw pointers are not evil as observer pointers.
edit: looks like I was late with this reply :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r678935221
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.cpp
##########
@@ -0,0 +1,386 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "MockConnectors.h"
+
+#include <fstream>
+#include <algorithm>
+#include <utility>
+#include <string>
+#include <memory>
+
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+std::size_t MockRow::size() const {
+ return column_names_.size();
+}
+
+std::string MockRow::getColumnName(std::size_t index) const {
+ return column_names_.at(index);
+}
+
+bool MockRow::isNull(std::size_t index) const {
+ return column_values_.at(index) == "NULL";
+}
+
+DataType MockRow::getDataType(std::size_t index) const {
+ return column_types_.at(index);
+}
+
+std::string MockRow::getString(std::size_t index) const {
+ return column_values_.at(index);
+}
+
+double MockRow::getDouble(std::size_t index) const {
+ return std::atof(column_values_.at(index).c_str());
+}
+
+int MockRow::getInteger(std::size_t index) const {
+ return std::atoi(column_values_.at(index).c_str());
+}
+
+long long MockRow::getLongLong(std::size_t index) const { // NOLINT
+ return std::atoll(column_values_.at(index).c_str());
+}
+
+unsigned long long MockRow::getUnsignedLongLong(std::size_t index) const { // NOLINT
+ return static_cast<unsigned long long>(std::atoll(column_values_.at(index).c_str())); // NOLINT
+}
+
+std::tm MockRow::getDate(std::size_t /*index*/) const {
+ throw std::runtime_error("date not implemented");
+}
+
+std::vector<std::string> MockRow::getValues() const {
+ return column_values_;
+}
+
+std::string MockRow::getValue(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return column_values_.at(it-column_names_.begin());
+ }
+ throw std::runtime_error("Unknown column name for getting value");
+}
+
+void MockRowset::addRow(const std::vector<std::string>& column_values) {
+ rows_.emplace_back(column_names_, column_types_, column_values);
+}
+
+void MockRowset::reset() {
+ current_row_ = rows_.begin();
+}
+
+bool MockRowset::is_done() {
+ return current_row_ == rows_.end();
+}
+
+Row& MockRowset::getCurrent() {
+ return *current_row_;
+}
+
+void MockRowset::next() {
+ ++current_row_;
+}
+
+std::vector<std::string> MockRowset::getColumnNames() const {
+ return column_names_;
+}
+
+std::vector<DataType> MockRowset::getColumnTypes() const {
+ return column_types_;
+}
+
+std::vector<MockRow> MockRowset::getRows() const {
+ return rows_;
+}
+
+std::size_t MockRowset::getColumnIndex(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return it-column_names_.begin();
+ }
+ throw std::runtime_error("Unknown column name for getting index");
+}
+
+void MockRowset::sort(const std::string& order_by_col, bool order_ascending) {
+ std::sort(rows_.begin(), rows_.end(), [&](const MockRow& first, const MockRow& second) {
+ if (order_ascending) {
+ return first.getValue(order_by_col) < second.getValue(order_by_col);
+ } else {
+ return first.getValue(order_by_col) > second.getValue(order_by_col);
+ }
+ });
+}
+
+std::unique_ptr<MockRowset> MockRowset::select(const std::vector<std::string>& cols, const std::function<bool(const MockRow&)>& condition, const std::string& order_by_col, bool order_ascending) {
+ if (!order_by_col.empty()) {
+ sort(order_by_col, order_ascending);
+ }
+
+ std::unique_ptr<MockRowset> rowset;
+ if (cols.empty()) {
+ rowset = utils::make_unique<MockRowset>(column_names_, column_types_);
+ } else {
+ std::vector<DataType> col_types;
+ for (const auto& col : cols) {
+ col_types.push_back(column_types_.at(getColumnIndex(col)));
+ }
+ rowset = utils::make_unique<MockRowset>(cols, col_types);
+ }
+
+ std::vector<std::string> used_cols = cols.empty() ? column_names_ : cols;
+ for (const auto& row : rows_) {
+ if (condition(row)) {
+ std::vector<std::string> values;
+ for (const auto& col : used_cols) {
+ values.push_back(row.getValue(col));
+ }
+ rowset->addRow(values);
+ }
+ }
+
+ return rowset;
+}
+
+std::unique_ptr<Rowset> MockDB::execute(const std::string& query, const std::vector<std::string>& args) {
+ if (minifi::utils::StringUtils::startsWith(query, "create table")) {
+ createTable(query);
+ } else if (minifi::utils::StringUtils::startsWith(query, "insert into")) {
+ insertInto(query, args);
+ } else if (minifi::utils::StringUtils::startsWith(query, "select")) {
+ return select(query, args);
+ } else {
+ throw std::runtime_error("Unknown query type");
+ }
+
+ return nullptr;
+}
+
+void MockDB::createTable(const std::string& query) {
+ std::smatch match;
+ std::regex expr("create table (\\w+)\\s*\\((.*)\\);");
+ std::regex_search(query, match, expr);
+ std::string table_name = match[1];
+ auto columns_with_type = minifi::utils::StringUtils::splitAndTrim(match[2], ",");
+ std::vector<std::string> col_names;
+ std::vector<DataType> col_types;
+ for (const auto& col_with_type : columns_with_type) {
+ auto splitted = minifi::utils::StringUtils::splitAndTrim(col_with_type, " ");
+ col_names.push_back(splitted[0]);
+ col_types.push_back(stringToDataType(splitted[1]));
+ }
+ tables_.emplace(table_name, MockRowset{col_names, col_types});
+ storeDb();
+}
+
+void MockDB::insertInto(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("insert into (\\w+)\\s*(\\((.*)\\))*\\s*values\\s*\\((.+)\\)");
+ std::regex_search(replaced_query, match, expr);
+ std::string table_name = match[1];
+ std::vector<std::string> values = minifi::utils::StringUtils::splitAndTrim(match[4], ",");
+ for (auto& value : values) {
+ value = minifi::utils::StringUtils::removeFramingCharacters(value, '\'');
+ }
+ auto insert_col_names = minifi::utils::StringUtils::splitAndTrim(match[3], ",");
+ if (!insert_col_names.empty()) {
+ auto col_names = tables_.at(table_name).getColumnNames();
+ std::vector<std::string> row;
+ for (const auto& col_name : col_names) {
+ auto it = std::find(insert_col_names.begin(), insert_col_names.end(), col_name);
+ if (it != insert_col_names.end()) {
+ row.push_back(values.at(it-insert_col_names.begin()));
+ } else {
+ row.push_back("NULL");
+ }
+ }
+ tables_.at(table_name).addRow(row);
+ } else {
+ tables_.at(table_name).addRow(values);
+ }
+
+ storeDb();
+}
+
+std::unique_ptr<Rowset> MockDB::select(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("select\\s+(.+)\\s+from\\s+(\\w+)\\s*(where ((.+(?= order by))|.+$))*\\s*(order by (.+))*");
+ std::regex_search(replaced_query, match, expr);
+ auto cols = minifi::utils::StringUtils::splitAndTrim(match[1], ",");
+ if (cols[0] == "*") {
+ cols = {};
+ }
+ std::string table_name = match[2];
+ std::string condition_str = match[4];
+ std::function<bool(const MockRow&)> condition;
+ if (!condition_str.empty()) {
+ if (condition_str == "int_col > 103") {
Review comment:
Added comment in 182d8a534045a230857d7c5dc5116466f5177bcf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r644548803
##########
File path: win_build_vs.bat
##########
@@ -65,7 +65,7 @@ for %%x in (%*) do (
mkdir %builddir%
pushd %builddir%\
-cmake -G %generator% -A %build_platform% -DINSTALLER_MERGE_MODULES=%installer_merge_modules% -DTEST_CUSTOM_WEL_PROVIDER=%test_custom_wel_provider% -DENABLE_SQL=%build_SQL% -DCMAKE_BUILD_TYPE_INIT=%cmake_build_type% -DCMAKE_BUILD_TYPE=%cmake_build_type% -DWIN32=WIN32 -DENABLE_LIBRDKAFKA=%build_kafka% -DENABLE_JNI=%build_jni% -DOPENSSL_OFF=OFF -DENABLE_COAP=%build_coap% -DENABLE_AWS=%build_AWS% -DENABLE_PDH=%build_PDH% -DENABLE_AZURE=%build_azure% -DENABLE_SFTP=%build_SFTP% -DUSE_SHARED_LIBS=OFF -DDISABLE_CONTROLLER=ON -DBUILD_ROCKSDB=ON -DFORCE_WINDOWS=ON -DUSE_SYSTEM_UUID=OFF -DDISABLE_LIBARCHIVE=OFF -DDISABLE_SCRIPTING=ON -DEXCLUDE_BOOST=ON -DENABLE_WEL=TRUE -DFAIL_ON_WARNINGS=OFF -DSKIP_TESTS=%skiptests% %strict_gsl_checks% %redist% -DENABLE_LINTER=%build_linter% .. && msbuild /m nifi-minifi-cpp.sln /property:Configuration=%cmake_build_type% /property:Platform=%build_platform% && copy main\%cmake_build_type%\minifi.exe main\
+cmake -G %generator% -A %build_platform% -DINSTALLER_MERGE_MODULES=%installer_merge_modules% -DTEST_CUSTOM_WEL_PROVIDER=%test_custom_wel_provider% -DENABLE_SQL=%build_SQL% -DUSE_REAL_ODBC_TEST_DRIVER=ON -DCMAKE_BUILD_TYPE_INIT=%cmake_build_type% -DCMAKE_BUILD_TYPE=%cmake_build_type% -DWIN32=WIN32 -DENABLE_LIBRDKAFKA=%build_kafka% -DENABLE_JNI=%build_jni% -DOPENSSL_OFF=OFF -DENABLE_COAP=%build_coap% -DENABLE_AWS=%build_AWS% -DENABLE_PDH=%build_PDH% -DENABLE_AZURE=%build_azure% -DENABLE_SFTP=%build_SFTP% -DUSE_SHARED_LIBS=OFF -DDISABLE_CONTROLLER=ON -DBUILD_ROCKSDB=ON -DFORCE_WINDOWS=ON -DUSE_SYSTEM_UUID=OFF -DDISABLE_LIBARCHIVE=OFF -DDISABLE_SCRIPTING=ON -DEXCLUDE_BOOST=ON -DENABLE_WEL=TRUE -DFAIL_ON_WARNINGS=OFF -DSKIP_TESTS=%skiptests% %strict_gsl_checks% %redist% -DENABLE_LINTER=%build_linter% .. && msbuild /m nifi-minifi-cpp.sln /property:Configuration=%cmake_build_type% /property:Platform=%build_platform% && copy main\%cmake_build_type%\minifi.exe main\
Review comment:
as I understand this will always make the tests try to use the real odbc driver on windows, shouldn't we introduce some flag which is only default-enabled in the ci?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r678901043
##########
File path: win_build_vs.bat
##########
@@ -65,7 +65,7 @@ for %%x in (%*) do (
mkdir %builddir%
pushd %builddir%\
-cmake -G %generator% -A %build_platform% -DINSTALLER_MERGE_MODULES=%installer_merge_modules% -DTEST_CUSTOM_WEL_PROVIDER=%test_custom_wel_provider% -DENABLE_SQL=%build_SQL% -DCMAKE_BUILD_TYPE_INIT=%cmake_build_type% -DCMAKE_BUILD_TYPE=%cmake_build_type% -DWIN32=WIN32 -DENABLE_LIBRDKAFKA=%build_kafka% -DENABLE_JNI=%build_jni% -DOPENSSL_OFF=OFF -DENABLE_COAP=%build_coap% -DENABLE_AWS=%build_AWS% -DENABLE_PDH=%build_PDH% -DENABLE_AZURE=%build_azure% -DENABLE_SFTP=%build_SFTP% -DUSE_SHARED_LIBS=OFF -DDISABLE_CONTROLLER=ON -DBUILD_ROCKSDB=ON -DFORCE_WINDOWS=ON -DUSE_SYSTEM_UUID=OFF -DDISABLE_LIBARCHIVE=OFF -DDISABLE_SCRIPTING=ON -DEXCLUDE_BOOST=ON -DENABLE_WEL=TRUE -DFAIL_ON_WARNINGS=OFF -DSKIP_TESTS=%skiptests% %strict_gsl_checks% %redist% -DENABLE_LINTER=%build_linter% .. && msbuild /m nifi-minifi-cpp.sln /property:Configuration=%cmake_build_type% /property:Platform=%build_platform% && copy main\%cmake_build_type%\minifi.exe main\
+cmake -G %generator% -A %build_platform% -DINSTALLER_MERGE_MODULES=%installer_merge_modules% -DTEST_CUSTOM_WEL_PROVIDER=%test_custom_wel_provider% -DENABLE_SQL=%build_SQL% -DUSE_REAL_ODBC_TEST_DRIVER=ON -DCMAKE_BUILD_TYPE_INIT=%cmake_build_type% -DCMAKE_BUILD_TYPE=%cmake_build_type% -DWIN32=WIN32 -DENABLE_LIBRDKAFKA=%build_kafka% -DENABLE_JNI=%build_jni% -DOPENSSL_OFF=OFF -DENABLE_COAP=%build_coap% -DENABLE_AWS=%build_AWS% -DENABLE_PDH=%build_PDH% -DENABLE_AZURE=%build_azure% -DENABLE_SFTP=%build_SFTP% -DUSE_SHARED_LIBS=OFF -DDISABLE_CONTROLLER=ON -DBUILD_ROCKSDB=ON -DFORCE_WINDOWS=ON -DUSE_SYSTEM_UUID=OFF -DDISABLE_LIBARCHIVE=OFF -DDISABLE_SCRIPTING=ON -DEXCLUDE_BOOST=ON -DENABLE_WEL=TRUE -DFAIL_ON_WARNINGS=OFF -DSKIP_TESTS=%skiptests% %strict_gsl_checks% %redist% -DENABLE_LINTER=%build_linter% .. && msbuild /m nifi-minifi-cpp.sln /property:Configuration=%cmake_build_type% /property:Platform=%build_platform% && copy main\%cmake_build_type%\minifi.exe main\
Review comment:
oh you are right, I was looking at the linked commit only
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r645542861
##########
File path: libminifi/include/utils/StringUtils.h
##########
@@ -137,6 +137,7 @@ class StringUtils {
static std::vector<std::string> split(const std::string &str, const std::string &delimiter);
static std::vector<std::string> splitAndTrim(const std::string &str, const std::string &delimiter);
+ static std::vector<std::string> splitAndTrimOnString(std::string str, const std::string& delimiter);
Review comment:
wow this is "great", I wondered why the following test passed knowing this:
```
TEST_CASE("TestStringUtils::split4", "[test split classname]") {
std::vector<std::string> expected = { "org", "apache", "nifi", "minifi", "utils", "StringUtils" };
REQUIRE(expected == StringUtils::split(org::apache::nifi::minifi::core::getClassName<org::apache::nifi::minifi::utils::StringUtils>(), "::"));
}
```
turns out two wrongs make a right here :D
the current implementation implicitly discards zero length segments, so `split("a,,b", ",") == ["a", "b"]` instead of `["a", "", "b"]`, I would definitely change this, I would keep a single function with `std::string` delimiter but with the "correct" semantics, if we would like to interpret each character as a delimiter later, we could add a `splitAny` or something along those lines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r670372104
##########
File path: libminifi/test/sql-tests/mocks/MockODBCService.h
##########
@@ -0,0 +1,66 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <memory>
+#include <string>
+
+#include "MockConnectors.h"
+#include "services/DatabaseService.h"
+#include "core/Resource.h"
+#include "data/DatabaseConnectors.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+namespace controllers {
+
+class MockODBCService : public DatabaseService {
+ public:
+ explicit MockODBCService(const std::string &name, utils::Identifier uuid = utils::Identifier())
+ : DatabaseService(name, uuid),
+ logger_(logging::LoggerFactory<MockODBCService>::getLogger()) {
+ initialize();
+ }
+
+ explicit MockODBCService(const std::string &name, const std::shared_ptr<Configure> &configuration)
+ : DatabaseService(name),
+ logger_(logging::LoggerFactory<MockODBCService>::getLogger()) {
+ setConfiguration(configuration);
+ initialize();
+ }
+
+ std::unique_ptr<sql::Connection> getConnection() const {
+ return std::unique_ptr<sql::Connection>(new MockODBCConnection(connection_string_));
Review comment:
Updated in 167cdb692db9fb1a8f429a8845447f98ee155103
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r644740799
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.cpp
##########
@@ -0,0 +1,386 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "MockConnectors.h"
+
+#include <fstream>
+#include <algorithm>
+#include <utility>
+#include <string>
+#include <memory>
+
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+std::size_t MockRow::size() const {
+ return column_names_.size();
+}
+
+std::string MockRow::getColumnName(std::size_t index) const {
+ return column_names_.at(index);
+}
+
+bool MockRow::isNull(std::size_t index) const {
+ return column_values_.at(index) == "NULL";
+}
+
+DataType MockRow::getDataType(std::size_t index) const {
+ return column_types_.at(index);
+}
+
+std::string MockRow::getString(std::size_t index) const {
+ return column_values_.at(index);
+}
+
+double MockRow::getDouble(std::size_t index) const {
+ return std::atof(column_values_.at(index).c_str());
+}
+
+int MockRow::getInteger(std::size_t index) const {
+ return std::atoi(column_values_.at(index).c_str());
+}
+
+long long MockRow::getLongLong(std::size_t index) const { // NOLINT
+ return std::atoll(column_values_.at(index).c_str());
+}
+
+unsigned long long MockRow::getUnsignedLongLong(std::size_t index) const { // NOLINT
+ return static_cast<unsigned long long>(std::atoll(column_values_.at(index).c_str())); // NOLINT
+}
+
+std::tm MockRow::getDate(std::size_t /*index*/) const {
+ throw std::runtime_error("date not implemented");
+}
+
+std::vector<std::string> MockRow::getValues() const {
+ return column_values_;
+}
+
+std::string MockRow::getValue(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return column_values_.at(it-column_names_.begin());
+ }
+ throw std::runtime_error("Unknown column name for getting value");
+}
+
+void MockRowset::addRow(const std::vector<std::string>& column_values) {
+ rows_.emplace_back(column_names_, column_types_, column_values);
+}
+
+void MockRowset::reset() {
+ current_row_ = rows_.begin();
+}
+
+bool MockRowset::is_done() {
+ return current_row_ == rows_.end();
+}
+
+Row& MockRowset::getCurrent() {
+ return *current_row_;
+}
+
+void MockRowset::next() {
+ ++current_row_;
+}
+
+std::vector<std::string> MockRowset::getColumnNames() const {
+ return column_names_;
+}
+
+std::vector<DataType> MockRowset::getColumnTypes() const {
+ return column_types_;
+}
+
+std::vector<MockRow> MockRowset::getRows() const {
+ return rows_;
+}
+
+std::size_t MockRowset::getColumnIndex(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return it-column_names_.begin();
+ }
+ throw std::runtime_error("Unknown column name for getting index");
+}
+
+void MockRowset::sort(const std::string& order_by_col, bool order_ascending) {
+ std::sort(rows_.begin(), rows_.end(), [&](const MockRow& first, const MockRow& second) {
+ if (order_ascending) {
+ return first.getValue(order_by_col) < second.getValue(order_by_col);
+ } else {
+ return first.getValue(order_by_col) > second.getValue(order_by_col);
+ }
+ });
+}
+
+std::unique_ptr<MockRowset> MockRowset::select(const std::vector<std::string>& cols, const std::function<bool(const MockRow&)>& condition, const std::string& order_by_col, bool order_ascending) {
+ if (!order_by_col.empty()) {
+ sort(order_by_col, order_ascending);
+ }
+
+ std::unique_ptr<MockRowset> rowset;
+ if (cols.empty()) {
+ rowset = utils::make_unique<MockRowset>(column_names_, column_types_);
+ } else {
+ std::vector<DataType> col_types;
+ for (const auto& col : cols) {
+ col_types.push_back(column_types_.at(getColumnIndex(col)));
+ }
+ rowset = utils::make_unique<MockRowset>(cols, col_types);
+ }
+
+ std::vector<std::string> used_cols = cols.empty() ? column_names_ : cols;
+ for (const auto& row : rows_) {
+ if (condition(row)) {
+ std::vector<std::string> values;
+ for (const auto& col : used_cols) {
+ values.push_back(row.getValue(col));
+ }
+ rowset->addRow(values);
+ }
+ }
+
+ return rowset;
+}
+
+std::unique_ptr<Rowset> MockDB::execute(const std::string& query, const std::vector<std::string>& args) {
+ if (minifi::utils::StringUtils::startsWith(query, "create table")) {
+ createTable(query);
+ } else if (minifi::utils::StringUtils::startsWith(query, "insert into")) {
+ insertInto(query, args);
+ } else if (minifi::utils::StringUtils::startsWith(query, "select")) {
+ return select(query, args);
+ } else {
+ throw std::runtime_error("Unknown query type");
+ }
+
+ return nullptr;
+}
+
+void MockDB::createTable(const std::string& query) {
+ std::smatch match;
+ std::regex expr("create table (\\w+)\\s*\\((.*)\\);");
+ std::regex_search(query, match, expr);
+ std::string table_name = match[1];
+ auto columns_with_type = minifi::utils::StringUtils::splitAndTrim(match[2], ",");
+ std::vector<std::string> col_names;
+ std::vector<DataType> col_types;
+ for (const auto& col_with_type : columns_with_type) {
+ auto splitted = minifi::utils::StringUtils::splitAndTrim(col_with_type, " ");
+ col_names.push_back(splitted[0]);
+ col_types.push_back(stringToDataType(splitted[1]));
+ }
+ tables_.emplace(table_name, MockRowset{col_names, col_types});
+ storeDb();
+}
+
+void MockDB::insertInto(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("insert into (\\w+)\\s*(\\((.*)\\))*\\s*values\\s*\\((.+)\\)");
+ std::regex_search(replaced_query, match, expr);
+ std::string table_name = match[1];
+ std::vector<std::string> values = minifi::utils::StringUtils::splitAndTrim(match[4], ",");
+ for (auto& value : values) {
+ value = minifi::utils::StringUtils::removeFramingCharacters(value, '\'');
+ }
+ auto insert_col_names = minifi::utils::StringUtils::splitAndTrim(match[3], ",");
+ if (!insert_col_names.empty()) {
+ auto col_names = tables_.at(table_name).getColumnNames();
+ std::vector<std::string> row;
+ for (const auto& col_name : col_names) {
+ auto it = std::find(insert_col_names.begin(), insert_col_names.end(), col_name);
+ if (it != insert_col_names.end()) {
+ row.push_back(values.at(it-insert_col_names.begin()));
+ } else {
+ row.push_back("NULL");
+ }
+ }
+ tables_.at(table_name).addRow(row);
+ } else {
+ tables_.at(table_name).addRow(values);
+ }
+
+ storeDb();
+}
+
+std::unique_ptr<Rowset> MockDB::select(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("select\\s+(.+)\\s+from\\s+(\\w+)\\s*(where ((.+(?= order by))|.+$))*\\s*(order by (.+))*");
+ std::regex_search(replaced_query, match, expr);
+ auto cols = minifi::utils::StringUtils::splitAndTrim(match[1], ",");
+ if (cols[0] == "*") {
+ cols = {};
+ }
+ std::string table_name = match[2];
+ std::string condition_str = match[4];
+ std::function<bool(const MockRow&)> condition;
+ if (!condition_str.empty()) {
+ if (condition_str == "int_col > 103") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) > 103; };
+ } else if (condition_str == "int_col > 102") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) > 102; };
+ } else if (condition_str == "int_col = 11") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) == 11; };
+ } else if (condition_str == "int_col = 11 and text_col = banana") {
Review comment:
Shouldn't "banana" be in apostrophes here? Unless it's another column.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r678893397
##########
File path: win_build_vs.bat
##########
@@ -65,7 +65,7 @@ for %%x in (%*) do (
mkdir %builddir%
pushd %builddir%\
-cmake -G %generator% -A %build_platform% -DINSTALLER_MERGE_MODULES=%installer_merge_modules% -DTEST_CUSTOM_WEL_PROVIDER=%test_custom_wel_provider% -DENABLE_SQL=%build_SQL% -DCMAKE_BUILD_TYPE_INIT=%cmake_build_type% -DCMAKE_BUILD_TYPE=%cmake_build_type% -DWIN32=WIN32 -DENABLE_LIBRDKAFKA=%build_kafka% -DENABLE_JNI=%build_jni% -DOPENSSL_OFF=OFF -DENABLE_COAP=%build_coap% -DENABLE_AWS=%build_AWS% -DENABLE_PDH=%build_PDH% -DENABLE_AZURE=%build_azure% -DENABLE_SFTP=%build_SFTP% -DUSE_SHARED_LIBS=OFF -DDISABLE_CONTROLLER=ON -DBUILD_ROCKSDB=ON -DFORCE_WINDOWS=ON -DUSE_SYSTEM_UUID=OFF -DDISABLE_LIBARCHIVE=OFF -DDISABLE_SCRIPTING=ON -DEXCLUDE_BOOST=ON -DENABLE_WEL=TRUE -DFAIL_ON_WARNINGS=OFF -DSKIP_TESTS=%skiptests% %strict_gsl_checks% %redist% -DENABLE_LINTER=%build_linter% .. && msbuild /m nifi-minifi-cpp.sln /property:Configuration=%cmake_build_type% /property:Platform=%build_platform% && copy main\%cmake_build_type%\minifi.exe main\
+cmake -G %generator% -A %build_platform% -DINSTALLER_MERGE_MODULES=%installer_merge_modules% -DTEST_CUSTOM_WEL_PROVIDER=%test_custom_wel_provider% -DENABLE_SQL=%build_SQL% -DUSE_REAL_ODBC_TEST_DRIVER=ON -DCMAKE_BUILD_TYPE_INIT=%cmake_build_type% -DCMAKE_BUILD_TYPE=%cmake_build_type% -DWIN32=WIN32 -DENABLE_LIBRDKAFKA=%build_kafka% -DENABLE_JNI=%build_jni% -DOPENSSL_OFF=OFF -DENABLE_COAP=%build_coap% -DENABLE_AWS=%build_AWS% -DENABLE_PDH=%build_PDH% -DENABLE_AZURE=%build_azure% -DENABLE_SFTP=%build_SFTP% -DUSE_SHARED_LIBS=OFF -DDISABLE_CONTROLLER=ON -DBUILD_ROCKSDB=ON -DFORCE_WINDOWS=ON -DUSE_SYSTEM_UUID=OFF -DDISABLE_LIBARCHIVE=OFF -DDISABLE_SCRIPTING=ON -DEXCLUDE_BOOST=ON -DENABLE_WEL=TRUE -DFAIL_ON_WARNINGS=OFF -DSKIP_TESTS=%skiptests% %strict_gsl_checks% %redist% -DENABLE_LINTER=%build_linter% .. && msbuild /m nifi-minifi-cpp.sln /property:Configuration=%cmake_build_type% /property:Platform=%build_platform% && copy main\%cmake_build_type%\minifi.exe main\
Review comment:
you seem to have added the `/RO` flag for the ci jobs, but the flag we are looking for in the bat file is `/ODBC`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r644915776
##########
File path: extensions/sql/data/DatabaseConnectors.h
##########
@@ -20,72 +20,68 @@
#include <memory>
#include <string>
-
-#include <soci/soci.h>
-
-#include "Utils.h"
+#include <vector>
+#include <ctime>
namespace org {
namespace apache {
namespace nifi {
namespace minifi {
namespace sql {
-/**
- * We do not intend to create an abstract facade here. We know that SOCI is the underlying
- * SQL library. We only wish to abstract ODBC specific information
- */
+enum class DataType {
+ STRING,
+ DOUBLE,
+ INTEGER,
+ LONG_LONG,
+ UNSIGNED_LONG_LONG,
+ DATE
+};
-class Statement {
+class Row {
public:
+ virtual ~Row() = default;
+ virtual std::size_t size() const = 0;
+ virtual std::string getColumnName(std::size_t index) const = 0;
+ virtual bool isNull(std::size_t index) const = 0;
+ virtual DataType getDataType(std::size_t index) const = 0;
+ virtual std::string getString(std::size_t index) const = 0;
+ virtual double getDouble(std::size_t index) const = 0;
+ virtual int getInteger(std::size_t index) const = 0;
+ virtual long long getLongLong(std::size_t index) const = 0;
+ virtual unsigned long long getUnsignedLongLong(std::size_t index) const = 0;
+ virtual std::tm getDate(std::size_t index) const = 0;
Review comment:
The specialization for `row::get` for the date type is `std::tm` in the implementing [SOCI library](http://soci.sourceforge.net/doc/master/types/) that's why I sticked with that type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r645454687
##########
File path: libminifi/include/utils/StringUtils.h
##########
@@ -137,6 +137,7 @@ class StringUtils {
static std::vector<std::string> split(const std::string &str, const std::string &delimiter);
static std::vector<std::string> splitAndTrim(const std::string &str, const std::string &delimiter);
+ static std::vector<std::string> splitAndTrimOnString(std::string str, const std::string& delimiter);
Review comment:
How is this different from splitAndTrim?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r648417253
##########
File path: libminifi/include/utils/StringUtils.h
##########
@@ -137,6 +137,7 @@ class StringUtils {
static std::vector<std::string> split(const std::string &str, const std::string &delimiter);
static std::vector<std::string> splitAndTrim(const std::string &str, const std::string &delimiter);
+ static std::vector<std::string> splitAndTrimOnString(std::string str, const std::string& delimiter);
Review comment:
Fixed the remaining issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r644598996
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.cpp
##########
@@ -0,0 +1,386 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "MockConnectors.h"
+
+#include <fstream>
+#include <algorithm>
+#include <utility>
+#include <string>
+#include <memory>
+
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+std::size_t MockRow::size() const {
+ return column_names_.size();
+}
+
+std::string MockRow::getColumnName(std::size_t index) const {
+ return column_names_.at(index);
+}
+
+bool MockRow::isNull(std::size_t index) const {
+ return column_values_.at(index) == "NULL";
+}
+
+DataType MockRow::getDataType(std::size_t index) const {
+ return column_types_.at(index);
+}
+
+std::string MockRow::getString(std::size_t index) const {
+ return column_values_.at(index);
+}
+
+double MockRow::getDouble(std::size_t index) const {
+ return std::atof(column_values_.at(index).c_str());
+}
+
+int MockRow::getInteger(std::size_t index) const {
+ return std::atoi(column_values_.at(index).c_str());
+}
+
+long long MockRow::getLongLong(std::size_t index) const { // NOLINT
+ return std::atoll(column_values_.at(index).c_str());
+}
+
+unsigned long long MockRow::getUnsignedLongLong(std::size_t index) const { // NOLINT
+ return static_cast<unsigned long long>(std::atoll(column_values_.at(index).c_str())); // NOLINT
+}
+
+std::tm MockRow::getDate(std::size_t /*index*/) const {
+ throw std::runtime_error("date not implemented");
+}
+
+std::vector<std::string> MockRow::getValues() const {
+ return column_values_;
+}
+
+std::string MockRow::getValue(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return column_values_.at(it-column_names_.begin());
+ }
+ throw std::runtime_error("Unknown column name for getting value");
+}
+
+void MockRowset::addRow(const std::vector<std::string>& column_values) {
+ rows_.emplace_back(column_names_, column_types_, column_values);
+}
+
+void MockRowset::reset() {
+ current_row_ = rows_.begin();
+}
+
+bool MockRowset::is_done() {
+ return current_row_ == rows_.end();
+}
+
+Row& MockRowset::getCurrent() {
+ return *current_row_;
+}
+
+void MockRowset::next() {
+ ++current_row_;
+}
+
+std::vector<std::string> MockRowset::getColumnNames() const {
+ return column_names_;
+}
+
+std::vector<DataType> MockRowset::getColumnTypes() const {
+ return column_types_;
+}
+
+std::vector<MockRow> MockRowset::getRows() const {
+ return rows_;
+}
+
+std::size_t MockRowset::getColumnIndex(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return it-column_names_.begin();
+ }
+ throw std::runtime_error("Unknown column name for getting index");
+}
+
+void MockRowset::sort(const std::string& order_by_col, bool order_ascending) {
+ std::sort(rows_.begin(), rows_.end(), [&](const MockRow& first, const MockRow& second) {
+ if (order_ascending) {
+ return first.getValue(order_by_col) < second.getValue(order_by_col);
+ } else {
+ return first.getValue(order_by_col) > second.getValue(order_by_col);
+ }
+ });
+}
+
+std::unique_ptr<MockRowset> MockRowset::select(const std::vector<std::string>& cols, const std::function<bool(const MockRow&)>& condition, const std::string& order_by_col, bool order_ascending) {
+ if (!order_by_col.empty()) {
+ sort(order_by_col, order_ascending);
+ }
+
+ std::unique_ptr<MockRowset> rowset;
+ if (cols.empty()) {
+ rowset = utils::make_unique<MockRowset>(column_names_, column_types_);
+ } else {
+ std::vector<DataType> col_types;
+ for (const auto& col : cols) {
+ col_types.push_back(column_types_.at(getColumnIndex(col)));
+ }
+ rowset = utils::make_unique<MockRowset>(cols, col_types);
+ }
+
+ std::vector<std::string> used_cols = cols.empty() ? column_names_ : cols;
+ for (const auto& row : rows_) {
+ if (condition(row)) {
+ std::vector<std::string> values;
+ for (const auto& col : used_cols) {
+ values.push_back(row.getValue(col));
+ }
+ rowset->addRow(values);
+ }
+ }
+
+ return rowset;
+}
+
+std::unique_ptr<Rowset> MockDB::execute(const std::string& query, const std::vector<std::string>& args) {
+ if (minifi::utils::StringUtils::startsWith(query, "create table")) {
+ createTable(query);
+ } else if (minifi::utils::StringUtils::startsWith(query, "insert into")) {
+ insertInto(query, args);
+ } else if (minifi::utils::StringUtils::startsWith(query, "select")) {
+ return select(query, args);
+ } else {
+ throw std::runtime_error("Unknown query type");
+ }
+
+ return nullptr;
+}
+
+void MockDB::createTable(const std::string& query) {
+ std::smatch match;
+ std::regex expr("create table (\\w+)\\s*\\((.*)\\);");
+ std::regex_search(query, match, expr);
+ std::string table_name = match[1];
+ auto columns_with_type = minifi::utils::StringUtils::splitAndTrim(match[2], ",");
+ std::vector<std::string> col_names;
+ std::vector<DataType> col_types;
+ for (const auto& col_with_type : columns_with_type) {
+ auto splitted = minifi::utils::StringUtils::splitAndTrim(col_with_type, " ");
+ col_names.push_back(splitted[0]);
+ col_types.push_back(stringToDataType(splitted[1]));
+ }
+ tables_.emplace(table_name, MockRowset{col_names, col_types});
+ storeDb();
+}
+
+void MockDB::insertInto(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("insert into (\\w+)\\s*(\\((.*)\\))*\\s*values\\s*\\((.+)\\)");
+ std::regex_search(replaced_query, match, expr);
+ std::string table_name = match[1];
+ std::vector<std::string> values = minifi::utils::StringUtils::splitAndTrim(match[4], ",");
+ for (auto& value : values) {
+ value = minifi::utils::StringUtils::removeFramingCharacters(value, '\'');
+ }
+ auto insert_col_names = minifi::utils::StringUtils::splitAndTrim(match[3], ",");
+ if (!insert_col_names.empty()) {
+ auto col_names = tables_.at(table_name).getColumnNames();
+ std::vector<std::string> row;
+ for (const auto& col_name : col_names) {
+ auto it = std::find(insert_col_names.begin(), insert_col_names.end(), col_name);
+ if (it != insert_col_names.end()) {
+ row.push_back(values.at(it-insert_col_names.begin()));
+ } else {
+ row.push_back("NULL");
+ }
+ }
+ tables_.at(table_name).addRow(row);
+ } else {
+ tables_.at(table_name).addRow(values);
+ }
+
+ storeDb();
+}
+
+std::unique_ptr<Rowset> MockDB::select(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("select\\s+(.+)\\s+from\\s+(\\w+)\\s*(where ((.+(?= order by))|.+$))*\\s*(order by (.+))*");
+ std::regex_search(replaced_query, match, expr);
+ auto cols = minifi::utils::StringUtils::splitAndTrim(match[1], ",");
+ if (cols[0] == "*") {
+ cols = {};
+ }
+ std::string table_name = match[2];
+ std::string condition_str = match[4];
+ std::function<bool(const MockRow&)> condition;
+ if (!condition_str.empty()) {
+ if (condition_str == "int_col > 103") {
Review comment:
maybe we could cover our test use-cases with some well-crafted regex-es (and maybe calling further regex-es on submatches if need be)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r644668234
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.cpp
##########
@@ -0,0 +1,386 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "MockConnectors.h"
+
+#include <fstream>
+#include <algorithm>
+#include <utility>
+#include <string>
+#include <memory>
+
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+std::size_t MockRow::size() const {
+ return column_names_.size();
+}
+
+std::string MockRow::getColumnName(std::size_t index) const {
+ return column_names_.at(index);
+}
+
+bool MockRow::isNull(std::size_t index) const {
+ return column_values_.at(index) == "NULL";
+}
+
+DataType MockRow::getDataType(std::size_t index) const {
+ return column_types_.at(index);
+}
+
+std::string MockRow::getString(std::size_t index) const {
+ return column_values_.at(index);
+}
+
+double MockRow::getDouble(std::size_t index) const {
+ return std::atof(column_values_.at(index).c_str());
+}
+
+int MockRow::getInteger(std::size_t index) const {
+ return std::atoi(column_values_.at(index).c_str());
+}
+
+long long MockRow::getLongLong(std::size_t index) const { // NOLINT
+ return std::atoll(column_values_.at(index).c_str());
+}
+
+unsigned long long MockRow::getUnsignedLongLong(std::size_t index) const { // NOLINT
+ return static_cast<unsigned long long>(std::atoll(column_values_.at(index).c_str())); // NOLINT
+}
+
+std::tm MockRow::getDate(std::size_t /*index*/) const {
+ throw std::runtime_error("date not implemented");
+}
+
+std::vector<std::string> MockRow::getValues() const {
+ return column_values_;
+}
+
+std::string MockRow::getValue(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return column_values_.at(it-column_names_.begin());
+ }
+ throw std::runtime_error("Unknown column name for getting value");
+}
+
+void MockRowset::addRow(const std::vector<std::string>& column_values) {
+ rows_.emplace_back(column_names_, column_types_, column_values);
+}
+
+void MockRowset::reset() {
+ current_row_ = rows_.begin();
+}
+
+bool MockRowset::is_done() {
+ return current_row_ == rows_.end();
+}
+
+Row& MockRowset::getCurrent() {
+ return *current_row_;
+}
+
+void MockRowset::next() {
+ ++current_row_;
+}
+
+std::vector<std::string> MockRowset::getColumnNames() const {
+ return column_names_;
+}
+
+std::vector<DataType> MockRowset::getColumnTypes() const {
+ return column_types_;
+}
+
+std::vector<MockRow> MockRowset::getRows() const {
+ return rows_;
+}
+
+std::size_t MockRowset::getColumnIndex(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return it-column_names_.begin();
+ }
+ throw std::runtime_error("Unknown column name for getting index");
+}
+
+void MockRowset::sort(const std::string& order_by_col, bool order_ascending) {
+ std::sort(rows_.begin(), rows_.end(), [&](const MockRow& first, const MockRow& second) {
+ if (order_ascending) {
+ return first.getValue(order_by_col) < second.getValue(order_by_col);
+ } else {
+ return first.getValue(order_by_col) > second.getValue(order_by_col);
+ }
+ });
+}
+
+std::unique_ptr<MockRowset> MockRowset::select(const std::vector<std::string>& cols, const std::function<bool(const MockRow&)>& condition, const std::string& order_by_col, bool order_ascending) {
+ if (!order_by_col.empty()) {
+ sort(order_by_col, order_ascending);
+ }
+
+ std::unique_ptr<MockRowset> rowset;
+ if (cols.empty()) {
+ rowset = utils::make_unique<MockRowset>(column_names_, column_types_);
+ } else {
+ std::vector<DataType> col_types;
+ for (const auto& col : cols) {
+ col_types.push_back(column_types_.at(getColumnIndex(col)));
+ }
+ rowset = utils::make_unique<MockRowset>(cols, col_types);
+ }
+
+ std::vector<std::string> used_cols = cols.empty() ? column_names_ : cols;
+ for (const auto& row : rows_) {
+ if (condition(row)) {
+ std::vector<std::string> values;
+ for (const auto& col : used_cols) {
+ values.push_back(row.getValue(col));
+ }
+ rowset->addRow(values);
+ }
+ }
+
+ return rowset;
+}
+
+std::unique_ptr<Rowset> MockDB::execute(const std::string& query, const std::vector<std::string>& args) {
+ if (minifi::utils::StringUtils::startsWith(query, "create table")) {
+ createTable(query);
+ } else if (minifi::utils::StringUtils::startsWith(query, "insert into")) {
+ insertInto(query, args);
+ } else if (minifi::utils::StringUtils::startsWith(query, "select")) {
+ return select(query, args);
+ } else {
+ throw std::runtime_error("Unknown query type");
+ }
+
+ return nullptr;
+}
+
+void MockDB::createTable(const std::string& query) {
+ std::smatch match;
+ std::regex expr("create table (\\w+)\\s*\\((.*)\\);");
+ std::regex_search(query, match, expr);
+ std::string table_name = match[1];
+ auto columns_with_type = minifi::utils::StringUtils::splitAndTrim(match[2], ",");
+ std::vector<std::string> col_names;
+ std::vector<DataType> col_types;
+ for (const auto& col_with_type : columns_with_type) {
+ auto splitted = minifi::utils::StringUtils::splitAndTrim(col_with_type, " ");
+ col_names.push_back(splitted[0]);
+ col_types.push_back(stringToDataType(splitted[1]));
+ }
+ tables_.emplace(table_name, MockRowset{col_names, col_types});
+ storeDb();
+}
+
+void MockDB::insertInto(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("insert into (\\w+)\\s*(\\((.*)\\))*\\s*values\\s*\\((.+)\\)");
+ std::regex_search(replaced_query, match, expr);
+ std::string table_name = match[1];
+ std::vector<std::string> values = minifi::utils::StringUtils::splitAndTrim(match[4], ",");
+ for (auto& value : values) {
+ value = minifi::utils::StringUtils::removeFramingCharacters(value, '\'');
+ }
+ auto insert_col_names = minifi::utils::StringUtils::splitAndTrim(match[3], ",");
+ if (!insert_col_names.empty()) {
+ auto col_names = tables_.at(table_name).getColumnNames();
+ std::vector<std::string> row;
+ for (const auto& col_name : col_names) {
+ auto it = std::find(insert_col_names.begin(), insert_col_names.end(), col_name);
+ if (it != insert_col_names.end()) {
+ row.push_back(values.at(it-insert_col_names.begin()));
+ } else {
+ row.push_back("NULL");
+ }
+ }
+ tables_.at(table_name).addRow(row);
+ } else {
+ tables_.at(table_name).addRow(values);
+ }
+
+ storeDb();
+}
+
+std::unique_ptr<Rowset> MockDB::select(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("select\\s+(.+)\\s+from\\s+(\\w+)\\s*(where ((.+(?= order by))|.+$))*\\s*(order by (.+))*");
+ std::regex_search(replaced_query, match, expr);
+ auto cols = minifi::utils::StringUtils::splitAndTrim(match[1], ",");
+ if (cols[0] == "*") {
+ cols = {};
+ }
+ std::string table_name = match[2];
+ std::string condition_str = match[4];
+ std::function<bool(const MockRow&)> condition;
+ if (!condition_str.empty()) {
+ if (condition_str == "int_col > 103") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) > 103; };
+ } else if (condition_str == "int_col > 102") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) > 102; };
+ } else if (condition_str == "int_col = 11") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) == 11; };
+ } else if (condition_str == "int_col = 11 and text_col = banana") {
+ condition = [&](const MockRow& row){ return std::atoi(row.getValue("int_col").c_str()) == 11 && row.getValue("text_col") == "banana"; };
+ } else {
+ throw std::runtime_error("Unimplemented WHERE condition");
+ }
+ } else {
+ condition = [](const MockRow&){ return true; };
+ }
+ std::string order = match[7];
+ std::string order_col;
+ bool descending = false;
+ if (!order.empty()) {
+ auto order_col_and_sort = minifi::utils::StringUtils::splitAndTrim(order, " ");
+ order_col = order_col_and_sort[0];
+ descending = order_col_and_sort[1] == "desc";
+ }
+ return tables_.at(table_name).select(cols, condition, order_col, !descending);
+}
+
+void MockDB::readDb() {
+ tables_.clear();
+ std::ifstream file(file_path_);
+ std::string line;
+ ParsePhase phase = ParsePhase::NEW_TABLE;
+ std::string table_name;
+ std::vector<std::string> column_names;
+ std::vector<DataType> column_types;
+ while (std::getline(file, line)) {
+ switch (phase) {
+ case ParsePhase::NEW_TABLE: {
+ table_name = minifi::utils::StringUtils::trim(line);
+ phase = ParsePhase::COLUMN_NAMES;
+ break;
+ }
+ case ParsePhase::COLUMN_NAMES: {
+ column_names = minifi::utils::StringUtils::splitAndTrim(line, "|");
+ phase = ParsePhase::COLUMN_TYPES;
+ break;
+ }
+ case ParsePhase::COLUMN_TYPES: {
+ column_types.clear();
+ auto type_strs = minifi::utils::StringUtils::splitAndTrim(line, "|");
+ for (const auto& type : type_strs) {
+ column_types.push_back(stringToDataType(type));
+ }
+ tables_.emplace(table_name, MockRowset(column_names, column_types));
+ phase = ParsePhase::ROW_VALUES;
+ break;
+ }
+ case ParsePhase::ROW_VALUES: {
+ line = minifi::utils::StringUtils::trim(line);
+ if (line.empty()) {
+ phase = ParsePhase::NEW_TABLE;
+ break;
+ }
+ auto cells = minifi::utils::StringUtils::splitAndTrim(line, "|");
+ tables_.at(table_name).addRow(cells);
+ break;
+ }
+ }
+ }
+}
+
+void MockDB::storeDb() {
+ std::ofstream file(file_path_);
+ for (const auto& table : tables_) {
+ file << table.first << std::endl;
+ auto& rowset = table.second;
+ for (const auto& col_name : rowset.getColumnNames()) {
+ file << col_name << "|";
+ }
+ file << std::endl;
+ for (const auto& col_type : rowset.getColumnTypes()) {
+ file << dataTypeToString(col_type) << "|";
+ }
+ file << std::endl;
+ for (const auto& row : rowset.getRows()) {
+ for (const auto& cell : row.getValues()) {
+ file << cell << "|";
+ }
+ file << std::endl;
+ }
+ file << std::endl;
+ }
+}
+
+DataType MockDB::stringToDataType(const std::string& type_str) {
+ if (type_str == "integer") return DataType::INTEGER;
+ if (type_str == "text") return DataType::STRING;
+ if (type_str == "real") return DataType::DOUBLE;
+ throw std::runtime_error("Unimplemented data type");
+}
+
+std::string MockDB::dataTypeToString(DataType data_type) {
+ switch (data_type) {
+ case DataType::INTEGER: return "integer";
+ case DataType::STRING: return "text";
+ case DataType::DOUBLE: return "real";
+ default: throw std::runtime_error("Unimplemented data type");
+ }
+}
+
+std::unique_ptr<Rowset> MockStatement::execute(const std::vector<std::string>& args) {
+ MockDB db(file_path_);
+ return db.execute(query_, args);
+}
+
+MockODBCConnection::MockODBCConnection(std::string connectionString)
+ : connection_string_(std::move(connectionString)) {
+ std::smatch match;
+ std::regex expr("Database=(.*)(;|$)");
Review comment:
My thinking was that as we are already on the way of dropping support for GCC 4.8 and we already have a ticket [MINIFICPP-1508](https://issues.apache.org/jira/browse/MINIFICPP-1508) for removing the `Regex` utility class it wouldn't be ideal to add more functionality to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r644587782
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.cpp
##########
@@ -0,0 +1,386 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "MockConnectors.h"
+
+#include <fstream>
+#include <algorithm>
+#include <utility>
+#include <string>
+#include <memory>
+
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+std::size_t MockRow::size() const {
+ return column_names_.size();
+}
+
+std::string MockRow::getColumnName(std::size_t index) const {
+ return column_names_.at(index);
+}
+
+bool MockRow::isNull(std::size_t index) const {
+ return column_values_.at(index) == "NULL";
+}
+
+DataType MockRow::getDataType(std::size_t index) const {
+ return column_types_.at(index);
+}
+
+std::string MockRow::getString(std::size_t index) const {
+ return column_values_.at(index);
+}
+
+double MockRow::getDouble(std::size_t index) const {
+ return std::atof(column_values_.at(index).c_str());
+}
+
+int MockRow::getInteger(std::size_t index) const {
+ return std::atoi(column_values_.at(index).c_str());
+}
+
+long long MockRow::getLongLong(std::size_t index) const { // NOLINT
+ return std::atoll(column_values_.at(index).c_str());
+}
+
+unsigned long long MockRow::getUnsignedLongLong(std::size_t index) const { // NOLINT
+ return static_cast<unsigned long long>(std::atoll(column_values_.at(index).c_str())); // NOLINT
+}
+
+std::tm MockRow::getDate(std::size_t /*index*/) const {
+ throw std::runtime_error("date not implemented");
+}
+
+std::vector<std::string> MockRow::getValues() const {
+ return column_values_;
+}
+
+std::string MockRow::getValue(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return column_values_.at(it-column_names_.begin());
+ }
+ throw std::runtime_error("Unknown column name for getting value");
+}
+
+void MockRowset::addRow(const std::vector<std::string>& column_values) {
+ rows_.emplace_back(column_names_, column_types_, column_values);
+}
+
+void MockRowset::reset() {
+ current_row_ = rows_.begin();
+}
+
+bool MockRowset::is_done() {
+ return current_row_ == rows_.end();
+}
+
+Row& MockRowset::getCurrent() {
+ return *current_row_;
+}
+
+void MockRowset::next() {
+ ++current_row_;
+}
+
+std::vector<std::string> MockRowset::getColumnNames() const {
+ return column_names_;
+}
+
+std::vector<DataType> MockRowset::getColumnTypes() const {
+ return column_types_;
+}
+
+std::vector<MockRow> MockRowset::getRows() const {
+ return rows_;
+}
+
+std::size_t MockRowset::getColumnIndex(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return it-column_names_.begin();
+ }
+ throw std::runtime_error("Unknown column name for getting index");
+}
+
+void MockRowset::sort(const std::string& order_by_col, bool order_ascending) {
+ std::sort(rows_.begin(), rows_.end(), [&](const MockRow& first, const MockRow& second) {
+ if (order_ascending) {
+ return first.getValue(order_by_col) < second.getValue(order_by_col);
+ } else {
+ return first.getValue(order_by_col) > second.getValue(order_by_col);
+ }
+ });
+}
+
+std::unique_ptr<MockRowset> MockRowset::select(const std::vector<std::string>& cols, const std::function<bool(const MockRow&)>& condition, const std::string& order_by_col, bool order_ascending) {
+ if (!order_by_col.empty()) {
+ sort(order_by_col, order_ascending);
+ }
+
+ std::unique_ptr<MockRowset> rowset;
+ if (cols.empty()) {
+ rowset = utils::make_unique<MockRowset>(column_names_, column_types_);
+ } else {
+ std::vector<DataType> col_types;
+ for (const auto& col : cols) {
+ col_types.push_back(column_types_.at(getColumnIndex(col)));
+ }
+ rowset = utils::make_unique<MockRowset>(cols, col_types);
+ }
+
+ std::vector<std::string> used_cols = cols.empty() ? column_names_ : cols;
+ for (const auto& row : rows_) {
+ if (condition(row)) {
+ std::vector<std::string> values;
+ for (const auto& col : used_cols) {
+ values.push_back(row.getValue(col));
+ }
+ rowset->addRow(values);
+ }
+ }
+
+ return rowset;
+}
+
+std::unique_ptr<Rowset> MockDB::execute(const std::string& query, const std::vector<std::string>& args) {
+ if (minifi::utils::StringUtils::startsWith(query, "create table")) {
+ createTable(query);
+ } else if (minifi::utils::StringUtils::startsWith(query, "insert into")) {
+ insertInto(query, args);
+ } else if (minifi::utils::StringUtils::startsWith(query, "select")) {
+ return select(query, args);
+ } else {
+ throw std::runtime_error("Unknown query type");
+ }
+
+ return nullptr;
+}
+
+void MockDB::createTable(const std::string& query) {
+ std::smatch match;
+ std::regex expr("create table (\\w+)\\s*\\((.*)\\);");
+ std::regex_search(query, match, expr);
+ std::string table_name = match[1];
+ auto columns_with_type = minifi::utils::StringUtils::splitAndTrim(match[2], ",");
+ std::vector<std::string> col_names;
+ std::vector<DataType> col_types;
+ for (const auto& col_with_type : columns_with_type) {
+ auto splitted = minifi::utils::StringUtils::splitAndTrim(col_with_type, " ");
+ col_names.push_back(splitted[0]);
+ col_types.push_back(stringToDataType(splitted[1]));
+ }
+ tables_.emplace(table_name, MockRowset{col_names, col_types});
+ storeDb();
+}
+
+void MockDB::insertInto(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("insert into (\\w+)\\s*(\\((.*)\\))*\\s*values\\s*\\((.+)\\)");
+ std::regex_search(replaced_query, match, expr);
+ std::string table_name = match[1];
+ std::vector<std::string> values = minifi::utils::StringUtils::splitAndTrim(match[4], ",");
+ for (auto& value : values) {
+ value = minifi::utils::StringUtils::removeFramingCharacters(value, '\'');
+ }
+ auto insert_col_names = minifi::utils::StringUtils::splitAndTrim(match[3], ",");
+ if (!insert_col_names.empty()) {
+ auto col_names = tables_.at(table_name).getColumnNames();
+ std::vector<std::string> row;
+ for (const auto& col_name : col_names) {
+ auto it = std::find(insert_col_names.begin(), insert_col_names.end(), col_name);
+ if (it != insert_col_names.end()) {
+ row.push_back(values.at(it-insert_col_names.begin()));
+ } else {
+ row.push_back("NULL");
+ }
+ }
+ tables_.at(table_name).addRow(row);
+ } else {
+ tables_.at(table_name).addRow(values);
+ }
+
+ storeDb();
+}
+
+std::unique_ptr<Rowset> MockDB::select(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("select\\s+(.+)\\s+from\\s+(\\w+)\\s*(where ((.+(?= order by))|.+$))*\\s*(order by (.+))*");
+ std::regex_search(replaced_query, match, expr);
+ auto cols = minifi::utils::StringUtils::splitAndTrim(match[1], ",");
+ if (cols[0] == "*") {
+ cols = {};
+ }
+ std::string table_name = match[2];
+ std::string condition_str = match[4];
+ std::function<bool(const MockRow&)> condition;
+ if (!condition_str.empty()) {
+ if (condition_str == "int_col > 103") {
Review comment:
Two problems with reusing Expression.cpp:
1. It's not available when compiling with expression language support disabled
2. It's parsing nifi expression language, but this looks like SQL to me
Not sure what would be the best way to solve this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1073:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1073#discussion_r644675201
##########
File path: libminifi/test/sql-tests/mocks/MockConnectors.cpp
##########
@@ -0,0 +1,386 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "MockConnectors.h"
+
+#include <fstream>
+#include <algorithm>
+#include <utility>
+#include <string>
+#include <memory>
+
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+std::size_t MockRow::size() const {
+ return column_names_.size();
+}
+
+std::string MockRow::getColumnName(std::size_t index) const {
+ return column_names_.at(index);
+}
+
+bool MockRow::isNull(std::size_t index) const {
+ return column_values_.at(index) == "NULL";
+}
+
+DataType MockRow::getDataType(std::size_t index) const {
+ return column_types_.at(index);
+}
+
+std::string MockRow::getString(std::size_t index) const {
+ return column_values_.at(index);
+}
+
+double MockRow::getDouble(std::size_t index) const {
+ return std::atof(column_values_.at(index).c_str());
+}
+
+int MockRow::getInteger(std::size_t index) const {
+ return std::atoi(column_values_.at(index).c_str());
+}
+
+long long MockRow::getLongLong(std::size_t index) const { // NOLINT
+ return std::atoll(column_values_.at(index).c_str());
+}
+
+unsigned long long MockRow::getUnsignedLongLong(std::size_t index) const { // NOLINT
+ return static_cast<unsigned long long>(std::atoll(column_values_.at(index).c_str())); // NOLINT
+}
+
+std::tm MockRow::getDate(std::size_t /*index*/) const {
+ throw std::runtime_error("date not implemented");
+}
+
+std::vector<std::string> MockRow::getValues() const {
+ return column_values_;
+}
+
+std::string MockRow::getValue(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return column_values_.at(it-column_names_.begin());
+ }
+ throw std::runtime_error("Unknown column name for getting value");
+}
+
+void MockRowset::addRow(const std::vector<std::string>& column_values) {
+ rows_.emplace_back(column_names_, column_types_, column_values);
+}
+
+void MockRowset::reset() {
+ current_row_ = rows_.begin();
+}
+
+bool MockRowset::is_done() {
+ return current_row_ == rows_.end();
+}
+
+Row& MockRowset::getCurrent() {
+ return *current_row_;
+}
+
+void MockRowset::next() {
+ ++current_row_;
+}
+
+std::vector<std::string> MockRowset::getColumnNames() const {
+ return column_names_;
+}
+
+std::vector<DataType> MockRowset::getColumnTypes() const {
+ return column_types_;
+}
+
+std::vector<MockRow> MockRowset::getRows() const {
+ return rows_;
+}
+
+std::size_t MockRowset::getColumnIndex(const std::string& col_name) const {
+ auto it = std::find(column_names_.begin(), column_names_.end(), col_name);
+ if (it != column_names_.end()) {
+ return it-column_names_.begin();
+ }
+ throw std::runtime_error("Unknown column name for getting index");
+}
+
+void MockRowset::sort(const std::string& order_by_col, bool order_ascending) {
+ std::sort(rows_.begin(), rows_.end(), [&](const MockRow& first, const MockRow& second) {
+ if (order_ascending) {
+ return first.getValue(order_by_col) < second.getValue(order_by_col);
+ } else {
+ return first.getValue(order_by_col) > second.getValue(order_by_col);
+ }
+ });
+}
+
+std::unique_ptr<MockRowset> MockRowset::select(const std::vector<std::string>& cols, const std::function<bool(const MockRow&)>& condition, const std::string& order_by_col, bool order_ascending) {
+ if (!order_by_col.empty()) {
+ sort(order_by_col, order_ascending);
+ }
+
+ std::unique_ptr<MockRowset> rowset;
+ if (cols.empty()) {
+ rowset = utils::make_unique<MockRowset>(column_names_, column_types_);
+ } else {
+ std::vector<DataType> col_types;
+ for (const auto& col : cols) {
+ col_types.push_back(column_types_.at(getColumnIndex(col)));
+ }
+ rowset = utils::make_unique<MockRowset>(cols, col_types);
+ }
+
+ std::vector<std::string> used_cols = cols.empty() ? column_names_ : cols;
+ for (const auto& row : rows_) {
+ if (condition(row)) {
+ std::vector<std::string> values;
+ for (const auto& col : used_cols) {
+ values.push_back(row.getValue(col));
+ }
+ rowset->addRow(values);
+ }
+ }
+
+ return rowset;
+}
+
+std::unique_ptr<Rowset> MockDB::execute(const std::string& query, const std::vector<std::string>& args) {
+ if (minifi::utils::StringUtils::startsWith(query, "create table")) {
+ createTable(query);
+ } else if (minifi::utils::StringUtils::startsWith(query, "insert into")) {
+ insertInto(query, args);
+ } else if (minifi::utils::StringUtils::startsWith(query, "select")) {
+ return select(query, args);
+ } else {
+ throw std::runtime_error("Unknown query type");
+ }
+
+ return nullptr;
+}
+
+void MockDB::createTable(const std::string& query) {
+ std::smatch match;
+ std::regex expr("create table (\\w+)\\s*\\((.*)\\);");
+ std::regex_search(query, match, expr);
+ std::string table_name = match[1];
+ auto columns_with_type = minifi::utils::StringUtils::splitAndTrim(match[2], ",");
+ std::vector<std::string> col_names;
+ std::vector<DataType> col_types;
+ for (const auto& col_with_type : columns_with_type) {
+ auto splitted = minifi::utils::StringUtils::splitAndTrim(col_with_type, " ");
+ col_names.push_back(splitted[0]);
+ col_types.push_back(stringToDataType(splitted[1]));
+ }
+ tables_.emplace(table_name, MockRowset{col_names, col_types});
+ storeDb();
+}
+
+void MockDB::insertInto(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("insert into (\\w+)\\s*(\\((.*)\\))*\\s*values\\s*\\((.+)\\)");
+ std::regex_search(replaced_query, match, expr);
+ std::string table_name = match[1];
+ std::vector<std::string> values = minifi::utils::StringUtils::splitAndTrim(match[4], ",");
+ for (auto& value : values) {
+ value = minifi::utils::StringUtils::removeFramingCharacters(value, '\'');
+ }
+ auto insert_col_names = minifi::utils::StringUtils::splitAndTrim(match[3], ",");
+ if (!insert_col_names.empty()) {
+ auto col_names = tables_.at(table_name).getColumnNames();
+ std::vector<std::string> row;
+ for (const auto& col_name : col_names) {
+ auto it = std::find(insert_col_names.begin(), insert_col_names.end(), col_name);
+ if (it != insert_col_names.end()) {
+ row.push_back(values.at(it-insert_col_names.begin()));
+ } else {
+ row.push_back("NULL");
+ }
+ }
+ tables_.at(table_name).addRow(row);
+ } else {
+ tables_.at(table_name).addRow(values);
+ }
+
+ storeDb();
+}
+
+std::unique_ptr<Rowset> MockDB::select(const std::string& query, const std::vector<std::string>& args) {
+ std::string replaced_query = query;
+ for (const auto& arg : args) {
+ replaced_query = minifi::utils::StringUtils::replaceOne(replaced_query, "?", arg);
+ }
+
+ std::smatch match;
+ std::regex expr("select\\s+(.+)\\s+from\\s+(\\w+)\\s*(where ((.+(?= order by))|.+$))*\\s*(order by (.+))*");
+ std::regex_search(replaced_query, match, expr);
+ auto cols = minifi::utils::StringUtils::splitAndTrim(match[1], ",");
+ if (cols[0] == "*") {
+ cols = {};
+ }
+ std::string table_name = match[2];
+ std::string condition_str = match[4];
+ std::function<bool(const MockRow&)> condition;
+ if (!condition_str.empty()) {
+ if (condition_str == "int_col > 103") {
Review comment:
I'll investigate some simple parsing solutions for handling our current cases. I think we need to find a good balance between maintainability and expandability. I used a hard coded condition as a start because we currently do not have too many conditions and it can be expanded easily, but if we add way more then it will become bloated. On the other side if we write a complex parser the maintainability would be way too much effort. I'll try to write a simple parser that would be sufficient for our cases without too much maintenance costs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org