You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@quickstep.apache.org by zu...@apache.org on 2017/01/30 00:55:23 UTC
[1/2] incubator-quickstep git commit: Documentation update after
third party related changes. [Forced Update!]
Repository: incubator-quickstep
Updated Branches:
refs/heads/exact-stat-unittest 231a691ee -> 484ef8204 (forced update)
Documentation update after third party related changes.
- Updated master README file.
- Updated BUILDING instruction file.
- Updated description of the third party directory.
Project: http://git-wip-us.apache.org/repos/asf/incubator-quickstep/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-quickstep/commit/0780b848
Tree: http://git-wip-us.apache.org/repos/asf/incubator-quickstep/tree/0780b848
Diff: http://git-wip-us.apache.org/repos/asf/incubator-quickstep/diff/0780b848
Branch: refs/heads/exact-stat-unittest
Commit: 0780b848462a14de87377b6de2a157a23fd3a805
Parents: 3210500
Author: Harshad Deshmukh <hb...@apache.org>
Authored: Sat Jan 28 23:04:21 2017 -0600
Committer: Harshad Deshmukh <hb...@apache.org>
Committed: Sun Jan 29 08:52:58 2017 -0600
----------------------------------------------------------------------
BUILDING.md | 1 +
README.md | 12 +++++++-----
third_party/README.md | 12 +++++++++---
3 files changed, 17 insertions(+), 8 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/0780b848/BUILDING.md
----------------------------------------------------------------------
diff --git a/BUILDING.md b/BUILDING.md
index 97552c6..02a3a58 100644
--- a/BUILDING.md
+++ b/BUILDING.md
@@ -77,6 +77,7 @@ this by running the following 2 commands in the root quickstep directory:
git submodule init
git submodule update
+ cd third_party && ./download_and_patch_prerequisites.sh
### Advanced Configuration
http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/0780b848/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index af17264..18b0301 100644
--- a/README.md
+++ b/README.md
@@ -38,18 +38,20 @@ And, it is **open source!**
2. Then, go to the code directory: ```cd quickstep```
3. Initialize the dependencies: ```git submodule init```
4. Checkout the dependencies: ```git submodule update```
-5. Go into the build directory: ```cd build```
-6. Create the Makefile: ```cmake -D CMAKE_BUILD_TYPE=Release ..```
-7. Build: ```make -j4```. Note you may replace the 4 with the number of cores
+5. Download additional third-party dependencies and apply patches:<br/>
+```cd third_party && ./download_and_patch_prerequisites.sh && cd ../```
+6. Go into the build directory: ```cd build```
+7. Create the Makefile: ```cmake -D CMAKE_BUILD_TYPE=Release ..```
+8. Build: ```make -j4```. Note you may replace the 4 with the number of cores
on your machine.
-8. Start quickstep: ```./quickstep_cli_shell --initialize_db=true```. You can
+9. Start quickstep: ```./quickstep_cli_shell --initialize_db=true```. You can
now fire SQL queries. To quit, you can type in ```quit;``` Your data is
stored in the directory ```qsstor```. Note the next time you start Quickstep,
you can omit the ``` --initialize_db``` flag (as the database has already
been initialized), and simply start Quickstep as: ```./quickstep_cli_shell```.
There are also a number of optional flags that you can specify, and to see
the full list, you can type in: ```./quickstep_cli_shell --help```
-9. Next let us load some data and fire some queries. A few points to note:
+10. Next let us load some data and fire some queries. A few points to note:
The SQL surface of Quickstep is small (it will grow over time). The
traditional SQL CREATE TABLE and SELECT statements work. The data types
that are supported include INTEGER, FLOAT, DOUBLE, VARCHAR, CHAR, DATE,
http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/0780b848/third_party/README.md
----------------------------------------------------------------------
diff --git a/third_party/README.md b/third_party/README.md
index cfb1a51..e434398 100644
--- a/third_party/README.md
+++ b/third_party/README.md
@@ -1,7 +1,13 @@
# Third-Party Libraries
This directory includes various open-source third-party code that is used by
-Quickstep. Some code has been modified slightly to fix build issues or to integrate
-with Quickstep. With the exception of the code in the `tmb` and the `protobuf_cmake`
-directories (which are part of the Quickstep project itself), all libraries here
+Quickstep. Here's the description of the files:
+
+`download_and_patch_prerequisites.sh` - Downloads the third party library source codes from their respective repositories and applies Quickstep specific patches.<br/>
+`reset_third_party_dir.sh` - Removes the downloaded and patched third party and resets the `third_party` directory.<br/>
+`patches/` - Contains the patch files applied on the original third party source code files.<br/>
+`src/` - Contains the patched third party source code.<br/>
+
+With the exception of the code in the `tmb`, `iwyu`, and the `protobuf_cmake`
+directories (which are part of the Quickstep project itself), all other libraries
belong to their original authors and are governed by their respective licenses.
[2/2] incubator-quickstep git commit: Add unit test for
CatalogRelationStatistics
Posted by zu...@apache.org.
Add unit test for CatalogRelationStatistics
Project: http://git-wip-us.apache.org/repos/asf/incubator-quickstep/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-quickstep/commit/484ef820
Tree: http://git-wip-us.apache.org/repos/asf/incubator-quickstep/tree/484ef820
Diff: http://git-wip-us.apache.org/repos/asf/incubator-quickstep/diff/484ef820
Branch: refs/heads/exact-stat-unittest
Commit: 484ef820413b4fa44c84c58f634ade293cc3f239
Parents: 0780b84
Author: Jianqiao Zhu <ji...@cs.wisc.edu>
Authored: Mon Jan 23 20:54:51 2017 -0600
Committer: Zuyu Zhang <zu...@apache.org>
Committed: Sun Jan 29 16:55:15 2017 -0800
----------------------------------------------------------------------
catalog/CMakeLists.txt | 27 +++
.../CatalogRelationStatistics_unittest.cpp | 214 +++++++++++++++++++
2 files changed, 241 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/484ef820/catalog/CMakeLists.txt
----------------------------------------------------------------------
diff --git a/catalog/CMakeLists.txt b/catalog/CMakeLists.txt
index 7de9a67..3c64e97 100644
--- a/catalog/CMakeLists.txt
+++ b/catalog/CMakeLists.txt
@@ -225,6 +225,31 @@ target_link_libraries(Catalog_unittest
quickstep_utility_PtrVector)
add_test(Catalog_unittest Catalog_unittest)
+add_executable(CatalogRelationStatistics_unittest
+ "${CMAKE_CURRENT_SOURCE_DIR}/tests/CatalogRelationStatistics_unittest.cpp")
+target_link_libraries(CatalogRelationStatistics_unittest
+ gtest
+ gtest_main
+ quickstep_catalog_Catalog
+ quickstep_catalog_CatalogDatabase
+ quickstep_catalog_CatalogRelation
+ quickstep_catalog_CatalogRelationStatistics
+ quickstep_cli_CommandExecutor
+ quickstep_cli_DropRelation
+ quickstep_parser_ParseStatement
+ quickstep_parser_SqlParserWrapper
+ quickstep_queryexecution_ForemanSingleNode
+ quickstep_queryexecution_QueryExecutionTypedefs
+ quickstep_queryexecution_QueryExecutionUtil
+ quickstep_queryexecution_Worker
+ quickstep_queryexecution_WorkerDirectory
+ quickstep_queryoptimizer_QueryHandle
+ quickstep_queryoptimizer_QueryProcessor
+ quickstep_storage_StorageConstants
+ quickstep_storage_StorageManager
+ tmb)
+add_test(CatalogRelationStatistics_unittest CatalogRelationStatistics_unittest)
+
if(QUICKSTEP_HAVE_LIBNUMA)
add_executable(NUMAPlacementScheme_unittest
"${CMAKE_CURRENT_SOURCE_DIR}/tests/NUMAPlacementScheme_unittest.cpp")
@@ -253,3 +278,5 @@ target_link_libraries(PartitionScheme_unittest
quickstep_types_operations_comparisons_Comparison
quickstep_types_operations_comparisons_EqualComparison)
add_test(PartitionScheme_unittest PartitionScheme_unittest)
+
+file(MAKE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/catalog_relation_statistics_test_data)
http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/484ef820/catalog/tests/CatalogRelationStatistics_unittest.cpp
----------------------------------------------------------------------
diff --git a/catalog/tests/CatalogRelationStatistics_unittest.cpp b/catalog/tests/CatalogRelationStatistics_unittest.cpp
new file mode 100644
index 0000000..9242d7f
--- /dev/null
+++ b/catalog/tests/CatalogRelationStatistics_unittest.cpp
@@ -0,0 +1,214 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ **/
+
+#include <cstdio>
+#include <fstream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "catalog/Catalog.hpp"
+#include "catalog/CatalogDatabase.hpp"
+#include "catalog/CatalogRelation.hpp"
+#include "catalog/CatalogRelationStatistics.hpp"
+#include "cli/CommandExecutor.hpp"
+#include "cli/DropRelation.hpp"
+#include "parser/ParseStatement.hpp"
+#include "parser/SqlParserWrapper.hpp"
+#include "query_execution/ForemanSingleNode.hpp"
+#include "query_execution/QueryExecutionTypedefs.hpp"
+#include "query_execution/QueryExecutionUtil.hpp"
+#include "query_execution/Worker.hpp"
+#include "query_execution/WorkerDirectory.hpp"
+#include "query_optimizer/QueryHandle.hpp"
+#include "query_optimizer/QueryProcessor.hpp"
+#include "storage/StorageConstants.hpp"
+#include "storage/StorageManager.hpp"
+
+#include "glog/logging.h"
+#include "gtest/gtest.h"
+
+#include "tmb/id_typedefs.h"
+
+namespace quickstep {
+
+namespace {
+const std::string kStoragePath = "./catalog_relation_statistics_test_data/";
+}
+
+class CatalogRelationStatisticsTest : public ::testing::Test {
+ protected:
+ virtual void SetUp() {
+ // Set up the environment for running end-to-end queries.
+ quickstep::ClientIDMap::Instance();
+
+ bus_.Initialize();
+
+ main_thread_client_id_ = bus_.Connect();
+ bus_.RegisterClientAsSender(main_thread_client_id_, kAdmitRequestMessage);
+ bus_.RegisterClientAsSender(main_thread_client_id_, kPoisonMessage);
+ bus_.RegisterClientAsReceiver(main_thread_client_id_, kWorkloadCompletionMessage);
+
+ std::string catalog_path(kStoragePath);
+ catalog_path.append(kCatalogFilename);
+
+ std::ofstream catalog_file(catalog_path.c_str());
+ Catalog catalog;
+ catalog.addDatabase(new CatalogDatabase(nullptr, "default"));
+ catalog.getProto().SerializeToOstream(&catalog_file);
+ catalog_file.close();
+
+ storage_manager_.reset(new StorageManager(kStoragePath));
+ query_processor_.reset(new QueryProcessor(std::move(catalog_path)));
+
+ worker_.reset(new Worker(0, &bus_));
+ worker_directory_.reset(
+ new WorkerDirectory(1, {worker_->getBusClientID()}, {-1}));
+
+ foreman_.reset(
+ new ForemanSingleNode(main_thread_client_id_,
+ worker_directory_.get(),
+ &bus_,
+ query_processor_->getDefaultDatabase(),
+ storage_manager_.get()));
+
+ worker_->start();
+ foreman_->start();
+ }
+
+ virtual void TearDown() {
+ for (const auto &relation : *query_processor_->getDefaultDatabase()) {
+ DropRelation::Drop(relation,
+ query_processor_->getDefaultDatabase(),
+ storage_manager_.get());
+ }
+
+ QueryExecutionUtil::BroadcastPoisonMessage(main_thread_client_id_, &bus_);
+ worker_->join();
+ foreman_->join();
+ }
+
+ void executeQuery(const std::string &query_string) {
+ SqlParserWrapper parser_wrapper;
+ parser_wrapper.feedNextBuffer(new std::string(query_string));
+
+ ParseResult result = parser_wrapper.getNextStatement();
+ DCHECK(result.condition == ParseResult::kSuccess);
+
+ const ParseStatement &statement = *result.parsed_statement;
+ std::unique_ptr<QueryHandle> query_handle =
+ std::make_unique<QueryHandle>(query_processor_->query_id(),
+ main_thread_client_id_,
+ statement.getPriority());
+ query_processor_->generateQueryHandle(statement, query_handle.get());
+
+ QueryExecutionUtil::ConstructAndSendAdmitRequestMessage(
+ main_thread_client_id_,
+ foreman_->getBusClientID(),
+ query_handle.release(),
+ &bus_);
+
+ QueryExecutionUtil::ReceiveQueryCompletionMessage(main_thread_client_id_, &bus_);
+ }
+
+ void executeAnalyze(const std::string &rel_name) {
+ SqlParserWrapper parser_wrapper;
+ parser_wrapper.feedNextBuffer(new std::string("\\analyze " + rel_name));
+
+ ParseResult result = parser_wrapper.getNextStatement();
+ DCHECK(result.condition == ParseResult::kSuccess);
+
+ const ParseStatement &statement = *result.parsed_statement;
+ DCHECK(statement.getStatementType() == ParseStatement::kCommand);
+ quickstep::cli::executeCommand(statement,
+ *(query_processor_->getDefaultDatabase()),
+ main_thread_client_id_,
+ foreman_->getBusClientID(),
+ &bus_,
+ storage_manager_.get(),
+ query_processor_.get(),
+ stdout);
+ }
+
+ const CatalogRelation *getRelationByName(const std::string &rel_name) const {
+ const CatalogRelation *relation =
+ query_processor_->getDefaultDatabase()->getRelationByName(rel_name);
+ DCHECK(relation != nullptr);
+ return relation;
+ }
+
+ private:
+ MessageBusImpl bus_;
+ tmb::client_id main_thread_client_id_;
+
+ std::unique_ptr<StorageManager> storage_manager_;
+ std::unique_ptr<QueryProcessor> query_processor_;
+
+ std::unique_ptr<Worker> worker_;
+ std::unique_ptr<WorkerDirectory> worker_directory_;
+ std::unique_ptr<ForemanSingleNode> foreman_;
+};
+
+TEST_F(CatalogRelationStatisticsTest, AnalyzeTest) {
+ executeQuery("CREATE TABLE analyzetest(x INT, y DOUBLE);");
+ executeQuery("INSERT INTO analyzetest VALUES(0, -0.5);");
+ executeQuery("INSERT INTO analyzetest VALUES(1, 0);");
+ executeQuery("INSERT INTO analyzetest VALUES(0, 0.5);");
+ executeAnalyze("analyzetest");
+
+ const CatalogRelation *relation = getRelationByName("analyzetest");
+ const CatalogRelationStatistics &stat = relation->getStatistics();
+
+ EXPECT_EQ(stat.getNumTuples(), 3u);
+
+ EXPECT_EQ(stat.getNumDistinctValues(0), 2u);
+ EXPECT_EQ(stat.getMinValue(0).getLiteral<int>(), 0);
+ EXPECT_EQ(stat.getMaxValue(0).getLiteral<int>(), 1);
+
+ EXPECT_EQ(stat.getNumDistinctValues(1), 3u);
+ EXPECT_EQ(stat.getMinValue(1).getLiteral<double>(), -0.5);
+ EXPECT_EQ(stat.getMaxValue(1).getLiteral<double>(), 0.5);
+}
+
+TEST_F(CatalogRelationStatisticsTest, ExactnessTest) {
+ executeQuery("CREATE TABLE exactnesstest(x INT);");
+
+ const CatalogRelationStatistics &stat =
+ getRelationByName("exactnesstest")->getStatistics();
+
+ EXPECT_FALSE(stat.isExact());
+
+ const std::vector<std::string> queries = {
+ "INSERT INTO exactnesstest VALUES(1);",
+ "INSERT INTO exactnesstest SELECT i FROM generate_series(2, 10) AS gs(i);",
+ "DELETE FROM exactnesstest WHERE x = 5;",
+ "UPDATE exactnesstest SET x = 100 WHERE x = 10;"
+ };
+
+ for (const std::string &query : queries) {
+ executeQuery(query);
+ EXPECT_FALSE(stat.isExact());
+
+ executeAnalyze("exactnesstest");
+ EXPECT_TRUE(stat.isExact());
+ }
+}
+
+} // namespace quickstep