You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by li...@apache.org on 2022/06/01 20:11:09 UTC
[arrow-adbc] branch main updated: Initial import (#1)
This is an automated email from the ASF dual-hosted git repository.
lidavidm pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-adbc.git
The following commit(s) were added to refs/heads/main by this push:
new 64b0127 Initial import (#1)
64b0127 is described below
commit 64b01275a6d8d80fef9ecc26b428443180aeb8df
Author: David Li <li...@gmail.com>
AuthorDate: Wed Jun 1 16:11:04 2022 -0400
Initial import (#1)
* Update readme and add license in root.
* ARROW-202: Integrate with appveyor ci for windows
This only adds yet a successful compilation for windows. Tests don't
run.
Author: Uwe L. Korn <uw...@xhochy.com>
Closes #213 from xhochy/ARROW-202 and squashes the following commits:
d5088a6 [Uwe L. Korn] Correctly reference Kudu in LICENSE and NOTICE
72a583b [Uwe L. Korn] Differentiate Boost libraries based on build type
6c75699 [Uwe L. Korn] Add license header
e33b08c [Uwe L. Korn] Pick up shared Boost libraries correctly
5da5f5d [Uwe L. Korn] ARROW-202: Integrate with appveyor ci for windows
* ARROW-557: [Python] Add option to explicitly opt in to HDFS tests, do not implicitly skip
I have
```
$ py.test pyarrow/tests/test_hdfs.py
================================== test session starts ==================================
platform linux2 -- Python 2.7.11, pytest-2.9.0, py-1.4.31, pluggy-0.3.1
rootdir: /home/wesm/code/arrow/python, inifile:
collected 15 items
pyarrow/tests/test_hdfs.py sssssssssssssss
```
But
```
$ py.test pyarrow/tests/test_hdfs.py --hdfs -v
================================== test session starts ==================================
platform linux2 -- Python 2.7.11, pytest-2.9.0, py-1.4.31, pluggy-0.3.1 -- /home/wesm/anaconda3/envs/py27/bin/python
cachedir: .cache
rootdir: /home/wesm/code/arrow/python, inifile:
collected 15 items
pyarrow/tests/test_hdfs.py::TestLibHdfs::test_hdfs_close PASSED
pyarrow/tests/test_hdfs.py::TestLibHdfs::test_hdfs_download_upload PASSED
pyarrow/tests/test_hdfs.py::TestLibHdfs::test_hdfs_file_context_manager PASSED
pyarrow/tests/test_hdfs.py::TestLibHdfs::test_hdfs_ls PASSED
pyarrow/tests/test_hdfs.py::TestLibHdfs::test_hdfs_mkdir PASSED
pyarrow/tests/test_hdfs.py::TestLibHdfs::test_hdfs_orphaned_file PASSED
pyarrow/tests/test_hdfs.py::TestLibHdfs::test_hdfs_read_multiple_parquet_files SKIPPED
pyarrow/tests/test_hdfs.py::TestLibHdfs::test_hdfs_read_whole_file PASSED
pyarrow/tests/test_hdfs.py::TestLibHdfs3::test_hdfs_close PASSED
pyarrow/tests/test_hdfs.py::TestLibHdfs3::test_hdfs_download_upload PASSED
pyarrow/tests/test_hdfs.py::TestLibHdfs3::test_hdfs_file_context_manager PASSED
pyarrow/tests/test_hdfs.py::TestLibHdfs3::test_hdfs_ls PASSED
pyarrow/tests/test_hdfs.py::TestLibHdfs3::test_hdfs_mkdir PASSED
pyarrow/tests/test_hdfs.py::TestLibHdfs3::test_hdfs_read_multiple_parquet_files SKIPPED
pyarrow/tests/test_hdfs.py::TestLibHdfs3::test_hdfs_read_whole_file PASSED
```
The `py.test pyarrow --only-hdfs` option will run only the HDFS tests.
Author: Wes McKinney <we...@twosigma.com>
Closes #353 from wesm/ARROW-557 and squashes the following commits:
52e03db [Wes McKinney] Add conftest.py file, hdfs group to opt in to HDFS tests with --hdfs
* ARROW-1104: Integrate in-memory object store into arrow
This supersedes https://github.com/apache/arrow/pull/467
This is ready for review. Next steps are
- Integration with the arrow CI
- Write docs on how to use the object store
There is one remaining compilation error (it doesn't find Python.h for one of the Travis configurations, if anybody has an idea on what is going on, let me know).
Author: Philipp Moritz <pc...@gmail.com>
Author: Robert Nishihara <ro...@gmail.com>
Closes #742 from pcmoritz/plasma-store-2 and squashes the following commits:
c100a453 [Philipp Moritz] fixes
d67160c5 [Philipp Moritz] build dlmalloc with -O3
16d1f716 [Philipp Moritz] fix test hanging
0f321e16 [Philipp Moritz] try to fix tests
80f9df40 [Philipp Moritz] make format
4c474d71 [Philipp Moritz] run plasma_store from the right directory
85aa1710 [Philipp Moritz] fix mac tests
61d421b5 [Philipp Moritz] fix formatting
4497e337 [Philipp Moritz] fix tests
00f17f24 [Philipp Moritz] fix licenses
81437920 [Philipp Moritz] fix linting
5370ae06 [Philipp Moritz] fix plasma protocol
a137e783 [Philipp Moritz] more fixes
b36c6aaa [Philipp Moritz] fix fling.cc
214c426c [Philipp Moritz] fix eviction policy
e7badc48 [Philipp Moritz] fix python extension
6432d3fa [Philipp Moritz] fix formatting
b21f0814 [Philipp Moritz] fix remaining comments about client
27f9c9e8 [Philipp Moritz] fix formatting
7b08fd2a [Philipp Moritz] replace ObjectID pass by value with pass by const reference and fix const correctness
ca80e9a6 [Philipp Moritz] remove plain pointer in plasma client, part II
627b7c75 [Philipp Moritz] fix python extension name
30bd68b7 [Philipp Moritz] remove plain pointer in plasma client, part I
77d98227 [Philipp Moritz] put all the object code into a common library
0fdd4cd5 [Philipp Moritz] link libarrow.a and remove hardcoded optimization flags
8daea699 [Philipp Moritz] fix includes according to google styleguide
65ac7433 [Philipp Moritz] remove offending c++ flag from c flags
7003a4a4 [Philipp Moritz] fix valgrind test by setting working directory
217ff3d8 [Philipp Moritz] add valgrind heuristic
9c703c20 [Philipp Moritz] integrate client tests
9e5ae0e1 [Philipp Moritz] port serialization tests to gtest
0b8593db [Robert Nishihara] Port change from Ray. Change listen backlog size from 5 to 128.
b9a5a06e [Philipp Moritz] fix includes
ed680f97 [Philipp Moritz] reformat the code
f40f85bd [Philipp Moritz] add clang-format exceptions
d6e60d26 [Philipp Moritz] do not compile plasma on windows
f936adb7 [Philipp Moritz] build plasma python client only if python is available
e11b0e86 [Philipp Moritz] fix pthread
74ecb199 [Philipp Moritz] don't link against Python libraries
b1e0335a [Philipp Moritz] fix linting
7f7e7e78 [Philipp Moritz] more linting
79ea0ca7 [Philipp Moritz] fix clang-tidy
99420e8f [Philipp Moritz] add rat exceptions
6cee1e25 [Philipp Moritz] fix
c93034fb [Philipp Moritz] add Apache 2.0 headers
63729130 [Philipp Moritz] fix malloc?
99537c94 [Philipp Moritz] fix compiler warnings
cb3f3a38 [Philipp Moritz] compile C files with CMAKE_C_FLAGS
e649c2af [Philipp Moritz] fix compilation
04c2edb3 [Philipp Moritz] add missing file
51ab9630 [Philipp Moritz] fix compiler warnings
9ef7f412 [Philipp Moritz] make the plasma store compile
e9f9bb4a [Philipp Moritz] Initial commit of the plasma store. Contributors: Philipp Moritz, Robert Nishihara, Richard Shin, Stephanie Wang, Alexey Tumanov, Ion Stoica @ RISElab, UC Berkeley (2017) [from https://github.com/ray-project/ray/commit/b94b4a35e04d8d2c0af4420518a4e9a94c1c9b9f]
* ARROW-1151: [C++] Add branch prediction to RETURN_NOT_OK
Also added some missing status checks to builder-benchmark
Author: Wes McKinney <we...@twosigma.com>
Closes #782 from wesm/ARROW-1151 and squashes the following commits:
9b488a0e [Wes McKinney] Try to fix snappy warning
06276119 [Wes McKinney] Restore check macros used in libplasma
83b3f36d [Wes McKinney] Add branch prediction to RETURN_NOT_OK
* ARROW-1154: [C++] Import miscellaneous computational utility code from parquet-cpp
I will make a corresponding PR to parquet-cpp to ensure that this code migration is complete enough.
Author: Wes McKinney <we...@twosigma.com>
Closes #785 from wesm/ARROW-1154 and squashes the following commits:
08b54c98 [Wes McKinney] Fix variety of compiler warnings
ddc7354b [Wes McKinney] Fixes to get PARQUET-1045 working
f5cd0259 [Wes McKinney] Import miscellaneous computational utility code from parquet-cpp
* ARROW-1185: [C++] Status class cleanup, warn_unused_result attribute and Clang warning fixes
This was tedious, but overdue. The Status class in Arrow as originally imported from Apache Kudu, which had been modified from standard use in Google projects. I simplified the implementation to bring it more in line with the Status implementation used in TensorFlow.
This also addresses ARROW-111 by providing an attribute to warn in Clang if a Status is ignored
Author: Wes McKinney <we...@twosigma.com>
Closes #814 from wesm/status-cleaning and squashes the following commits:
7b7e6517 [Wes McKinney] Bring Status implementation somewhat more in line with TensorFlow and other Google codebases, remove unused posix code. Add warn_unused_result attribute and fix clang warnings
* ARROW-1630: [Serialization] Support Python datetime objects
An additional pair of eyes would be helpful, somewhat strangely the tests are passing for some datetime objects and not for others.
Author: Philipp Moritz <pc...@gmail.com>
Closes #1153 from pcmoritz/serialize-datetime and squashes the following commits:
f3696ae4 [Philipp Moritz] add numpy to LICENSE.txt
a94bca7d [Philipp Moritz] put PyDateTime_IMPORT higher up
0ae645e9 [Philipp Moritz] windows fixes
cbd1b222 [Philipp Moritz] get rid of gmtime_r
f3ea6699 [Philipp Moritz] use numpy datetime code to implement time conversions
e644f4f5 [Philipp Moritz] linting
f38cbd46 [Philipp Moritz] fixes
6e549c47 [Philipp Moritz] serialize datetime
* ARROW-1559: [C++] Add Unique kernel and refactor DictionaryBuilder to be a stateful kernel
Only intended to implement selective categorical conversion in `to_pandas()` but it seems that there is a lot missing to do this in a clean fashion.
Author: Wes McKinney <we...@twosigma.com>
Closes #1266 from xhochy/ARROW-1559 and squashes the following commits:
50249652 [Wes McKinney] Fix MSVC linker issue
b6cb1ece [Wes McKinney] Export CastOptions
4ea3ce61 [Wes McKinney] Return NONE Datum in else branch of functions
4f969c6b [Wes McKinney] Move deprecation suppression after flag munging
7f557cc0 [Wes McKinney] Code review comments, disable C4996 warning (equivalent to -Wno-deprecated) in MSVC builds
84717461 [Wes McKinney] Do not compute hash table threshold on each iteration
ae8f2339 [Wes McKinney] Fix double to int64_t conversion warning
c1444a26 [Wes McKinney] Fix doxygen warnings
2de85961 [Wes McKinney] Add test cases for unique, dictionary_encode
383b46fd [Wes McKinney] Add Array methods for Unique, DictionaryEncode
0962f06b [Wes McKinney] Add cast method for Column, chunked_array and column factory functions
62c3cefd [Wes McKinney] Datum stubs
27151c47 [Wes McKinney] Implement Cast for chunked arrays, fix kernel implementation. Change kernel API to write to a single Datum
1bf2e2f4 [Wes McKinney] Fix bug with column using wrong type
eaadc3e5 [Wes McKinney] Use macros to reduce code duplication in DoubleTableSize
6b4f8f3c [Wes McKinney] Fix datetime64->date32 casting error raised by refactor
2c77a19e [Wes McKinney] Some Decimal->Decimal128 renaming. Add DecimalType base class
c07f91b3 [Wes McKinney] ARROW-1559: Add unique kernel
* ARROW-1693: [JS] Expand JavaScript implementation, build system, fix integration tests
This PR adds a workaround for reading the metadata layout for C++ dictionary-encoded vectors.
I added tests that validate against the C++/Java integration suite. In order to make the new tests pass, I had to update the generated flatbuffers format and add a few types the JS version didn't have yet (Bool, Date32, and Timestamp). It also uses the new `isDelta` flag on DictionaryBatches to determine whether the DictionaryBatch vector should replace or append to the existing dictionary.
I also added a script for generating test arrow files from the C++ and Java implementations, so we don't break the tests updating the format in the future. I saved the generated Arrow files in with the tests because I didn't see a way to pipe the JSON test data through the C++/Java json-to-arrow commands without writing to a file. If I missed something and we can do it all in-memory, I'd be happy to make that change!
This PR is marked WIP because I added an [integration test](https://github.com/apache/arrow/commit/6e98874d9f4bfae7758f8f731212ae7ceb3f1321#diff-18c6be12406c482092d4b1f7bd70a8e1R22) that validates the JS reader reads C++ and Java files the same way, but unfortunately it doesn't. Debugging, I noticed a number of other differences between the buffer layout metadata between the C++ and Java versions. If we go ahead with @jacques-n [comment in ARROW-1693](https://issues.apache.org/jira/br [...]
cc @TheNeuralBit
Author: Paul Taylor <pa...@me.com>
Author: Wes McKinney <we...@twosigma.com>
Closes #1294 from trxcllnt/generate-js-test-files and squashes the following commits:
f907d5a7 [Paul Taylor] fix aggressive closure-compiler mangling in the ES5 UMD bundle
57c7df45 [Paul Taylor] remove arrow files from perf tests
5972349c [Paul Taylor] update performance tests to use generated test data
14be77f4 [Paul Taylor] fix Date64Vector TypedArray, enable datetime integration tests
5660eb34 [Wes McKinney] Use openjdk8 for integration tests, jdk7 for main Java CI job
019e8e24 [Paul Taylor] update closure compiler with full support for ESModules, and remove closure-compiler-scripts
48111290 [Paul Taylor] Add support for reading Arrow buffers < MetadataVersion 4
c72134a5 [Paul Taylor] compile JS source in integration tests
c83a700d [Wes McKinney] Hack until ARROW-1837 resolved. Constrain unsigned integers max to signed max for bit width
fd3ed475 [Wes McKinney] Uppercase hex values
224e041c [Wes McKinney] Remove hard-coded file name to prevent primitive JSON file from being clobbered
0882d8e9 [Paul Taylor] separate JS unit tests from integration tests in CI
1f6a81b4 [Paul Taylor] add missing mkdirp for test json data
19136fbf [Paul Taylor] remove test data files in favor of auto-generating them in CI
9f195682 [Paul Taylor] Generate test files when the test run if they don't exist
0cdb74e0 [Paul Taylor] Add a cli arg to integration_test.py generate test JSON files for JS
cc744564 [Paul Taylor] resolve LICENSE.txt conflict
33916230 [Paul Taylor] move js license to top-level license.txt
d0b61f49 [Paul Taylor] add validate package script back in, make npm-release.sh suitable for ASF release process
7e3be574 [Paul Taylor] Copy license.txt and notice.txt into target dirs from arrow root.
c8125d2d [Paul Taylor] Update readme to reflect new Table.from signature
49ac3398 [Paul Taylor] allow unrecognized cli args in gulpfile
3c52587e [Paul Taylor] re-enable node_js job in travis
cb142f11 [Paul Taylor] add npm release script, remove unused package scripts
d51793dd [Paul Taylor] run tests on src folder for accurate jest coverage statistics
c087f482 [Paul Taylor] generate test data in build scripts
1d814d00 [Paul Taylor] excise test data csvs
14d48964 [Paul Taylor] stringify Struct Array cells
1f004968 [Paul Taylor] rename FixedWidthListVector to FixedWidthNumericVector
be73c918 [Paul Taylor] add BinaryVector, change ListVector to always return an Array
02fb3006 [Paul Taylor] compare iterator results in integration tests
e67a66a1 [Paul Taylor] remove/ignore test snapshots (getting too big)
de7d96a3 [Paul Taylor] regenerate test arrows from master
a6d3c83e [Paul Taylor] enable integration tests
44889fbe [Paul Taylor] report errors generating test arrows
fd68d510 [Paul Taylor] always increment validity buffer index while reading
562eba7d [Paul Taylor] update test snapshots
d4399a8a [Paul Taylor] update integration tests, add custom jest vector matcher
8d44dcd7 [Paul Taylor] update tests
6d2c03d4 [Paul Taylor] clean arrows folders before regenerating test data
4166a9ff [Paul Taylor] hard-code reader to Arrow spec and ignore field layout metadata
c60305d6 [Paul Taylor] refactor: flatten vector folder, add more types
ba984c61 [Paul Taylor] update dependencies
5eee3eaa [Paul Taylor] add integration tests to compare how JS reads cpp vs. java arrows
d4ff57aa [Paul Taylor] update test snapshots
407b9f5b [Paul Taylor] update reader/table tests for new generated arrows
85497069 [Paul Taylor] update cli args to execute partial test runs for debugging
eefc256d [Paul Taylor] remove old test arrows, add new generated test arrows
0cd31ab9 [Paul Taylor] add generate-arrows script to tests
3ff71384 [Paul Taylor] Add bool, date, time, timestamp, and ARROW-1693 workaround in reader
4a34247c [Paul Taylor] export Row type
141194e7 [Paul Taylor] use fieldNode.length as vector length
c45718e7 [Paul Taylor] support new DictionaryBatch isDelta flag
9d8fef97 [Paul Taylor] split DateVector into Date32 and Date64 types
8592ff3c [Paul Taylor] update generated format flatbuffers
* ARROW-1703: [C++] Vendor exact version of jemalloc we depend on
Author: Uwe L. Korn <uw...@xhochy.com>
Closes #1334 from xhochy/ARROW-1703 and squashes the following commits:
7282583f [Uwe L. Korn] ARROW-1703: [C++] Vendor exact version of jemalloc we depend on
* ARROW-2798: [Plasma] Use hashing function that takes into account all UniqueID bytes
Now, the hashing of UniqueID in plasma is too simple which has caused a problem. In some cases(for example, in github/ray, UniqueID is composed of a taskID and a index), the UniqueID may be like "ffffffffffffffffffff00", "ffffffffffffffffff01", "fffffffffffffffffff02" ... . The current hashing method is only to copy the first few bytes of a UniqueID and the result is that most of the hashed ids are same, so when the hashed ids put to plasma store, it will become very slow when se [...]
In fact, the same PR has been merged into ray, see https://github.com/ray-project/ray/pull/2174.
and I have tested the perf between the new hashing method and the original one with putting lots of objects continuously, it seems the new hashing method doesn't cost more time.
Author: songqing <zh...@163.com>
Closes #2220 from songqing/oid-hashing and squashes the following commits:
5c803aa0 <songqing> modify murmurhash LICENSE
8b8aa3e1 <songqing> add murmurhash LICENSE
d8d5f93f <songqing> lint fix
426cd1e2 <songqing> lint fix
4767751d <songqing> Use hashing function that takes into account all UniqueID bytes
* ARROW-2634: [Go] Add Go license details to LICENSE.txt
Author: Wes McKinney <we...@apache.org>
Closes #2221 from wesm/ARROW-2634 and squashes the following commits:
c65a8193 <Wes McKinney> Add Go license details to LICENSE.txt
* ARROW-3050: [C++] Adopt HiveServer2 client codebase from
cloudera/hs2client. Add Thrift to thirdparty toolchain
This patch incorporates patches developed at cloudera/hs2client (Apache 2.0) by
the following authors:
* 12 Wes McKinney <we...@apache.org>, <we...@cloudera.com>
* 2 Thomas Tauber-Marshall <tm...@cloudera.com>
* 2 陈晓发 <ch...@chelun.com>
* 2 Matthew Jacobs <ja...@gmail.com>, <mj...@cloudera.com>
* 1 Miki Tebeka <mi...@gmail.com>
* 1 Tim Armstrong <ta...@cloudera.com>
* 1 henryr <he...@gmail.com>
Closes #2444
Change-Id: I88aed528a9f4d2069a4908f6a09230ade2fbe50a
* ARROW-1325: [R] Initial R package that builds against the arrow C++ library
This is very minimal in functionality, it just gives a simple R package that calls a function from the arrow C++ library.
Author: Romain Francois <ro...@purrple.cat>
Author: Wes McKinney <we...@apache.org>
Closes #2489 from romainfrancois/r-bootstrap and squashes the following commits:
89f14b4ba <Wes McKinney> Add license addendums
9e3ffb4d2 <Romain Francois> skip using rpath linker option
79c50011d <Romain Francois> follow up from @wesm comments on #2489
a1a5e7c33 <Romain Francois> + installation instructions
fb412ca1d <Romain Francois> not checking for headers on these files
2848fd168 <Romain Francois> initial R :package: with travis setup and testthat suite, that links to arrow c++ library and calls arrow::int32()
* ARROW-3187: [C++] Add support for using glog (Google logging library)
1. `glog` provides richer information.
2. `glog` can print good call stack while crashing, which is very helpful for debugging.
3. Make logging pluggable with `glog` or original log using a macro. Users can enable/disable `glog` using the cmake option `ARROW_USE_GLOG`.
Author: Yuhong Guo <yu...@antfin.com>
Author: Wes McKinney <we...@apache.org>
Closes #2522 from guoyuhong/glog and squashes the following commits:
b359640d4 <Yuhong Guo> Revert some useless changes.
38560c06e <Yuhong Guo> Change back the test code to fix logging-test
e3203a598 <Wes McKinney> Some fixes, run logging-test
4a9d1728b <Wes McKinney> Fix Flatbuffers download url
f36430836 <Yuhong Guo> Add test code to only include glog lib and init it without other use.
c8269fd88 <Yuhong Guo> Change ARROW_JEMALLOC_LINK_LIBS setting to ARROW_LINK_LIBS
34e6841f8 <Yuhong Guo> Add pthread
48afa3484 <Yuhong Guo> Address comment
12f9ba7e9 <Yuhong Guo> Disable glog from ARROW_BUILD_TOOLCHAIN
62f20002d <Yuhong Guo> Add -pthread to glog
673dbebe5 <Yuhong Guo> Try to fix ci FAILURE
69c1e7979 <Yuhong Guo> Add pthread for glog
fbe9cc932 <Yuhong Guo> Change Thirdpart to use EP_CXX_FLAGS
6f4d1b8fc <Yuhong Guo> Add lib64 to lib path suffix.
84532e338 <Yuhong Guo> Add glog to Dockerfile
ccc03cb12 <Yuhong Guo> Fix a bug
7bacd53ef <Yuhong Guo> Add LICENSE information.
9a3834caa <Yuhong Guo> Enable glog and fix building error
2b1f7e00e <Yuhong Guo> Turn glog off.
7d92091a6 <Yuhong Guo> Hide glog symbols from libarrow.so
a6ff67110 <Yuhong Guo> Support offline build of glog
14865ee93 <Yuhong Guo> Try to fix MSVC building failure
53cecebef <Yuhong Guo> Change log level to enum and refine code
09c6af7b9 <Yuhong Guo> Enable glog in plasma
* ARROW-3182: [Gandiva] Integrate gandiva to arrow build. Update licenses to apache license.
Fix clang-format, cpplint warnings, -Wconversion warnings and other warnings
with -DBUILD_WARNING_LEVEL=CHECKIN. Fix some build toolchain issues, Arrow
target dependencies. Remove some unused CMake code
* ARROW-3536: [C++] Add UTF8 validation functions
The baseline UTF8 decoder is adapted from Bjoern Hoehrmann's DFA-based implementation.
The common case of runs of ASCII chars benefit from a fast path handling 8 bytes at a time.
Benchmark results (on a Ryzen 7 machine with gcc 7.3):
```
-----------------------------------------------------------------------------
Benchmark Time CPU Iterations
-----------------------------------------------------------------------------
BM_ValidateTinyAscii/repeats:1 3 ns 3 ns 245245630 3.26202GB/s
BM_ValidateTinyNonAscii/repeats:1 7 ns 7 ns 104679950 1.54295GB/s
BM_ValidateSmallAscii/repeats:1 10 ns 10 ns 66365983 13.0928GB/s
BM_ValidateSmallAlmostAscii/repeats:1 37 ns 37 ns 18755439 3.69415GB/s
BM_ValidateSmallNonAscii/repeats:1 68 ns 68 ns 10267387 1.82934GB/s
BM_ValidateLargeAscii/repeats:1 4140 ns 4140 ns 171331 22.5003GB/s
BM_ValidateLargeAlmostAscii/repeats:1 24472 ns 24468 ns 28565 3.80816GB/s
BM_ValidateLargeNonAscii/repeats:1 50420 ns 50411 ns 13830 1.84927GB/s
```
The case of tiny strings is probably the most important for the use case of CSV type inference.
PS: benchmarks on the same machine with clang 6.0:
```
-----------------------------------------------------------------------------
Benchmark Time CPU Iterations
-----------------------------------------------------------------------------
BM_ValidateTinyAscii/repeats:1 3 ns 3 ns 213945214 2.84658GB/s
BM_ValidateTinyNonAscii/repeats:1 8 ns 8 ns 90916423 1.33072GB/s
BM_ValidateSmallAscii/repeats:1 7 ns 7 ns 91498265 17.4425GB/s
BM_ValidateSmallAlmostAscii/repeats:1 34 ns 34 ns 20750233 4.08138GB/s
BM_ValidateSmallNonAscii/repeats:1 58 ns 58 ns 12063206 2.14002GB/s
BM_ValidateLargeAscii/repeats:1 3999 ns 3999 ns 175099 23.2937GB/s
BM_ValidateLargeAlmostAscii/repeats:1 21783 ns 21779 ns 31738 4.27822GB/s
BM_ValidateLargeNonAscii/repeats:1 55162 ns 55153 ns 12526 1.69028GB/s
```
Author: Antoine Pitrou <an...@python.org>
Closes #2916 from pitrou/ARROW-3536-utf8-validation and squashes the following commits:
9c9713b78 <Antoine Pitrou> Improve benchmarks
e6f23963a <Antoine Pitrou> Use a larger state table allowing for single lookups
29d6e347c <Antoine Pitrou> Help clang code gen
e621b220f <Antoine Pitrou> Use memcpy for safe aligned reads, and improve speed of non-ASCII runs
89f6843d9 <Antoine Pitrou> ARROW-3536: Add UTF8 validation functions
* ARROW-3800: [C++] Vendor a string_view backport
Vendor the `std::string_view` backport from https://github.com/martinmoene/string-view-lite
Author: Antoine Pitrou <an...@python.org>
Closes #2974 from pitrou/ARROW-3800-string-view-backport and squashes the following commits:
4353414b6 <Antoine Pitrou> ARROW-3800: Vendor a string_view backport
* ARROW-3738: [C++] Parse ISO8601-like timestamps in CSV columns
Second granularity is allowed (we might want to add support for fractions of seconds, e.g. in the "YYYY-MM-DD[T ]hh:mm:ss.ssssss" format).
Timestamp conversion also participates in CSV type inference, since it's unlikely to produce false positives (e.g. a semantically "string" column that would be entirely made of valid timestamp strings).
Author: Antoine Pitrou <an...@python.org>
Closes #2952 from pitrou/ARROW-3738-csv-timestamps and squashes the following commits:
005a6e3f7 <Antoine Pitrou> ARROW-3738: Parse ISO8601-like timestamps in CSV columns
* ARROW-2653: [C++] Refactor hash table support
1. Get rid of all macros and sprinkled out hash table handling code
2. Improve performance by more careful selection of hash functions
(and better collision resolution strategy)
Integer hashing benefits from a very fast specialization.
Small string hashing benefits from a fast specialization with less branches
and less computation.
Generic string hashing falls back on hardware CRC32 or Murmur2-64, which has probably sufficient
performance given the typical distribution of string key length.
3. Add some tests and benchmarks
Author: Antoine Pitrou <an...@python.org>
Closes #3005 from pitrou/ARROW-2653 and squashes the following commits:
0c2dcc3de <Antoine Pitrou> ARROW-2653: Refactor hash table support
* ARROW-4017: [C++] Move vendored libraries in dedicated directory
Also update mapbox::variant to v1.1.5 (I'm not sure which version was previously vendored).
Author: Antoine Pitrou <an...@python.org>
Closes #3184 from pitrou/ARROW-4017-vendored-libraries and squashes the following commits:
fe69566d7 <Antoine Pitrou> ARROW-4017: Move vendored libraries in dedicated directory
* ARROW-3819: [Packaging] Update conda variant files to conform with feedstock after compiler migration
Crossbow builds:
- [kszucs/crossbow/build-403](https://github.com/kszucs/crossbow/branches/all?utf8=%E2%9C%93&query=build-403)
- [kszucs/crossbow/build-404](https://github.com/kszucs/crossbow/branches/all?utf8=%E2%9C%93&query=build-404)
- [kszucs/crossbow/build-405](https://github.com/kszucs/crossbow/branches/all?utf8=%E2%9C%93&query=build-405)
- [kszucs/crossbow/build-406](https://github.com/kszucs/crossbow/branches/all?utf8=%E2%9C%93&query=build-406)
- [kszucs/crossbow/build-407](https://github.com/kszucs/crossbow/branches/all?utf8=%E2%9C%93&query=build-407)
Author: Krisztián Szűcs <sz...@gmail.com>
Closes #3368 from kszucs/conda_forge_migration and squashes the following commits:
e0a5a6422 <Krisztián Szűcs> use --croot
3749a2ff9 <Krisztián Szűcs> git on osx; set FEEDSTOSK_ROOT
ca7217d7f <Krisztián Szűcs> support channel sources from variant files
33cba7118 <Krisztián Szűcs> fix conda path on linux
2505828b7 <Krisztián Szűcs> fix task names
0c4a10bc3 <Krisztián Szűcs> conda recipes for python 3.7; compiler migration
* ARROW-4198: [Gandiva] Added support to cast timestamp
Added howard hinnant date project as a third party library.
Used system timezone database for timezone information.
Author: Antoine Pitrou <pi...@free.fr>
Author: shyam <sh...@dremio.com>
Closes #3352 from shyambits2004/timestamp and squashes the following commits:
882a5cf6 <Antoine Pitrou> Tweak wording of vendored date library README
7f524805 <Antoine Pitrou> Small tweaks to license wording for the date library
9ee8eff4 <shyam> ARROW-4198 : Added support to cast timestamp
* ARROW-4546: Update LICENSE.txt with parquet-cpp licenses
- Ported parquet-cpp external license references
- Removed spurious duplicates (boost, mapbox)
Author: François Saint-Jacques <fs...@gmail.com>
Closes #3692 from fsaintjacques/ARROW-4546-parquet-license and squashes the following commits:
a5aa81e48 <François Saint-Jacques> ARROW-4546: Update LICENSE with parquet-cpp licenses
* ARROW-4690: Building TensorFlow compatible wheels for Arrow
This includes a Dockerfile that can be used to create wheels based on ubuntu 14.04 which are compatible with TensorFlow.
TODO before this can be merged:
- [x] write documentation how to build this
- [x] do more testing
Author: Philipp Moritz <pc...@gmail.com>
Closes #3766 from pcmoritz/ubuntu-wheels and squashes the following commits:
f708c29b <Philipp Moritz> remove tensorflow import check
599ce2e7 <Philipp Moritz> fix manylinux1 build instructions
f1fbedf8 <Philipp Moritz> remove tensorflow hacks
bf47f579 <Philipp Moritz> improve wording
4fb1d38b <Philipp Moritz> add documentation
078be98b <Philipp Moritz> add licenses
0ab0bccb <Philipp Moritz> cleanup
c7ab1395 <Philipp Moritz> fix
eae775d5 <Philipp Moritz> update
2820363e <Philipp Moritz> update
ed683309 <Philipp Moritz> update
e8c96ecf <Philipp Moritz> update
8a3b19e8 <Philipp Moritz> update
0fcc3730 <Philipp Moritz> update
fd387797 <Philipp Moritz> update
78dcf42d <Philipp Moritz> update
7726bb6a <Philipp Moritz> update
82ae4828 <Philipp Moritz> update
f44082ea <Philipp Moritz> update
deb30bfd <Philipp Moritz> update
50e40320 <Philipp Moritz> update
58f6c121 <Philipp Moritz> update
5e8ca589 <Philipp Moritz> update
5fa73dd5 <Philipp Moritz> update
595d0fe1 <Philipp Moritz> update
79006722 <Philipp Moritz> add libffi-dev
9ff5236d <Philipp Moritz> update
ca972ad0 <Philipp Moritz> update
60805e22 <Philipp Moritz> update
7a66ba35 <Philipp Moritz> update
1b56d1f1 <Philipp Moritz> zlib
eedef794 <Philipp Moritz> update
3ae2b5ab <Philipp Moritz> update
df297e1c <Philipp Moritz> add python build script
358e4f85 <Philipp Moritz> update
65afcebe <Philipp Moritz> update
11ccfc7e <Philipp Moritz> update
f1784245 <Philipp Moritz> update
b3039c8b <Philipp Moritz> update
9064c3ca <Philipp Moritz> update
c39f92a9 <Philipp Moritz> install tensorflow
ec4e2210 <Philipp Moritz> unicode
773ca2b6 <Philipp Moritz> link python
b690d64a <Philipp Moritz> update
5ce7f0d6 <Philipp Moritz> update
a9302fce <Philipp Moritz> install python-dev
f12e0cfe <Philipp Moritz> multibuild python 2.7
9342006b <Philipp Moritz> add git
ab2ef8e7 <Philipp Moritz> fix cmake install
cef997b5 <Philipp Moritz> install cmake and ninja
5d560faf <Philipp Moritz> add build-essential
adf2f705 <Philipp Moritz> add curl
f8d66963 <Philipp Moritz> remove xz
e439356e <Philipp Moritz> apt update
79fe557e <Philipp Moritz> add docker image for ubuntu wheel
* ARROW-4611: [C++] Rework CMake logic
This changes refactors much of our CMake logic to make use of built-in CMake paths and remove custom logic. It also switches to the use of more modern dependency management via CMake targets instead of plain text variables.
This includes the following fixes:
- Use CMake's standard find features, e.g. respecting the `*_ROOT` variables: https://issues.apache.org/jira/browse/ARROW-4383
- Add a Dockerfile for Fedora: https://issues.apache.org/jira/browse/ARROW-4730
- Add a Dockerfile for Ubuntu Xenial: https://issues.apache.org/jira/browse/ARROW-4731
- Add a Dockerfile for Ubuntu Bionic: https://issues.apache.org/jira/browse/ARROW-4849
- Add a Dockerfile for Debian Testing: https://issues.apache.org/jira/browse/ARROW-4732
- Change the clang-7 entry to use system packages without any dependency on conda(-forge): https://issues.apache.org/jira/browse/ARROW-4733
- Support `double-conversion<3.1`: https://issues.apache.org/jira/browse/ARROW-4617
- Use google benchmark from toolchain: https://issues.apache.org/jira/browse/ARROW-4609
- Use the `compilers` metapackage to install the correct binutils when using conda, otherwise system binutils to fix https://issues.apache.org/jira/browse/ARROW-4485
- RapidJSON throws compiler errors with GCC 8+ https://issues.apache.org/jira/browse/ARROW-4750
- Handle `EXPECT_OK` collision: https://issues.apache.org/jira/browse/ARROW-4760
- Activate flight build in ci/docker_build_cpp.sh: https://issues.apache.org/jira/browse/ARROW-4614
- Build Gandiva in the docker containers: https://issues.apache.org/jira/browse/ARROW-4644
Author: Uwe L. Korn <uw...@xhochy.com>
Closes #3688 from xhochy/build-on-fedora and squashes the following commits:
88e11fcfb <Uwe L. Korn> ARROW-4611: Rework CMake logic
* ARROW-4900: [C++] polyfill __cpuidex on mingw-w64
Author: Jeroen Ooms <je...@gmail.com>
Closes #3923 from jeroen/cpuidex and squashes the following commits:
59429f02 <Jeroen Ooms> Mention mingw-w64 polyfill in LICENSE.txt
28619330 <Jeroen Ooms> run clang-format
9e780465 <Jeroen Ooms> polyfill for __cpuidex on mingw-w64
* ARROW-5252: [C++] Use standard-compliant std::variant backport
Replace mapbox::variant with Michael Park's variant implementation.
Author: Antoine Pitrou <an...@python.org>
Closes #4259 from pitrou/ARROW-5252-variant-backport and squashes the following commits:
03dbc0e14 <Antoine Pitrou> ARROW-5252: Use standard-compliant std::variant backport
* ARROW-5648: [C++] Avoid using codecvt
Some antiquated C++ build chains miss the standard <codecvt> header.
Use a small vendored UTF8 implementation instead.
Author: Antoine Pitrou <an...@python.org>
Closes #4616 from pitrou/ARROW-5648-simple-utf8 and squashes the following commits:
54b1b2f68 <Antoine Pitrou> ARROW-5648: Avoid using codecvt
* ARROW-4800: [C++] Introduce a Result<T> class
- Mostly an adaptation of StatusOr from google/asylo (both header and unittests).
- Demonstrate usage in ipc/writer*
- If this PR is accepted I can do a follow-up PR to port over useful testing utilities.
Author: Micah Kornfield <em...@gmail.com>
Author: emkornfield <em...@gmail.com>
Closes #4501 from emkornfield/error_or and squashes the following commits:
82e48c453 <Micah Kornfield> fix linter. Add unittest.
aad79b183 <Micah Kornfield> rename to Return
1d7dbfbcd <Micah Kornfield> Use bkietz's suggestion. cleanup test
d8e80431c <Micah Kornfield> fix compile errors
cc626079c <Micah Kornfield> try non anonyous namespace
86e43ac89 <Micah Kornfield> export before
8a4b3ccf3 <Micah Kornfield> try explicit instantation for msvc
f12f6d027 <Micah Kornfield> Revert "remove ARROW_EXPORT from test and try add link to gtest_main"
9581b05b1 <Micah Kornfield> remove ARROW_EXPORT from test and try add link to gtest_main
7a21e577a <Micah Kornfield> try exporting private test classes for appveyor
0b44389da <Micah Kornfield> fix format
de9d2d0d9 <Micah Kornfield> remove duplicate code. fix format
504fcd7bf <emkornfield> Update cpp/src/arrow/error_or.h
31d9906c5 <Micah Kornfield> use vendored variant
aa540da09 <Micah Kornfield> fix append
6f459a5f9 <Micah Kornfield> address review comments
7a1e54de4 <Micah Kornfield> Add Arrow export
2886733fb <Micah Kornfield> use ARROW_RETURN_NOT_OK
f7ed04f00 <Micah Kornfield> address comments
3e2b3691a <Micah Kornfield> follow recommendation of docs for macro
d5e43d034 <Micah Kornfield> ARROW-4800: Introduce an ErrorOr class
* ARROW-5683: [R] Add snappy to Rtools Windows builds
Following https://github.com/r-windows/rtools-backports/pull/7 and https://github.com/r-windows/rtools-packages/pull/19, R Windows builds can now support Snappy compression. This patch tweaks the necessary files and unskips tests, in addition to some other PKGBUILD script cleanup.
Passing build here: https://ci.appveyor.com/project/nealrichardson/arrow/builds/25507388
Author: Neal Richardson <ne...@gmail.com>
Closes #4681 from nealrichardson/r-snappy and squashes the following commits:
0996a30e2 <Neal Richardson> Add license info for rtools-backports
1af08413c <Neal Richardson> Revert "Only run mine for now"
a5d967f0c <Neal Richardson> Get snappy from backports after all
a1c6390b6 <Neal Richardson> -lsnappy
a41544914 <Neal Richardson> Only run mine for now
b911568df <Neal Richardson> More comments; unskip tests
d7a74192a <Neal Richardson> Add snappy to PKGBUILD; prune some default cmake flags and add comment
* ARROW-5725: [Crossbow] Port conda recipes to azure pipelines
- [x] artifact uploading
- [x] osx build
- [x] win build (using appveyor, because of https://github.com/conda-forge/conda-forge.github.io/issues/703)
- [x] linux build
- [x] package gandiva
Author: Krisztián Szűcs <sz...@gmail.com>
Closes #4649 from kszucs/crossbow-azure and squashes the following commits:
20f6cecd3 <Krisztián Szűcs> update conda-win artifact patterns
0c470bb6a <Krisztián Szűcs> readme fixes
b6a86076b <Krisztián Szűcs> configure output folder for the artifacts
68d88a833 <Krisztián Szűcs> combine status api and checks api
111964957 <Krisztián Szűcs> fix artifact patterns
95cb44217 <Krisztián Szűcs> use FETCH_HEAD in the CI templates
cbb9c9ce7 <Krisztián Szűcs> rat
8f58839e1 <Krisztián Szűcs> use the default python on osx
f75efae18 <Krisztián Szűcs> use pip module for installing dependencies
2a598945d <Krisztián Szűcs> tabulate win template
9db3db1dd <Krisztián Szűcs> use pip3
2aa497748 <Krisztián Szűcs> azure template for docker tests
750f624c1 <Krisztián Szűcs> asset uploading script
e0d8fb9b2 <Krisztián Szűcs> git commit additional log
7fbce5df8 <Krisztián Szűcs> use appveyor for the win packages; upload assets scripts
d6c4ce9fa <Krisztián Szűcs> touch done_canary
611222e28 <Krisztián Szűcs> docker shm
ba0e88cce <Krisztián Szűcs> update old templates; query cxx include paths
0d76f1364 <Krisztián Szűcs> win
0c8464a4b <Krisztián Szűcs> parquet-cpp depend on exact arrow-cpp version
aecc2b19e <Krisztián Szűcs> displayName order
c42ebf595 <Krisztián Szűcs> quoting gandiva flags
8abd34779 <Krisztián Szűcs> move displayName after the script
bdf705ff0 <Krisztián Szűcs> OSX configuration
a874d1f99 <Krisztián Szűcs> gandiva flags
f50af1b51 <Krisztián Szűcs> path gymnastics
5cd9fa0b5 <Krisztián Szűcs> use pyarrow as recipe root
4b005892f <Krisztián Szűcs> try to fix assertion error
9ef81c567 <Krisztián Szűcs> use feedstock_root
0e826ac43 <Krisztián Szűcs> fix recipe directories
adae7c0f3 <Krisztián Szűcs> build all three recipes
7b60c9d07 <Krisztián Szűcs> pass arrow_version
ce740d799 <Krisztián Szűcs> fixing build_steps.sh path
df31ff7dc <Krisztián Szűcs> trying to fix feedstock and recipe roots
501d55341 <Krisztián Szűcs> set config
b2425e650 <Krisztián Szűcs> fix working directory
53e8eb24f <Krisztián Szűcs> don't use azure templates
b2fd21a24 <Krisztián Szűcs> use variables ]
2037f78fc <Krisztián Szűcs> port conda recipes to azure pipelines
* ARROW-5934: [Python] Bundle arrow's LICENSE with the wheels
Setuptools can handle multiple license files, but they get overwritten because of the same naming, so move the content python/LICENSE.txt to the top level LICENSE.txt. I've investigated a solution to concatenate these, but it would require to extend multiple distutils/setuptools/wheel commands in a really error-prone fashion.
- Distribute the top level LICENSE.txt file with the wheels
- Distribute the top level LICENSE.txt with the source distribution, however `python setup.py bdist` leaves LICENSE.txt file as a garbage in the rebository, it cannot cleanup properly probably because the relative directory reference.
Add 3rdparty licenses because of the static-linkage in the wheels:
- lz4: https://github.com/lz4/lz4/blob/dev/lib/LICENSE#L11-L13
- zstd: https://github.com/facebook/zstd/blob/dev/LICENSE#L13-L15
- rapidjson: because of transitive dependencies https://github.com/Tencent/rapidjson/blob/master/license.txt#L21
- uriparser: https://github.com/uriparser/uriparser/blob/master/COPYING#L15-L18
- double-conversion: https://github.com/google/double-conversion/blob/3f9cd30e1bca91c0a036ad8b2b7eb8e2d2290dd2/LICENSE#L8-L11
- snappy: https://github.com/google/snappy/blob/master/COPYING#L10-L13
- brotli: https://github.com/google/brotli/blob/master/LICENSE#L10
- jemalloc is already added
- protobuf is already added
- gflags: https://github.com/gflags/gflags/blob/master/COPYING.txt#L10-L13
- c-ares: https://github.com/c-ares/c-ares/blob/master/LICENSE.md
- glog: https://github.com/google/glog/blob/master/COPYING#L10-L13
- boost is already added
- flatbuffers is already added
- grpc: https://github.com/grpc/grpc/blob/master/NOTICE.txt
- gtest: do we redistribute any binaries from it?
- gbenchmark: do we redistribute any binaries from it?
- apache thrift added
- apache orc added
- zlip is redistributed as a shared library with all wheels (in the future we should switch to static linkage)
- openssl and libcrypto is redistributed as shared libraries with the windows wheels, so added the openssl license preceding version 3
bzip2 doesn't require to include its license with binary redistributions: https://github.com/asimonov-im/bzip2/blob/master/LICENSE
Authored-by: Krisztián Szűcs <sz...@gmail.com>
Signed-off-by: Wes McKinney <we...@apache.org>
Closes #4880 from kszucs/wheel-license and squashes the following commits:
b839964a2 <Krisztián Szűcs> openssl
f220c6609 <Krisztián Szűcs> zlib
064444d25 <Krisztián Szűcs> distribute notice as well
bee17ac0e <Krisztián Szűcs> orc notice
25f738cf8 <Krisztián Szűcs> Thrift and ORC
54a643bd8 <Krisztián Szűcs> grpc
baa77962f <Krisztián Szűcs> glog
0a3070e5a <Krisztián Szűcs> c-ares
749574f00 <Krisztián Szűcs> gflags
52d7e19a8 <Krisztián Szűcs> typo
0697927c6 <Krisztián Szűcs> brotli
51b9264fe <Krisztián Szűcs> re2
5418a0e2c <Krisztián Szűcs> snappy and double-conversion
5647ab923 <Krisztián Szűcs> lz4 and rapidjson
29fa7e046 <Krisztián Szűcs> wheel licensing
* ARROW-6258: [R] Add macOS build scripts
When installing the R package from source on macOS, if the configure script cannot find libarrow with pkgconfig, and if `apache-arrow` has not been installed via Homebrew (neither of which is the case on CRAN), an "autobrew" step happens: a script is downloaded and sourced, which uses a fork of Homebrew to download and install binary dependencies for bundling with the R package.
This patch alters the configure script to let you `FORCE_AUTOBREW`, which is useful for testing, and it will use a local `autobrew` script file, if found, rather than downloading it. The patch also adds the `autobrew` script and the `apache-arrow.rb` brew formula to the `r/tools` directory, alongside the similar script that downloads the Arrow C++ binary on Windows. The two scripts are copied exactly from their "upstream" versions (noted on L18 of each file), with two minor modificati [...]
See this in action at https://github.com/nealrichardson/arrow-r-nightly/blob/34d27bf482fa1d9f490003a8396fabdff5beea37/.travis.yml. Ordinarily I would add a Travis-CI job to `apache/arrow` for this, but I know we're anxious not to delay build times further, so I'll run this job nightly. Nightly runs will solve https://issues.apache.org/jira/browse/ARROW-5134, and it will also allow us to host an R package repository with nightly binaries for macOS (and Windows too, using the existing A [...]
To install a binary from that repository on macOS, `install.packages("arrow", repos="https://dl.bintray.com/nealrichardson/arrow-r")`.
One TODO: get @jeroen 's approval to include these scripts here under the Apache license and add a citation to LICENSE.txt.
Closes #5095 from nealrichardson/force-autobrew and squashes the following commits:
499296d37 <Neal Richardson> Add license information for autobrew
a63765bd7 <Neal Richardson> :rat:
1a8a77700 <Neal Richardson> Add autobrew scripts
f48e6ba1e <Neal Richardson> Check for local autobrew script
284fd871b <Neal Richardson> Add FORCE_AUTOBREW arg to r/configure to support testing
Authored-by: Neal Richardson <ne...@gmail.com>
Signed-off-by: François Saint-Jacques <fs...@gmail.com>
* ARROW-6454: [LICENSE] Add LLVM's license due to static linkage
Also adding to the conda-forge recipe https://github.com/conda-forge/arrow-cpp-feedstock/pull/96, which we should keep until the next release.
Closes #5250 from kszucs/llvm-license and squashes the following commits:
f732908ad <Krisztián Szűcs> add LLVM's license
Authored-by: Krisztián Szűcs <sz...@gmail.com>
Signed-off-by: Wes McKinney <we...@apache.org>
* ARROW-4649: [C++/CI/R] Add nightly job that tests the homebrew formula
This doesn't just test `brew install apache-arrow --HEAD` as the ticket suggested--it brings the Homebrew formula under our source control and tests against that. This will enable us to know when we need to update the Homebrew formula due to changes in dependencies or cmake configuration.
I've been testing out this approach on a different repository: https://travis-ci.org/nealrichardson/arrow-brew/builds/568531245
Closes #5360 from nealrichardson/homebrew-nightly and squashes the following commits:
4a50a3779 <Neal Richardson> Shuffle again
ea3f9b5fc <Neal Richardson> Move autobrew script again and fix URL to match regular homebrew
851ac6eae <Neal Richardson> Sort hunks :muscle:
1d6512b2a <Neal Richardson> Move homebrew formulae inside dev/tasks. Update autobrew package version in release script
ff1489ea7 <Neal Richardson> Use regular snapshot version for homebrew formula
79e9191f0 <Neal Richardson> Remove autoconf from homebrew formula because new jemalloc no longer requires it
8816279b4 <Neal Richardson> Fix autobrew audit check
b25ac6de6 <Neal Richardson> Fix licensing/:rat: for ci/apache-arrow.rb
89dea4ff0 <Neal Richardson> Parametrize homebrew task and also test 'autobrew'
d61290a87 <Neal Richardson> Remove bottle shas
a133385c1 <Neal Richardson> Fix test (hopefully?)
6d00fe444 <Neal Richardson> Fix regular expression
34af2a911 <Neal Richardson> Add missing test assertion
40fc370c6 <Neal Richardson> Attempt to set the homebrew formula arrow version in the release script
ea4368e49 <Neal Richardson> Re-alphabetize the seds in 00-prepare.sh
262c2415b <Neal Richardson> Rename homebrew task yml; add to LICENSE
163c67a9d <Neal Richardson> Add homebrew formula and crossbow nightly task
Authored-by: Neal Richardson <ne...@gmail.com>
Signed-off-by: Sutou Kouhei <ko...@clear-code.com>
* ARROW-6678: [C++][Parquet] Binary data stored in Parquet metadata must be base64-encoded to be UTF-8 compliant
I have added a simple base64 implementation (Zlib license) to arrow/vendored from
https://github.com/ReneNyffenegger/cpp-base64
Closes #5493 from wesm/ARROW-6678 and squashes the following commits:
c058e8694 <Wes McKinney> Simplify, add MSVC exports
06f75cd5b <Wes McKinney> Fix Python unit test that needs to base64-decode now
eabb121ba <Wes McKinney> Fix LICENSE.txt, add iwyu export
b3a584a29 <Wes McKinney> Add vendored base64 C++ implementation and ensure that Thrift KeyValue in Parquet metadata is UTF-8
Authored-by: Wes McKinney <we...@apache.org>
Signed-off-by: Micah Kornfield <em...@gmail.com>
* ARROW-6679: [RELEASE] Add license info for the autobrew scripts
cf. https://github.com/jeroen/autobrew/blob/gh-pages/LICENCE.txt
Closes #5501 from nealrichardson/autobrew-license and squashes the following commits:
3e790f5b8 <Neal Richardson> MIT license for autobrew
Authored-by: Neal Richardson <ne...@gmail.com>
Signed-off-by: Wes McKinney <we...@apache.org>
* ARROW-7101: [CI] Refactor docker-compose setup and use it with GitHub Actions
## Projecting ideas from ursabot
### Parametric docker images
The images are better parameterized now, meaning that we can build more variant of the same service. Couple of examples:
```console
UBUNTU=16.04 docker-compose build ubuntu-cpp
ARCH=arm64v8 UBUNTU=18.04 docker-compose build ubuntu-cpp
PYTHON=3.6 docker-compose build conda-python
ARCH=arm32v7 PYTHON=3.6 PANDAS=0.25 docker-compose build conda-python-pandas
```
Each variant has it's own docker image following a string naming schema:
`{org}/{arch}-{platform}-{platform-version}[[-{variant}-{variant-version}]..]:latest`
### Use *_build.sh and *_test.sh for each job
The docker images provide the environment, and each language backend usually should implement two scripts, a `build.sh` and a `test.sh`. This way dependent build like the docker python, r or c glib are able to reuse the build script of the ancestor without running its tests.
With small enough scripts, if the environment is properly set up even the non-docker builds should be reproducible locally. GitHub Actions support bash scripts across all three platforms, so we can reuse the same `*_build.sh` and `*_test.sh` scripts to execute the builds either in docker, on the CI or locally.
## Using GitHub Actions for running the builds
Regardless of the CI we're going to choose, the isolation constraint of different platforms requires some sort of virtualisation. Currently linux (and windows, but I have not tried it yet) has lightweight containerisation, so we should keep the linux builds isolated in docker containers. The rest of the platforms (windows and macOS) should be executed on the CI system.
GitHub Actions support all three major platforms, linux, windows and macOS. I've added cross platform builds for a couple of languages, like Rust, and Go, the rest are work in progress.
### Workflow
A workflow should define all builds of a language, mostly because the path filters can be defined on workflow level. For example the python builds should be triggered if either a cpp/** or a python/** file changes which can be covered in the same workflow file.
## Feature parity with the current builds
Reaching feature parity with all of the builds below is not a goal for this PR, the difficult ones should at least have a tracking JIRA ticket.
### Travis-CI
- [x] **Lint, Release tests**:
- `Lint / C++, Python, R, Rust, Docker, RAT`
- `Dev / Source Release`
- [x] **C++ unit tests w/ conda-forge toolchain, coverage**: without coverage
- `C++ / AMD64 Conda C++`
- [x] **Python 3.6 unit tests, conda-forge toolchain, coverage**: without coverage
- `Python / AMD64 Conda Python 3.6`
- [x] **[OS X] C++ w/ Xcode 9.3**:
- `C++ / AMD64 MacOS 10.14 C++`: with Xcode 10.3
- [x] **[OS X] Python w/ Xcode 9.3**:
- `Python / AMD64 MacOS 10.14 Python 3`: with Xcode 10.3
- [x] **Java OpenJDK8 and OpenJDK11**:
- `Java / AMD64 Debian Java JDK 8 Maven 3.5.2`
- `Java / AMD64 Debian Java JDK 11 Maven 3.6.2`
- [x] **Protocol / Flight Integration Tests**:
- `Dev / Protocol Test`
- [x] **NodeJS**: without running lint and coverage
- `NodeJS / AMD64 Debian NodeJS 11`
- [x] **C++ & GLib & Ruby w/ gcc 5.4**:
- `C++ / AMD64 Debian 10 C++`: with GCC 8.3
- `C++ / AMD64 Ubuntu 16.04 C++`: with GCC 5.4
- `C++ / AMD64 Ubuntu 18.04 C++`: with GCC 7.4
- `C GLib / AMD64 Ubuntu 18.04 C GLib`
- `Ruby / AMD64 Ubuntu 18.04 Ruby`
- [x] **[OS X] C++ & GLib & Ruby w/ XCode 10.2 & Homebrew**
- `C++ / AMD64 MacOS 10.14 C++`: with Xcode 10.3
- `C GLib / AMD64 MacOS 10.14 C Glib`: with Xcode 10.3
- `Ruby / AMD64 MacOS 10.14 Ruby`: with Xcode 10.3
- [x] **Go**: without coverage
- `Go / AMD64 Debian Go 1.12`
- [x] **R (with and without libarrow)**:
- `R / AMD64 Conda R 3.6`: with libarrow
- `R / AMD64 Ubuntu 18.04 R 3.6` with libarrow
### Appveyor
- ~JOB=Build, GENERATOR=Ninja, CONFIGURATION=Release, APPVEYOR_BUILD_WORKER_IMAGE=Visual Studio 2017~
- ~JOB=Toolchain, GENERATOR=Ninja, CONFIGURATION=Release, ARROW_S3=ON, ARROW_BUILD_FLIGHT=ON, ARROW_BUILD_GANDIVA=ON~
- ~JOB=Build_Debug, GENERATOR=Ninja, CONFIGURATION=Debug~
- ~JOB=MinGW32, MINGW_ARCH=i686, MINGW_PACKAGE_PREFIX=mingw-w64-i686, MINGW_PREFIX=c:\msys64\mingw32, MSYSTEM=MINGW32, USE_CLCACHE=false~
- ~JOB=MinGW64, MINGW_ARCH=x86_64, MINGW_PACKAGE_PREFIX=mingw-w64-x86_64, MINGW_PREFIX=c:\msys64\mingw64, MSYSTEM=MINGW64, USE_CLCACHE=false~
- [x] **JOB=Rust, TARGET=x86_64-pc-windows-msvc, USE_CLCACHE=false**:
- `Rust / AMD64 Windows 2019 Rust nightly-2019-09-25`
- [x] **JOB=C#, APPVEYOR_BUILD_WORKER_IMAGE=Visual Studio 2017, USE_CLCACHE=false**
- `C# / AMD64 Windows 2019 C# 2.2.103`
- [x] **JOB=Go, MINGW_PACKAGE_PREFIX=mingw-w64-x86_64 ...**:
- `Go / AMD64 Windows 2019 Go 1.12`
- ~JOB=R with libarrow, USE_CLCACHE=false, TEST_R_WITH_ARROW=TRUE, RWINLIB_LOCAL=%APPVEYOR_BUILD_FOLDER%\libarrow.zip~
### Github Actions
- [x] **Windows MSVC C++ / Build (Visual Studio 16 2019)**:
- `C++ / AMD64 Windows 2019 C++`: without tests
- [x] **Windows MSVC C++ / Build (Visual Studio 15 2017)**:
- `C++ / AMD64 Windows 2016 C++`: without tests
- [x] **Linux docker-compose / Test (C++ w/ clang-7 & system packages)**: all have llvm for gandiva but the compiler is set to gcc
- `C++ / AMD64 Debian 10 C++`: with GCC 8.3
- `C++ / AMD64 Ubuntu 16.04 C++`: with GCC 5.4
- `C++ / AMD64 Ubuntu 18.04 C++`: with GCC 7.4
- [x] **Linux docker-compose / Test (Rust)**: without rustfmt
- `Rust / AMD64 Debian Rust nightly-2019-09-25`
- [x] **Linux docker-compose / Test (Lint, Release tests)**:
- `Lint / C++, Python, R, Rust, Docker, RAT`
- `Dev / Source Release`
### Nightly Crossbow tests
The packaging builds are out of the scope if this PR, but the nightly **dockerized test** task are in.
Nightly tests:
- [x] docker-r
- [x] docker-r-conda
- [x] docker-r-sanitizer
- [x] docker-rust
- [x] docker-cpp
- [x] docker-cpp-cmake32
- [x] docker-cpp-release
- [x] docker-cpp-static-only
- [x] docker-c_glib
- [x] docker-go
- [x] docker-python-2.7
- [x] docker-python-3.6
- [x] docker-python-3.7
- [x] docker-python-2.7-nopandas
- [x] docker-python-3.6-nopandas
- [x] docker-java
- [x] docker-js
- [x] docker-docs
- [x] docker-lint
- [x] docker-iwyu: included in the lint
- [x] docker-clang-format: included in the lint
- [x] docker-pandas-master
- [x] docker-dask-integration
- [x] docker-hdfs-integration
- [x] docker-spark-integration
- [x] docker-turbodbc-integration
# TODOs left:
- [x] Fix the Apidoc generation for c_glib
- [x] Fix the JNI test for Gandiva and ORC
- [x] Test that crossbow tests are passing
- ~Optionally restore the travis configuration to incrementally decommission old builds~
## Follow-up JIRAs:
- [Archery] Consider porting the docker tool of ursabot to archery
- [Archery] Consider to use archery with or instead of the pre-commit hooks
- [Archery] Create a wrapper script in archery for docker compose in order to run the containers with the host's user and group
- [C++] GCC 5.4.0 has a compile errors, reproduce with UBUNTU=16.04 docker-compose run ubuntu-cpp
- [C++][CI] Test the ported fuzzit integration image
- [C++][CI] Turn off unnecessary features in the integration tests (spark/turbodbc/dask/hdfs)
- [C++][CI] Revisit ASAN UBSAN settings in every C++ based image
- [CI] Consider re-adding the removed debian testing image is removed
- [Go][CI] Pre-install the go dependencies in the dockerfile using go get
- [JS][CI] Pre-install the JS dependencies in the dockerfile
- [Rust][CI] Pre-install the rust dependencies in the dockerfile
- [Java][CI] Pre-install the java dependencies in the dockerfile
- [Ruby][CI] Pre-install the ruby dependencies in the dockerfile and remove it from the test script
- [C#][CI] Pre-install the C# dependencies in the dockerfile
- [R][CI] Fix the r-sanitizer build https://issues.apache.org/jira/browse/ARROW-6957
- [GLIB][MacOS] Fail to execute lua examples (fails to load 'lgi.corelgilua51' despite that lgi is installed)
- [C++][CMake] Automatically set ARROW_GANDIVA_PC_CXX_FLAGS for conda and OSX sdk (see cpp_build.sh)
- [C++][CI] Hiveserver2 instegration test fails to connect to impala container
- [CI][Spark] Support specific Spark version in the integration tet including latest
- [JS][CI] Move nodejs linting from js_build.sh to archery
- [Python][CI] create a docker image for python ASV benchmarks and fix the script
- [CI] Find a short but related prefix for the env vars used for the docker-compose file to prevent collisions
- [C#] the docker container fails to run because of the ubuntu host versions, see https://github.com/dotnet/core/issues/3509
- [C++][Windows] Enable more features on the windows GHA build
- [Doc] document docker-compose usage in the developer sphinx guide
- [CI][C++] Add .ccache to the docker-compose mounts
- [Archery][CI] Refactor the ci/scripts to a sourceable bash functions or to archery directly
- [C++][CI] Use scripts/util_coredump.sh to show automatic backtraces
- [C++] Fix the hanging C++ tests in Windows 2019
- [CI] Ask INFRA to set up the DOCKERHUB_* secrets for GitHub actions
- [C++][CI] Running Gandiva tests fails on Fedora:
Reproduce with: `docker-compose run -e ARROW_GANDIVA=ON fedora-cpp`
```
Running gandiva-internals-test, redirecting output into /build/cpp/build/test-logs/gandiva-internals-test.txt (attempt 1/1)
1364
: CommandLine Error: Option 'x86-experimental-vector-widening-legalization' registered more than once!
1365
LLVM ERROR: inconsistency in registered CommandLine options
1366
/build/cpp/src/gandiva
```
- [JS][CI] NodeJS build fails on Github Actions Windows node
```
> NODE_NO_WARNINGS=1 gulp build
# 'NODE_NO_WARNINGS' is not recognized as an internal or external command,
# operable program or batch file.
# npm ERR! code ELIFECYCLE
# npm ERR! errno 1
# npm ERR! apache-arrow@1.0.0-SNAPSHOT build: `NODE_NO_WARNINGS=1 gulp build`
# npm ERR! Exit status 1
# npm ERR!
# npm ERR! Failed at the apache-arrow@1.0.0-SNAPSHOT build script.
# npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
```
Closes #5589 from kszucs/docker-refactor and squashes the following commits:
5105d12e6 <Krisztián Szűcs> Rename pull-request folder to dev_cron
e9e9a7eec <Krisztián Szűcs> Use underscores for naming the workflow files
a92c99d03 <Krisztián Szűcs> Disable hanging C++ tests on windows
f158c89b5 <Krisztián Szűcs> Attempt to push from apache/arrow master; Don't push from crossbow tasks
0e1d470a1 <Krisztián Szűcs> Turn off ORC on macOS C++ test due to link error
258db5cff <Krisztián Szűcs> Only push docker images from apache/arrow repository
acdfcf086 <Krisztián Szűcs> Remove ORC from the brewfile
5102b85b1 <Krisztián Szűcs> Fix nodeJS workflow
032d6a388 <Krisztián Szűcs> Turn off 2 python builds
7f15b97a8 <Krisztián Szűcs> Filter branches
48b8d128a <Krisztián Szűcs> Fix workflows
36ad9d297 <Krisztián Szűcs> Disable builds
0f603af0c <Krisztián Szűcs> master only and cron workflows
28cc2d78d <Krisztián Szűcs> Rename Java JNI workflow
bcd8af7b7 <Krisztián Szűcs> Port the remaining travis utility scripts
ed5688154 <Krisztián Szűcs> Usage comments; recommend installing pandas from the docs because of its removal from conda_env_python
3c8c023ce <Krisztián Szűcs> Use Arch in volumes; some comments; remove conda version 'latest' from the images
771b023a8 <Krisztián Szűcs> Cleanup files; separate JNI builds
97ff8a122 <Krisztián Szűcs> Push docker images only from master
dc00b4297 <Krisztián Szűcs> Enable path filters
e0e2e1f46 <Krisztián Szűcs> Fix pandas master build
3814e0828 <Krisztián Szűcs> Fix manylinux volumes
c18edda70 <Krisztián Szűcs> Add CentOS version to the manylinux image names
c8b9dd6b1 <Krisztián Szűcs> Missing --pyargs argument for the python test command
33e646981 <Krisztián Szűcs> Turn off gandiva and flight for the HDFS test
b9c547889 <Krisztián Szűcs> Refactor docker-compose file and use it with github actions.
Authored-by: Krisztián Szűcs <sz...@gmail.com>
Signed-off-by: Krisztián Szűcs <sz...@gmail.com>
* ARROW-6633: [C++] Vendor double-conversion library
Since this is both a mandatory dependency and a small-ish library, vendor it to make the build chain simplier. Also, make its use private, because of Windows DLL exports. This incurs a small (~8-10%) performance hit on specific workloads.
Closes #5832 from pitrou/ARROW-6633-vendor-double-conversion and squashes the following commits:
b1e12c7bc <Antoine Pitrou> Remove unneeded code
85877648e <Antoine Pitrou> Add license
3b89b191e <Antoine Pitrou> Make use of double-conversion private.
9e1da51d1 <Antoine Pitrou> ARROW-6633: Vendor double-conversion library
Authored-by: Antoine Pitrou <an...@python.org>
Signed-off-by: François Saint-Jacques <fs...@gmail.com>
* ARROW-7178: [C++] Vendor forward compatible std::optional
Add a version of std::optional vendored from
https://github.com/martinmoene/optional-lite using git tag v3.2.0
Closes #5849 from gawain-bolton/ARROW-7178_vendor_forward_compatible_std_optional and squashes the following commits:
825213c4e <gawain.bolton> Amend LICENSE.txt for cpp/src/arrow/vendored/optional.hpp
1bb25a962 <gawain.bolton> ARROW-7178: Vendor forward compatible std::optional
Authored-by: gawain.bolton <ga...@cfm.fr>
Signed-off-by: Benjamin Kietzman <be...@gmail.com>
* ARROW-7169: [C++] Vendor uriparser library
The library is only used internally (the types and headers are not exposed).
Vendoring it makes it easier for core functionality to depend on it.
Closes #5865 from pitrou/ARROW-7169-vendor-uriparser and squashes the following commits:
83bb7c22a <Antoine Pitrou> ARROW-7169: Vendor uriparser library
Authored-by: Antoine Pitrou <an...@python.org>
Signed-off-by: Krisztián Szűcs <sz...@gmail.com>
* ARROW-6341: [Python] Implement low-level bindings for Dataset
Closes #5237 from kszucs/ARROW-6341 and squashes the following commits:
45121f77a <Krisztián Szűcs> Fix tests for MockFs
069d8f55e <Krisztián Szűcs> Don't expose SimpleDataSource
48b5556ef <Krisztián Szűcs> Test projected partitions
27dbe56ab <Krisztián Szűcs> Don't deprecate RandomFileAccess just remove in favor of CRandomFileAccess
64ca71245 <Krisztián Szűcs> Execute instead of scan
bd0d1d2e7 <Krisztián Szűcs> more type_name
f2ab5ebfe <Krisztián Szűcs> type_name
3988865ca <Krisztián Szűcs> Rebase again
b8949a3ce <Krisztián Szűcs> Result iterator api
7553caa96 <Krisztián Szűcs> Clang format
a1d82546f <Krisztián Szűcs> Remove ScanContext
6260734a3 <Krisztián Szűcs> Fix api changes
e6a562356 <Krisztián Szűcs> Expose root_partition setter; resolve a couple of review issues
c9ba0fb93 <Krisztián Szűcs> Fix review comments
f589ecb23 <Krisztián Szűcs> Removed todo notes
9b38a40cf <Krisztián Szűcs> Docstring additions
3210e9160 <Krisztián Szűcs> Fixing review issues
4384b74cf <Krisztián Szűcs> Enable PYARROW_BUILD_DATASET
f52e735a0 <Krisztián Szűcs> Remove DataFragment and ScanOptions
13eaf46a0 <Krisztián Szűcs> Remove move workaround
620ba6ffa <Krisztián Szűcs> schema as property
e9f77bd6b <Krisztián Szűcs> Expose root_partition
01510bcf8 <Krisztián Szűcs> Some docstrings
5beb0d26c <Krisztián Szűcs> Pxd definition fixes
f89dc4913 <Krisztián Szűcs> Data fragments
c9881c858 <Krisztián Szűcs> Downcast data fragments
032a4358c <Krisztián Szűcs> More expressions and testing
1a4a8544a <Krisztián Szűcs> Fix import errors if dataset is not enabled
2da1b5c76 <Krisztián Szűcs> Please the linters
d1bc74efe <Krisztián Szűcs> HivePartitionScheme
bf5dd17f4 <Krisztián Szűcs> Release the gil for std::move
a76dc6c3c <Krisztián Szűcs> Remove the move headers from flight
8cdfe1054 <Krisztián Szűcs> Expose more methods
53b64910e <Krisztián Szűcs> Expose Scalar/Comparison/Boolean expressions
444ae58a0 <Krisztián Szűcs> Expressions and scalar wrapping
01029a666 <Krisztián Szűcs> Test parquet data discovery
2e416ea8f <Krisztián Szűcs> Expressions
d14cf502b <Krisztián Szűcs> PartitionScheme
bd6e1d656 <Krisztián Szűcs> FileSystemDataSourceDiscovery
0c0e3752f <Krisztián Szűcs> Working scanner
18cfd949b <Krisztián Szűcs> Resolve issues with handling iterator results
43c3d2bb2 <Krisztián Szűcs> MockFileSystem and use Results
3bbc51bd0 <Krisztián Szűcs> MockFileSystem
5e279ead2 <Krisztián Szűcs> Basic scanning
c0d6715aa <Krisztián Szűcs> Rebase datasets
132e2f553 <Krisztián Szűcs> license
d51b560d7 <Krisztián Szűcs> data fragments and data sources
a0b35c7a0 <Krisztián Szűcs> skeleton and headers
Authored-by: Krisztián Szűcs <sz...@gmail.com>
Signed-off-by: Benjamin Kietzman <be...@gmail.com>
* ARROW-7675: [R][CI] Move Windows CI from Appveyor to GHA
This also includes https://issues.apache.org/jira/browse/ARROW-7202
Closes #6581 from nealrichardson/r-windows-gha
Lead-authored-by: Neal Richardson <ne...@gmail.com>
Co-authored-by: Sutou Kouhei <ko...@clear-code.com>
Signed-off-by: Sutou Kouhei <ko...@clear-code.com>
* ARROW-8064: [Dev] Implement Comment bot via Github actions
See it in action: https://github.com/kszucs/arrow/pull/16#issuecomment-597252743
Main drawback that is is much slower than ursabot, but we can optimize it by:
- porting crossbow to only depend on pygithub instead of libgit2 (will consume the rate limit, but should fit in)
- use caching or docker
Theoretically CROSSBOW_GITHUB_TOKEN is set as a github actions secret, see https://issues.apache.org/jira/browse/INFRA-19954
We can trigger a build once this is merged into master.
Closes #6571 from kszucs/master and squashes the following commits:
7a604a875 <Krisztián Szűcs> note that the license is BSD2
8586eb727 <Krisztián Szűcs> add license reference
def872434 <Krisztián Szűcs> RAT
a96e7e24c <Krisztián Szűcs> flake8
6f5da639d <Krisztián Szűcs> add requirements to docker whitelist
6678c2e0a <Krisztián Szűcs> update archery dependencies
33f65d48c <Krisztián Szűcs> revert removing the rest of the workflows
a82b8790a <Krisztián Szűcs> test dep
06a7716cb <Krisztián Szűcs> responses test dep
ba2522989 <Krisztián Szűcs> fix archery workflow syntax
9352ee05c <Krisztián Szűcs> run archery unittests
deb857ff1 <Krisztián Szűcs> checkout@v2 and fetch tags
215495a3d <Krisztián Szűcs> fix result path
748832f75 <Krisztián Szűcs> message formatter
ea1b7c863 <Krisztián Szűcs> no dry run
6c83b0c40 <Krisztián Szűcs> dry run
4789ac5d5 <Krisztián Szűcs> response ormatter
1b0b15d5a <Krisztián Szűcs> cleanup
2270a35a9 <Krisztián Szűcs> validate
035024fa0 <Krisztián Szűcs> validate callback
e791c627b <Krisztián Szűcs> diag
641227f73 <Krisztián Szűcs> diab
b22b20400 <Krisztián Szűcs> token
d95e86b7b <Krisztián Szűcs> path to event payload
3e9a27909 <Krisztián Szűcs> pygithub
ca1592d5d <Krisztián Szűcs> typo
3c1358eff <Krisztián Szűcs> triger event handler
55e65faf3 <Krisztián Szűcs> crossbow command
92568eb5e <Krisztián Szűcs> first draft of bot
99ea0c2b5 <Krisztián Szűcs> cat
3c0f16d83 <Krisztián Szűcs> remove all other workflows
1f8f21de9 <Krisztián Szűcs> diag event handling
2f613dd15 <Krisztián Szűcs> Check event handling (#15)
Authored-by: Krisztián Szűcs <sz...@gmail.com>
Signed-off-by: Krisztián Szűcs <sz...@gmail.com>
* ARROW-8111: [C++] User-defined timestamp parser option to CSV, new TimestampParser interface, and strptime-compatible impl
This builds on the work from #6631 while adding unit tests and additional benchmarks.
I also renamed arrow/util/parsing.h to arrow/util/value_parsing.h to make it slightly more discoverable.
Closes #7088 from wesm/ARROW-8111
Lead-authored-by: Wes McKinney <we...@apache.org>
Co-authored-by: Antoine Pitrou <an...@python.org>
Co-authored-by: Alexey Prutskov <al...@intel.com>
Co-authored-by: Artem Alekseev <ar...@intel.com>
Signed-off-by: Antoine Pitrou <an...@python.org>
* ARROW-8968: [C++][Gandiva] set data layout for pre-compiled IR to llvm::module
This PR explicitly sets data layout for pre-compiled IR to `llvm:module`. Without the PR, the following warning message is shown at link time of gandiva.
Most of this code comes from https://reviews.llvm.org/D80130 by @imaihal in llvm.
```
~/arrow/cpp/src/gandiva$ ../../build/debug/gandiva-binary-test -V
Running main() from /home/ishizaki/arrow/cpp/googletest_ep-prefix/src/googletest_ep/googletest/src/gtest_main.cc
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from TestBinary
[ RUN ] TestBinary.TestSimple
warning: Linking two modules of different data layouts: 'precompiled' is 'E-m:e-i1:8:16-i8:8:16-i64:64-f128:64-a:8:16-n32:64' whereas 'codegen' is 'E-m:e-i1:8:16-i8:8:16-i64:64-f128:64-v128:64-a:8:16-n32:64'
[ OK ] TestBinary.TestSimple (41 ms)
[----------] 1 test from TestBinary (41 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (41 ms total)
[ PASSED ] 1 test.
```
Closes #7295 from kiszk/ARROW-8968
Authored-by: Kazuaki Ishizaki <is...@jp.ibm.com>
Signed-off-by: Wes McKinney <we...@apache.org>
* ARROW-8795: [C++] Limited iOS support
Per suggestion from [JIRA](https://issues.apache.org/jira/browse/ARROW-8795), opening up a pull request to potentially get the ball rolling on having iOS as one of the supported platforms.
Per writeup in [this doc](https://github.com/UnfoldedInc/deck.gl-native-dependencies/blob/master/docs/iOS-BUILD.md#arrow-v0170), there are a few issues with cmake's Xcode generator that require a few workarounds even when using their most recent versions.
The writeup also describes issues that we've encountered building the library, as well as issues that were not resolved as they use optional features we are currently not making use of.
Closes #7189 from ilijapuaca/feature/ios-support
Authored-by: Ilija Puaca <ip...@gmail.com>
Signed-off-by: Antoine Pitrou <an...@python.org>
* ARROW-9004: [C++][Gandiva] Support building with LLVM 10
LLVM 7 and 8 are still supported.
Clang Tools still use 8 because Clang Tools 10 reports an error for
RaipdJSON in sanitizer build. We should work on this as a separated
task.
e.g.:
https://github.com/kou/arrow/runs/725786805?check_suite_focus=true
/usr/include/rapidjson/internal/stack.h:117:13: runtime error: applying non-zero offset 16 to null pointer
#0 0x7f4d336c0e2c in void arrow::rapidjson::internal::Stack<arrow::rapidjson::CrtAllocator>::Reserve<arrow::rapidjson::GenericValue<arrow::rapidjson::UTF8<char>, arrow::rapidjson::MemoryPoolAllocator<arrow::rapidjson::CrtAllocator> > >(unsigned long) /usr/include/rapidjson/internal/stack.h:117:13
Ubuntu 16.04 still use LLVM 8 because LLVM 10 requires C++ 14 but g++
5 on Ubuntu 16.04 is too old to use
cpp/src/arrow/vendored/datetime/date.h.
CentOS 8 packages still use LLVM 8 because there are no packages for
LLVM 10 on CentOS 8.
manylinux1 still use LLVM 8 because g++ 4.8.2 is old to build LLVM 9
or later.
Conda package for Windows still use LLVM 8 because llvmdev 9 or later
use Visual Studio 2017. We need to use Visual Studio 2017 not 2015 to
use llvmdev 9 or later.
Closes #7323 from kou/cpp-llvm-10
Authored-by: Sutou Kouhei <ko...@clear-code.com>
Signed-off-by: Sutou Kouhei <ko...@clear-code.com>
* ARROW-7605: [C++] Create and install "dependency bundle" static library including jemalloc, mimalloc, and any BUNDLED static library so that static linking to libarrow.a is possible
This PR is a renewed attempt to address the brokenness of our static libraries and their exported CMake targets.
To summarize what's wrong: if any BUNDLED library source is used, or if the jemalloc or mimalloc allocators are used, then the `libarrow.a` static library that we've been installing is unusable. If someone tries to link with it, they get a linker error looking for the missing dependencies. Additionally, the CMake target names exported for `arrow_static` includes the list of all of these bundled libraries, so even attempting to use the `arrow_static` target does not work since the thir [...]
As one downstream problem caused by this, we've been unable thus far to ship our non-system allocators in packages (R in particular) that do static linking.
Since it is neither safe nor reasonable to expect third party projects to attempt to provide substitute static libraries for the missing dependencies, I have implemented the following:
* Using an MIT-licensed CMake solution, I splice together all of the bundled static libraries to create and install `libarrow_bundled_dependencies.a` (and `arrow_bundled_dependencies.lib` with MSVC). I have verified that this bundle is viable and can be used to create statically linked executables
* Bundled dependencies are no longer exported in ArrowTargets-*.cmake. I spent a bunch of time trying to figure out how to get CMake to automatically pull in the "dependency bundle" when using `arrow_static`, but I couldn't figure out how to do it.
* Added a `ARROW_LINK_SHARED` option to cpp/examples/minimal_build/CMakeLists.txt to show how to create statically linked executables using arrow_static and the dependency bundle.
* Added a "run_static.sh" script to enable a bundled statically-linked build to be tested with Docker (perhaps we should add this to the CI job?)
* Added "run_static.bat" to exhibit a statically linked build with MSVC on Windows
Finally, I verified that I can now successfully create statically linked builds now on Linux, macOS, and Windows.
[1]: https://gist.github.com/cristianadam/ef920342939a89fae3e8a85ca9459b49
Closes #7696 from wesm/dependency-bundling
Lead-authored-by: Wes McKinney <we...@apache.org>
Co-authored-by: Neal Richardson <ne...@gmail.com>
Co-authored-by: Wes McKinney <we...@apache.org>
Signed-off-by: Wes McKinney <we...@apache.org>
* ARROW-9402: [C++] Rework portable wrappers for checked integer arithmetic
Vendor relevant code from the portable-snippets library (~ public domain):
https://github.com/nemequ/portable-snippets/tree/master/safe-math
Also fix some bugs in checked arithmetic (null values had their value slots checked).
Add compute scaffolding for stateful binary scalar functions.
Closes #7784 from pitrou/ARROW-9402-overflow-arith
Authored-by: Antoine Pitrou <an...@python.org>
Signed-off-by: Antoine Pitrou <an...@python.org>
* ARROW-10035: [C++] Update vendored libraries
Update some vendored libraries to their newest upstream version.
Closes #8234 from pitrou/ARROW-10035-bump-vendored
Authored-by: Antoine Pitrou <an...@python.org>
Signed-off-by: Antoine Pitrou <an...@python.org>
* ARROW-10328: [C++] Vendor fast_float number parsing library
This library is 2x to 3x faster for parsing strings to binary floating-point numbers.
Closes #8494 from pitrou/ARROW-10328-fast-float
Authored-by: Antoine Pitrou <an...@python.org>
Signed-off-by: Antoine Pitrou <an...@python.org>
* ARROW-8113: [C++] Lighter weight variant<>
- [x] Unit tests for `util::Variant<>`
- [x] Compilation time and code size comparison to ensure this refactoring is beneficial
- [x] Benchmark comparison to ensure compute and nested parquet are not impacted
- [x] More docstrings and comments
Locally (release build, no ccache):
```
$ ninja
ninja: no work to do.
$ export OBJS=`ls src/arrow/CMakeFiles/arrow_objlib.dir/compute/**/*.o`
$ git checkout master && rm $OBJS && time ninja $OBJS
real 0m16.918s
user 2m41.702s
sys 0m2.495s
$ du -h --total $OBJS | tail -n 1
total 7.3M
$ git checkout 8113-Implement-a-lighter-weigh && rm $OBJS && time ninja $OBJS
real 0m14.363s
user 2m19.939s
sys 0m2.125s
$ du -h --total $OBJS | tail -n 1
6.6M total
```
Closes #8472 from bkietz/8113-Implement-a-lighter-weigh
Lead-authored-by: Benjamin Kietzman <be...@gmail.com>
Co-authored-by: Antoine Pitrou <an...@python.org>
Signed-off-by: Antoine Pitrou <an...@python.org>
* ARROW-10948: [C++] Always use GTestConfig.cmake
Because FindGTest.cmake can't detect GoogleTest version. We need
version to ensure using GoogleTest 1.10.0 or later.
Closes #8945 from kou/cpp-gtest
Authored-by: Sutou Kouhei <ko...@clear-code.com>
Signed-off-by: Sutou Kouhei <ko...@clear-code.com>
* ARROW-11680: [C++] Add vendored version of folly's spsc queue
See issue for rationale / description.
To consider:
* I based licensing text on other vendored libraries but I'm no expert in how exactly to dot all the i's and cross all the t's.
* The constructor can throw if allocation of the ring buffer fails. In general, this sort of queue shouldn't be all that large anyways so this seems unlikely. We could modify the constructor to be private with a factory method that returns `Result` but at the moment I'm not making any changes to the existing code and so I'm hesitant to start doing so for this reason.
* ~~There is a test case for the above point but I could take it out if desired.~~ Had to remove test since a massive allocation did not throw on mac.
* ~~There is a comparative benchmark for boost. I'm not sure if we want to leave that in?~~ Had to remove benchmark since some build environments did not have boost::lockfree::spsc_queue
* I've renamed folly::ProducerConsumerQueue to arrow::util::SpscQueue. I feel the name ProducerConsumerQueue is too vague and could lead to improper usage. This queue is not safe for multiple readers or multiple writers.
Closes #9519 from westonpace/feature/folly
Lead-authored-by: Weston Pace <we...@gmail.com>
Co-authored-by: Antoine Pitrou <an...@python.org>
Signed-off-by: Antoine Pitrou <an...@python.org>
* ARROW-12068: [Python] Stop using distutils
According to PEP 632, distutils will be deprecated in Python 3.10 and removed in 3.12.
* switch to `setuptools` for general packaging
* use the `Version` class from the `packaging` project instead of `distutils.LooseVersion`
Closes #9849 from pitrou/ARROW-12068-remove-distutils
Authored-by: Antoine Pitrou <an...@python.org>
Signed-off-by: Antoine Pitrou <an...@python.org>
* ARROW-10797: [C++] Vendor and use PCG random generator library
Closes #8879 from pitrou/ARROW-10797-pcg
Authored-by: Antoine Pitrou <an...@python.org>
Signed-off-by: Antoine Pitrou <an...@python.org>
* PARQUET-490: [C++][Parquet] Basic support for reading DELTA_BINARY_PACKED data
Only works for int32/int64 when delta_bit_width_<=32.
1. Update the implement of DeltaBitPackDecoder
2. Revise the code of putting/getting ZigZag int and the unit test
3. Add the implement of putting/getting ZigZag int64 and unit test
Closes #10627 from shanhuuang/ARROW-13206
Lead-authored-by: shanhuuang <sh...@gmail.com>
Co-authored-by: pincheng.hs <pi...@alibaba-inc.com>
Co-authored-by: Antoine Pitrou <an...@python.org>
Signed-off-by: Antoine Pitrou <an...@python.org>
* ARROW-13739: [R] Support dplyr::count() and tally()
Closes #11306 from thisisnic/ARROW-13739_dplyr_count_tally
Lead-authored-by: Nic Crane <th...@gmail.com>
Co-authored-by: Nic <th...@gmail.com>
Co-authored-by: Neal Richardson <ne...@gmail.com>
Signed-off-by: Neal Richardson <ne...@gmail.com>
* ARROW-14550: [Doc] Remove the JSON license; a non-free one.
see https://issues.apache.org/jira/browse/ARROW-14550
Closes #11596 from attila-lendvai/license
Authored-by: Attila Lendvai <at...@lendvai.name>
Signed-off-by: Krisztián Szűcs <sz...@gmail.com>
* ARROW-10317: [Python] Document compute function options
Add docstrings for parameters of the various function options classes.
Automatically ingest those parameter docstrings into the generated compute function docstrings
(this uses vendored code from `numpydoc`).
Example with the `min_max` parameter docs:
* before:
```
array : Array-like
Argument to compute function
skip_nulls : optional
Parameter for ScalarAggregateOptions constructor. Either `options`
or `skip_nulls` can be passed, but not both at the same time.
min_count : optional
Parameter for ScalarAggregateOptions constructor. Either `options`
or `min_count` can be passed, but not both at the same time.
options : pyarrow.compute.ScalarAggregateOptions, optional
Parameters altering compute function semantics.
memory_pool : pyarrow.MemoryPool, optional
If not passed, will allocate memory from the default memory pool.
```
* after:
```
array : Array-like
Argument to compute function.
skip_nulls : bool, default True
Whether to skip (ignore) nulls in the input.
If False, any null in the input forces the output to null.
min_count : int, default 1
Minimum number of non-null values in the input. If the number
of non-null values is below `min_count`, the output is null.
options : pyarrow.compute.ScalarAggregateOptions, optional
Alternative way of passing options.
memory_pool : pyarrow.MemoryPool, optional
If not passed, will allocate memory from the default memory pool.
```
Closes #12076 from pitrou/ARROW-10317-py-doc-function-options
Authored-by: Antoine Pitrou <an...@python.org>
Signed-off-by: Antoine Pitrou <an...@python.org>
* ARROW-15896: [Python][C++] Add errno detail for filesystem "file not found" errors
This allows raising FileNotFoundError in Python, which is more user-friendly.
Closes #12626 from pitrou/ARROW-15896-filesystem-errno
Authored-by: Antoine Pitrou <an...@python.org>
Signed-off-by: Antoine Pitrou <an...@python.org>
* ARROW-16232: [C++] Include OpenTelemetry in LICENSE.txt
Closes #12921 from lidavidm/ARROW-16232
Authored-by: David Li <li...@gmail.com>
Signed-off-by: Sutou Kouhei <ko...@clear-code.com>
* ARROW-16228: [CI][Packaging][Conan] Add a job to test minimum build
ci/conan/ is based on
https://github.com/conan-io/conan-center-index/tree/master/recipes/arrow/
.
We'll update ci/conan/ to follow the latest cpp/ changes such as new
build options and send a pull request the updates to
https://github.com/conan-io/conan-center-index/ when we release a new
version.
Closes #12918 from kou/ci-conan
Authored-by: Sutou Kouhei <ko...@clear-code.com>
Signed-off-by: Sutou Kouhei <ko...@clear-code.com>
* ARROW-16183: [C++][FlightRPC] Support bundled UCX
Closes #12881 from cyb70289/16183-bundled-ucx
Authored-by: Yibo Cai <yi...@arm.com>
Signed-off-by: David Li <li...@gmail.com>
* WIP: [C++] Initial ADBC sketch
* WIP: [C++] Sketch API for partitioned result sets
* WIP: [C++] Clean up SQLite demo implementation
* WIP: [C++] Add basic Flight SQL implementation
* WIP: [C++] Add sketch of distributed results
* WIP: [C++] Clarify some docstrings
* WIP: [C++] Clarify some docstrings
* WIP: [C++] Make error handling more natural, remove requirement of common library
* WIP: [C++] Add explicit string length parameters
* WIP: [C++] Add basic metadata method definitions
* WIP: [C++] Expand FlightSql test
* WIP: [C++] Extract test_util.h
* WIP: [C++] Split out Flight SQL tests
* WIP: [C++] Reorganize headers
* WIP: [C++] Move methods out of structs
* WIP: [C++] Simplify wrapper
* WIP: [C++] Move methods out of structs
* WIP: [C++] Make statements configurable in a forward-compatible way
* WIP: [C++] Clarify sorting of partitions
* WIP: [C++] Don't use enums
* WIP: [C++] Sketch out API revisions
* WIP: [C++] Add additional database initialization
* WIP: [C++] Sketch out alternative initialization
* WIP: [C++] Add include guards
* WIP: [C++] Sketch out driver side of driver manager
* WIP: [C++] Sketch out driver manager
* WIP: [C++] Don't require a particular entrypoint
* WIP: [C++] Start sketching out driver manager stubs
* WIP: [C++] Minimal driver manager stubs
* WIP: [C++] Add stubs for basic query execution
* WIP: [C++] Fix Flight SQL driver
* WIP: [C++] Assume strings are null-terminated
* WIP: [C++] Fix typo
* WIP: [C++] Add prepared statements without parameters
* WIP: [C++] Update driver manager to match
* WIP: [C++] Try to fix Mac build errors
* WIP: [C++] Add binding for SQLite
* Reorganize for new repo
* Update readme
Co-authored-by: Jacques Nadeau <ja...@apache.org>
Co-authored-by: Uwe L. Korn <uw...@xhochy.com>
Co-authored-by: Wes McKinney <we...@twosigma.com>
Co-authored-by: Philipp Moritz <pc...@gmail.com>
Co-authored-by: Paul Taylor <pa...@me.com>
Co-authored-by: songqing <zh...@163.com>
Co-authored-by: Wes McKinney <we...@apache.org>
Co-authored-by: Romain Francois <ro...@purrple.cat>
Co-authored-by: Yuhong Guo <yu...@antfin.com>
Co-authored-by: praveenbingo <pr...@dremio.com>
Co-authored-by: Antoine Pitrou <an...@python.org>
Co-authored-by: Krisztián Szűcs <sz...@gmail.com>
Co-authored-by: Antoine Pitrou <pi...@free.fr>
Co-authored-by: François Saint-Jacques <fs...@gmail.com>
Co-authored-by: Jeroen Ooms <je...@gmail.com>
Co-authored-by: Micah Kornfield <em...@gmail.com>
Co-authored-by: Neal Richardson <ne...@gmail.com>
Co-authored-by: gawain.bolton <ga...@cfm.fr>
Co-authored-by: Sutou Kouhei <ko...@clear-code.com>
Co-authored-by: Alexey Prutskov <al...@intel.com>
Co-authored-by: Artem Alekseev <ar...@intel.com>
Co-authored-by: Kazuaki Ishizaki <is...@jp.ibm.com>
Co-authored-by: Ilija Puaca <ip...@gmail.com>
Co-authored-by: Wes McKinney <we...@apache.org>
Co-authored-by: Benjamin Kietzman <be...@gmail.com>
Co-authored-by: Weston Pace <we...@gmail.com>
Co-authored-by: shanhuuang <sh...@gmail.com>
Co-authored-by: pincheng.hs <pi...@alibaba-inc.com>
Co-authored-by: Nic Crane <th...@gmail.com>
Co-authored-by: Attila Lendvai <at...@lendvai.name>
Co-authored-by: Yibo Cai <yi...@arm.com>
---
.clang-format | 21 +
.cmake-format | 76 +
.gitignore | 89 +
.pre-commit-config.yaml | 34 +
LICENSE.txt | 2325 +++++++++++++++++++++++
README.md | 24 +
adbc.h | 661 +++++++
adbc_driver_manager/CMakeLists.txt | 47 +
adbc_driver_manager/adbc_driver_manager.cc | 248 +++
adbc_driver_manager/adbc_driver_manager.h | 50 +
adbc_driver_manager/adbc_driver_manager_test.cc | 103 +
cmake_modules/BuildUtils.cmake | 920 +++++++++
cmake_modules/DefineOptions.cmake | 348 ++++
cmake_modules/README.md | 6 +
drivers/flight_sql/flight_sql.cc | 520 +++++
drivers/flight_sql/flight_sql_test.cc | 192 ++
drivers/sqlite/sqlite.cc | 616 ++++++
drivers/sqlite/sqlite_test.cc | 303 +++
drivers/sqlite/test_sqlite_internal.cc | 30 +
drivers/sqlite/test_sqlite_internal.h | 32 +
drivers/test_util.h | 74 +
drivers/util.cc | 59 +
drivers/util.h | 43 +
23 files changed, 6821 insertions(+)
diff --git a/.clang-format b/.clang-format
new file mode 100644
index 0000000..9448dc8
--- /dev/null
+++ b/.clang-format
@@ -0,0 +1,21 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+---
+BasedOnStyle: Google
+ColumnLimit: 90
+DerivePointerAlignment: false
+IncludeBlocks: Preserve
diff --git a/.cmake-format b/.cmake-format
new file mode 100644
index 0000000..3e77733
--- /dev/null
+++ b/.cmake-format
@@ -0,0 +1,76 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# cmake-format configuration file
+# Use `archery lint --cmake-format --fix` to reformat all cmake files in the
+# source tree
+
+# -----------------------------
+# Options affecting formatting.
+# -----------------------------
+with section("format"):
+ # How wide to allow formatted cmake files
+ line_width = 90
+
+ # How many spaces to tab for indent
+ tab_size = 2
+
+ # If a positional argument group contains more than this many arguments,
+ # then force it to a vertical layout.
+ max_pargs_hwrap = 4
+
+ # If the statement spelling length (including space and parenthesis) is
+ # smaller than this amount, then force reject nested layouts.
+ # This value only comes into play when considering whether or not to nest
+ # arguments below their parent. If the number of characters in the parent
+ # is less than this value, we will not nest.
+ min_prefix_chars = 32
+
+ # If true, separate flow control names from their parentheses with a space
+ separate_ctrl_name_with_space = False
+
+ # If true, separate function names from parentheses with a space
+ separate_fn_name_with_space = False
+
+ # If a statement is wrapped to more than one line, than dangle the closing
+ # parenthesis on it's own line
+ dangle_parens = False
+
+ # What style line endings to use in the output.
+ line_ending = 'unix'
+
+ # Format command names consistently as 'lower' or 'upper' case
+ command_case = 'lower'
+
+ # Format keywords consistently as 'lower' or 'upper' case
+ keyword_case = 'unchanged'
+
+# ------------------------------------------------
+# Options affecting comment reflow and formatting.
+# ------------------------------------------------
+with section("markup"):
+ # enable comment markup parsing and reflow
+ enable_markup = False
+
+ # If comment markup is enabled, don't reflow the first comment block in
+ # eachlistfile. Use this to preserve formatting of your
+ # copyright/licensestatements.
+ first_comment_is_literal = True
+
+ # If comment markup is enabled, don't reflow any comment block which
+ # matchesthis (regex) pattern. Default is `None` (disabled).
+ literal_comment_pattern = None
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..1406c30
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,89 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+apache-rat-*.jar
+arrow-src.tar
+arrow-src.tar.gz
+
+# Compiled source
+*.a
+*.dll
+*.o
+*.py[ocd]
+*.so
+*.so.*
+*.bundle
+*.dylib
+.build_cache_dir
+dependency-reduced-pom.xml
+MANIFEST
+compile_commands.json
+build.ninja
+
+# Generated Visual Studio files
+*.vcxproj
+*.vcxproj.*
+*.sln
+*.iml
+
+# Linux perf sample data
+perf.data
+perf.data.old
+
+cpp/.idea/
+.clangd/
+cpp/.clangd/
+.cache/clangd/
+cpp/.cache/clangd/
+cpp/apidoc/xml/
+docs/example.gz
+docs/example1.dat
+docs/example3.dat
+python/.eggs/
+python/doc/
+# Egg metadata
+*.egg-info
+
+.vscode
+.idea/
+.pytest_cache/
+pkgs
+docker_cache
+.gdb_history
+*.orig
+.*.swp
+.*.swo
+
+site/
+
+# R files
+**/.Rproj.user
+**/*.Rcheck/
+**/.Rhistory
+.Rproj.user
+
+# macOS
+cpp/Brewfile.lock.json
+.DS_Store
+
+# docker volumes used for caching
+.docker
+
+# generated native binaries created by java JNI build
+java-dist/
+java-native-c/
+java-native-cpp/
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
new file mode 100644
index 0000000..f8c9ab1
--- /dev/null
+++ b/.pre-commit-config.yaml
@@ -0,0 +1,34 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To use this, install the python package `pre-commit` and
+# run once `pre-commit install`. This will setup a git pre-commit-hook
+# that is executed on each commit and will report the linting problems.
+# To run all hooks on all files use `pre-commit run -a`
+
+repos:
+ - repo: https://github.com/pocc/pre-commit-hooks
+ rev: v1.3.5
+ hooks:
+ - id: clang-format
+ args: [-i]
+ types_or: [c, c++]
+ - repo: https://github.com/cheshirekow/cmake-format-precommit
+ rev: v0.6.13
+ hooks:
+ - id: cmake-format
+ args: [--in-place, --config-files, .cmake-format]
diff --git a/LICENSE.txt b/LICENSE.txt
new file mode 100644
index 0000000..dd3cec0
--- /dev/null
+++ b/LICENSE.txt
@@ -0,0 +1,2325 @@
+
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+
+--------------------------------------------------------------------------------
+
+src/plasma/fling.cc and src/plasma/fling.h: Apache 2.0
+
+Copyright 2013 Sharvil Nanavati
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+--------------------------------------------------------------------------------
+
+src/plasma/thirdparty/ae: Modified / 3-Clause BSD
+
+Copyright (c) 2006-2010, Salvatore Sanfilippo <antirez at gmail dot com>
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+ * Redistributions of source code must retain the above copyright notice,
+ this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above copyright
+ notice, this list of conditions and the following disclaimer in the
+ documentation and/or other materials provided with the distribution.
+ * Neither the name of Redis nor the names of its contributors may be used
+ to endorse or promote products derived from this software without
+ specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.
+
+--------------------------------------------------------------------------------
+
+src/plasma/thirdparty/dlmalloc.c: CC0
+
+This is a version (aka dlmalloc) of malloc/free/realloc written by
+Doug Lea and released to the public domain, as explained at
+http://creativecommons.org/publicdomain/zero/1.0/ Send questions,
+comments, complaints, performance data, etc to dl@cs.oswego.edu
+
+--------------------------------------------------------------------------------
+
+src/plasma/common.cc (some portions)
+
+Copyright (c) Austin Appleby (aappleby (AT) gmail)
+
+Some portions of this file are derived from code in the MurmurHash project
+
+All code is released to the public domain. For business purposes, Murmurhash is
+under the MIT license.
+
+https://sites.google.com/site/murmurhash/
+
+--------------------------------------------------------------------------------
+
+src/arrow/util (some portions): Apache 2.0, and 3-clause BSD
+
+Some portions of this module are derived from code in the Chromium project,
+copyright (c) Google inc and (c) The Chromium Authors and licensed under the
+Apache 2.0 License or the under the 3-clause BSD license:
+
+ Copyright (c) 2013 The Chromium Authors. All rights reserved.
+
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions are
+ met:
+
+ * Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above
+ copyright notice, this list of conditions and the following disclaimer
+ in the documentation and/or other materials provided with the
+ distribution.
+ * Neither the name of Google Inc. nor the names of its
+ contributors may be used to endorse or promote products derived from
+ this software without specific prior written permission.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+--------------------------------------------------------------------------------
+
+This project includes code from Daniel Lemire's FrameOfReference project.
+
+https://github.com/lemire/FrameOfReference/blob/6ccaf9e97160f9a3b299e23a8ef739e711ef0c71/src/bpacking.cpp
+https://github.com/lemire/FrameOfReference/blob/146948b6058a976bc7767262ad3a2ce201486b93/scripts/turbopacking64.py
+
+Copyright: 2013 Daniel Lemire
+Home page: http://lemire.me/en/
+Project page: https://github.com/lemire/FrameOfReference
+License: Apache License Version 2.0 http://www.apache.org/licenses/LICENSE-2.0
+
+--------------------------------------------------------------------------------
+
+This project includes code from the TensorFlow project
+
+Copyright 2015 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+--------------------------------------------------------------------------------
+
+This project includes code from the NumPy project.
+
+https://github.com/numpy/numpy/blob/e1f191c46f2eebd6cb892a4bfe14d9dd43a06c4e/numpy/core/src/multiarray/multiarraymodule.c#L2910
+
+https://github.com/numpy/numpy/blob/68fd82271b9ea5a9e50d4e761061dfcca851382a/numpy/core/src/multiarray/datetime.c
+
+Copyright (c) 2005-2017, NumPy Developers.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+ * Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+
+ * Redistributions in binary form must reproduce the above
+ copyright notice, this list of conditions and the following
+ disclaimer in the documentation and/or other materials provided
+ with the distribution.
+
+ * Neither the name of the NumPy Developers nor the names of any
+ contributors may be used to endorse or promote products derived
+ from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+--------------------------------------------------------------------------------
+
+This project includes code from the Boost project
+
+Boost Software License - Version 1.0 - August 17th, 2003
+
+Permission is hereby granted, free of charge, to any person or organization
+obtaining a copy of the software and accompanying documentation covered by
+this license (the "Software") to use, reproduce, display, distribute,
+execute, and transmit the Software, and to prepare derivative works of the
+Software, and to permit third-parties to whom the Software is furnished to
+do so, all subject to the following:
+
+The copyright notices in the Software and this entire statement, including
+the above license grant, this restriction and the following disclaimer,
+must be included in all copies of the Software, in whole or in part, and
+all derivative works of the Software, unless such copies or derivative
+works are solely in the form of machine-executable object code generated by
+a source language processor.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT
+SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE
+FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE,
+ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+DEALINGS IN THE SOFTWARE.
+
+--------------------------------------------------------------------------------
+
+This project includes code from the FlatBuffers project
+
+Copyright 2014 Google Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+--------------------------------------------------------------------------------
+
+This project includes code from the tslib project
+
+Copyright 2015 Microsoft Corporation. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+--------------------------------------------------------------------------------
+
+This project includes code from the jemalloc project
+
+https://github.com/jemalloc/jemalloc
+
+Copyright (C) 2002-2017 Jason Evans <ja...@canonware.com>.
+All rights reserved.
+Copyright (C) 2007-2012 Mozilla Foundation. All rights reserved.
+Copyright (C) 2009-2017 Facebook, Inc. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+1. Redistributions of source code must retain the above copyright notice(s),
+ this list of conditions and the following disclaimer.
+2. Redistributions in binary form must reproduce the above copyright notice(s),
+ this list of conditions and the following disclaimer in the documentation
+ and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
+OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
+EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
+INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
+OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
+ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+--------------------------------------------------------------------------------
+
+This project includes code from the Go project, BSD 3-clause license + PATENTS
+weak patent termination clause
+(https://github.com/golang/go/blob/master/PATENTS).
+
+Copyright (c) 2009 The Go Authors. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+ * Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above
+copyright notice, this list of conditions and the following disclaimer
+in the documentation and/or other materials provided with the
+distribution.
+ * Neither the name of Google Inc. nor the names of its
+contributors may be used to endorse or promote products derived from
+this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+--------------------------------------------------------------------------------
+
+This project includes code from the hs2client
+
+https://github.com/cloudera/hs2client
+
+Copyright 2016 Cloudera Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+--------------------------------------------------------------------------------
+
+The script ci/scripts/util_wait_for_it.sh has the following license
+
+Copyright (c) 2016 Giles Hall
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
+of the Software, and to permit persons to whom the Software is furnished to do
+so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+--------------------------------------------------------------------------------
+
+The script r/configure has the following license (MIT)
+
+Copyright (c) 2017, Jeroen Ooms and Jim Hester
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
+of the Software, and to permit persons to whom the Software is furnished to do
+so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+--------------------------------------------------------------------------------
+
+cpp/src/arrow/util/logging.cc, cpp/src/arrow/util/logging.h and
+cpp/src/arrow/util/logging-test.cc are adapted from
+Ray Project (https://github.com/ray-project/ray) (Apache 2.0).
+
+Copyright (c) 2016 Ray Project (https://github.com/ray-project/ray)
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+--------------------------------------------------------------------------------
+The files cpp/src/arrow/vendored/datetime/date.h, cpp/src/arrow/vendored/datetime/tz.h,
+cpp/src/arrow/vendored/datetime/tz_private.h, cpp/src/arrow/vendored/datetime/ios.h,
+cpp/src/arrow/vendored/datetime/ios.mm,
+cpp/src/arrow/vendored/datetime/tz.cpp are adapted from
+Howard Hinnant's date library (https://github.com/HowardHinnant/date)
+It is licensed under MIT license.
+
+The MIT License (MIT)
+Copyright (c) 2015, 2016, 2017 Howard Hinnant
+Copyright (c) 2016 Adrian Colomitchi
+Copyright (c) 2017 Florian Dang
+Copyright (c) 2017 Paul Thompson
+Copyright (c) 2018 Tomasz Kamiński
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+--------------------------------------------------------------------------------
+
+The file cpp/src/arrow/util/utf8.h includes code adapted from the page
+ https://bjoern.hoehrmann.de/utf-8/decoder/dfa/
+with the following license (MIT)
+
+Copyright (c) 2008-2009 Bjoern Hoehrmann <bj...@hoehrmann.de>
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+--------------------------------------------------------------------------------
+
+The file cpp/src/arrow/vendored/string_view.hpp has the following license
+
+Boost Software License - Version 1.0 - August 17th, 2003
+
+Permission is hereby granted, free of charge, to any person or organization
+obtaining a copy of the software and accompanying documentation covered by
+this license (the "Software") to use, reproduce, display, distribute,
+execute, and transmit the Software, and to prepare derivative works of the
+Software, and to permit third-parties to whom the Software is furnished to
+do so, all subject to the following:
+
+The copyright notices in the Software and this entire statement, including
+the above license grant, this restriction and the following disclaimer,
+must be included in all copies of the Software, in whole or in part, and
+all derivative works of the Software, unless such copies or derivative
+works are solely in the form of machine-executable object code generated by
+a source language processor.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT
+SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE
+FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE,
+ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+DEALINGS IN THE SOFTWARE.
+
+--------------------------------------------------------------------------------
+
+The files in cpp/src/arrow/vendored/xxhash/ have the following license
+(BSD 2-Clause License)
+
+xxHash Library
+Copyright (c) 2012-2014, Yann Collet
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+ list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice, this
+ list of conditions and the following disclaimer in the documentation and/or
+ other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+You can contact the author at :
+- xxHash homepage: http://www.xxhash.com
+- xxHash source repository : https://github.com/Cyan4973/xxHash
+
+--------------------------------------------------------------------------------
+
+The files in cpp/src/arrow/vendored/double-conversion/ have the following license
+(BSD 3-Clause License)
+
+Copyright 2006-2011, the V8 project authors. All rights reserved.
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+ * Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above
+ copyright notice, this list of conditions and the following
+ disclaimer in the documentation and/or other materials provided
+ with the distribution.
+ * Neither the name of Google Inc. nor the names of its
+ contributors may be used to endorse or promote products derived
+ from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+--------------------------------------------------------------------------------
+
+The files in cpp/src/arrow/vendored/uriparser/ have the following license
+(BSD 3-Clause License)
+
+uriparser - RFC 3986 URI parsing library
+
+Copyright (C) 2007, Weijia Song <so...@gmail.com>
+Copyright (C) 2007, Sebastian Pipping <se...@pipping.org>
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+
+ * Redistributions of source code must retain the above
+ copyright notice, this list of conditions and the following
+ disclaimer.
+
+ * Redistributions in binary form must reproduce the above
+ copyright notice, this list of conditions and the following
+ disclaimer in the documentation and/or other materials
+ provided with the distribution.
+
+ * Neither the name of the <ORGANIZATION> nor the names of its
+ contributors may be used to endorse or promote products
+ derived from this software without specific prior written
+ permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
+OF THE POSSIBILITY OF SUCH DAMAGE.
+
+--------------------------------------------------------------------------------
+
+The files under dev/tasks/conda-recipes have the following license
+
+BSD 3-clause license
+Copyright (c) 2015-2018, conda-forge
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+ list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+ this list of conditions and the following disclaimer in the documentation
+ and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its contributors
+ may be used to endorse or promote products derived from this software without
+ specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
+TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+--------------------------------------------------------------------------------
+
+The files in cpp/src/arrow/vendored/utfcpp/ have the following license
+
+Copyright 2006-2018 Nemanja Trifunovic
+
+Permission is hereby granted, free of charge, to any person or organization
+obtaining a copy of the software and accompanying documentation covered by
+this license (the "Software") to use, reproduce, display, distribute,
+execute, and transmit the Software, and to prepare derivative works of the
+Software, and to permit third-parties to whom the Software is furnished to
+do so, all subject to the following:
+
+The copyright notices in the Software and this entire statement, including
+the above license grant, this restriction and the following disclaimer,
+must be included in all copies of the Software, in whole or in part, and
+all derivative works of the Software, unless such copies or derivative
+works are solely in the form of machine-executable object code generated by
+a source language processor.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT
+SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE
+FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE,
+ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+DEALINGS IN THE SOFTWARE.
+
+--------------------------------------------------------------------------------
+
+This project includes code from Apache Kudu.
+
+ * cpp/cmake_modules/CompilerInfo.cmake is based on Kudu's cmake_modules/CompilerInfo.cmake
+
+Copyright: 2016 The Apache Software Foundation.
+Home page: https://kudu.apache.org/
+License: http://www.apache.org/licenses/LICENSE-2.0
+
+--------------------------------------------------------------------------------
+
+This project includes code from Apache Impala (incubating), formerly
+Impala. The Impala code and rights were donated to the ASF as part of the
+Incubator process after the initial code imports into Apache Parquet.
+
+Copyright: 2012 Cloudera, Inc.
+Copyright: 2016 The Apache Software Foundation.
+Home page: http://impala.apache.org/
+License: http://www.apache.org/licenses/LICENSE-2.0
+
+--------------------------------------------------------------------------------
+
+This project includes code from Apache Aurora.
+
+* dev/release/{release,changelog,release-candidate} are based on the scripts from
+ Apache Aurora
+
+Copyright: 2016 The Apache Software Foundation.
+Home page: https://aurora.apache.org/
+License: http://www.apache.org/licenses/LICENSE-2.0
+
+--------------------------------------------------------------------------------
+
+This project includes code from the Google styleguide.
+
+* cpp/build-support/cpplint.py is based on the scripts from the Google styleguide.
+
+Copyright: 2009 Google Inc. All rights reserved.
+Homepage: https://github.com/google/styleguide
+License: 3-clause BSD
+
+--------------------------------------------------------------------------------
+
+This project includes code from Snappy.
+
+* cpp/cmake_modules/{SnappyCMakeLists.txt,SnappyConfig.h} are based on code
+ from Google's Snappy project.
+
+Copyright: 2009 Google Inc. All rights reserved.
+Homepage: https://github.com/google/snappy
+License: 3-clause BSD
+
+--------------------------------------------------------------------------------
+
+This project includes code from the manylinux project.
+
+* python/manylinux1/scripts/{build_python.sh,python-tag-abi-tag.py,
+ requirements.txt} are based on code from the manylinux project.
+
+Copyright: 2016 manylinux
+Homepage: https://github.com/pypa/manylinux
+License: The MIT License (MIT)
+
+--------------------------------------------------------------------------------
+
+This project includes code from the cymove project:
+
+* python/pyarrow/includes/common.pxd includes code from the cymove project
+
+The MIT License (MIT)
+Copyright (c) 2019 Omer Ozarslan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE
+OR OTHER DEALINGS IN THE SOFTWARE.
+
+--------------------------------------------------------------------------------
+
+The projects includes code from the Ursabot project under the dev/archery
+directory.
+
+License: BSD 2-Clause
+
+Copyright 2019 RStudio, Inc.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+ list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+ this list of conditions and the following disclaimer in the documentation
+ and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+--------------------------------------------------------------------------------
+
+This project include code from mingw-w64.
+
+* cpp/src/arrow/util/cpu-info.cc has a polyfill for mingw-w64 < 5
+
+Copyright (c) 2009 - 2013 by the mingw-w64 project
+Homepage: https://mingw-w64.org
+License: Zope Public License (ZPL) Version 2.1.
+
+---------------------------------------------------------------------------------
+
+This project include code from Google's Asylo project.
+
+* cpp/src/arrow/result.h is based on status_or.h
+
+Copyright (c) Copyright 2017 Asylo authors
+Homepage: https://asylo.dev/
+License: Apache 2.0
+
+--------------------------------------------------------------------------------
+
+This project includes code from Google's protobuf project
+
+* cpp/src/arrow/result.h ARROW_ASSIGN_OR_RAISE is based off ASSIGN_OR_RETURN
+* cpp/src/arrow/util/bit_stream_utils.h contains code from wire_format_lite.h
+
+Copyright 2008 Google Inc. All rights reserved.
+Homepage: https://developers.google.com/protocol-buffers/
+License:
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+ * Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above
+copyright notice, this list of conditions and the following disclaimer
+in the documentation and/or other materials provided with the
+distribution.
+ * Neither the name of Google Inc. nor the names of its
+contributors may be used to endorse or promote products derived from
+this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+Code generated by the Protocol Buffer compiler is owned by the owner
+of the input file used when generating it. This code is not
+standalone and requires a support library to be linked with it. This
+support library is itself covered by the above license.
+
+--------------------------------------------------------------------------------
+
+3rdparty dependency LLVM is statically linked in certain binary distributions.
+Additionally some sections of source code have been derived from sources in LLVM
+and have been clearly labeled as such. LLVM has the following license:
+
+==============================================================================
+The LLVM Project is under the Apache License v2.0 with LLVM Exceptions:
+==============================================================================
+
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+
+
+---- LLVM Exceptions to the Apache 2.0 License ----
+
+As an exception, if, as a result of your compiling your source code, portions
+of this Software are embedded into an Object form of such source code, you
+may redistribute such embedded portions in such Object form without complying
+with the conditions of Sections 4(a), 4(b) and 4(d) of the License.
+
+In addition, if you combine or link compiled forms of this Software with
+software that is licensed under the GPLv2 ("Combined Software") and if a
+court of competent jurisdiction determines that the patent provision (Section
+3), the indemnity provision (Section 9) or other Section of the License
+conflicts with the conditions of the GPLv2, you may retroactively and
+prospectively choose to deem waived or otherwise exclude such Section(s) of
+the License, but only in their entirety and only with respect to the Combined
+Software.
+
+==============================================================================
+Software from third parties included in the LLVM Project:
+==============================================================================
+The LLVM Project contains third party software which is under different license
+terms. All such code will be identified clearly using at least one of two
+mechanisms:
+1) It will be in a separate directory tree with its own `LICENSE.txt` or
+ `LICENSE` file at the top containing the specific license and restrictions
+ which apply to that software, or
+2) It will contain specific license and restriction terms at the top of every
+ file.
+
+--------------------------------------------------------------------------------
+
+3rdparty dependency gRPC is statically linked in certain binary
+distributions, like the python wheels. gRPC has the following license:
+
+Copyright 2014 gRPC authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+--------------------------------------------------------------------------------
+
+3rdparty dependency Apache Thrift is statically linked in certain binary
+distributions, like the python wheels. Apache Thrift has the following license:
+
+Apache Thrift
+Copyright (C) 2006 - 2019, The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+--------------------------------------------------------------------------------
+
+3rdparty dependency Apache ORC is statically linked in certain binary
+distributions, like the python wheels. Apache ORC has the following license:
+
+Apache ORC
+Copyright 2013-2019 The Apache Software Foundation
+
+This product includes software developed by The Apache Software
+Foundation (http://www.apache.org/).
+
+This product includes software developed by Hewlett-Packard:
+(c) Copyright [2014-2015] Hewlett-Packard Development Company, L.P
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+--------------------------------------------------------------------------------
+
+3rdparty dependency zstd is statically linked in certain binary
+distributions, like the python wheels. ZSTD has the following license:
+
+BSD License
+
+For Zstandard software
+
+Copyright (c) 2016-present, Facebook, Inc. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+ * Redistributions of source code must retain the above copyright notice, this
+ list of conditions and the following disclaimer.
+
+ * Redistributions in binary form must reproduce the above copyright notice,
+ this list of conditions and the following disclaimer in the documentation
+ and/or other materials provided with the distribution.
+
+ * Neither the name Facebook nor the names of its contributors may be used to
+ endorse or promote products derived from this software without specific
+ prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+--------------------------------------------------------------------------------
+
+3rdparty dependency lz4 is statically linked in certain binary
+distributions, like the python wheels. lz4 has the following license:
+
+LZ4 Library
+Copyright (c) 2011-2016, Yann Collet
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+ list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice, this
+ list of conditions and the following disclaimer in the documentation and/or
+ other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+--------------------------------------------------------------------------------
+
+3rdparty dependency Brotli is statically linked in certain binary
+distributions, like the python wheels. Brotli has the following license:
+
+Copyright (c) 2009, 2010, 2013-2016 by the Brotli Authors.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+
+--------------------------------------------------------------------------------
+
+3rdparty dependency rapidjson is statically linked in certain binary
+distributions, like the python wheels. rapidjson and its dependencies have the
+following licenses:
+
+Tencent is pleased to support the open source community by making RapidJSON
+available.
+
+Copyright (C) 2015 THL A29 Limited, a Tencent company, and Milo Yip.
+All rights reserved.
+
+If you have downloaded a copy of the RapidJSON binary from Tencent, please note
+that the RapidJSON binary is licensed under the MIT License.
+If you have downloaded a copy of the RapidJSON source code from Tencent, please
+note that RapidJSON source code is licensed under the MIT License, except for
+the third-party components listed below which are subject to different license
+terms. Your integration of RapidJSON into your own projects may require
+compliance with the MIT License, as well as the other licenses applicable to
+the third-party components included within RapidJSON. To avoid the problematic
+JSON license in your own projects, it's sufficient to exclude the
+bin/jsonchecker/ directory, as it's the only code under the JSON license.
+A copy of the MIT License is included in this file.
+
+Other dependencies and licenses:
+
+ Open Source Software Licensed Under the BSD License:
+ --------------------------------------------------------------------
+
+ The msinttypes r29
+ Copyright (c) 2006-2013 Alexander Chemeris
+ All rights reserved.
+
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions are met:
+
+ * Redistributions of source code must retain the above copyright notice,
+ this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above copyright notice,
+ this list of conditions and the following disclaimer in the documentation
+ and/or other materials provided with the distribution.
+ * Neither the name of copyright holder nor the names of its contributors
+ may be used to endorse or promote products derived from this software
+ without specific prior written permission.
+
+ THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND ANY
+ EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+ WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+ DISCLAIMED. IN NO EVENT SHALL THE REGENTS AND CONTRIBUTORS BE LIABLE FOR
+ ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+ SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+ CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
+ DAMAGE.
+
+ Terms of the MIT License:
+ --------------------------------------------------------------------
+
+ Permission is hereby granted, free of charge, to any person obtaining a
+ copy of this software and associated documentation files (the "Software"),
+ to deal in the Software without restriction, including without limitation
+ the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ and/or sell copies of the Software, and to permit persons to whom the
+ Software is furnished to do so, subject to the following conditions:
+
+ The above copyright notice and this permission notice shall be included
+ in all copies or substantial portions of the Software.
+
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ DEALINGS IN THE SOFTWARE.
+
+--------------------------------------------------------------------------------
+
+3rdparty dependency snappy is statically linked in certain binary
+distributions, like the python wheels. snappy has the following license:
+
+Copyright 2011, Google Inc.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+ * Redistributions of source code must retain the above copyright notice,
+ this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above copyright notice,
+ this list of conditions and the following disclaimer in the documentation
+ and/or other materials provided with the distribution.
+ * Neither the name of Google Inc. nor the names of its contributors may be
+ used to endorse or promote products derived from this software without
+ specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+===
+
+Some of the benchmark data in testdata/ is licensed differently:
+
+ - fireworks.jpeg is Copyright 2013 Steinar H. Gunderson, and
+ is licensed under the Creative Commons Attribution 3.0 license
+ (CC-BY-3.0). See https://creativecommons.org/licenses/by/3.0/
+ for more information.
+
+ - kppkn.gtb is taken from the Gaviota chess tablebase set, and
+ is licensed under the MIT License. See
+ https://sites.google.com/site/gaviotachessengine/Home/endgame-tablebases-1
+ for more information.
+
+ - paper-100k.pdf is an excerpt (bytes 92160 to 194560) from the paper
+ “Combinatorial Modeling of Chromatin Features Quantitatively Predicts DNA
+ Replication Timing in _Drosophila_” by Federico Comoglio and Renato Paro,
+ which is licensed under the CC-BY license. See
+ http://www.ploscompbiol.org/static/license for more ifnormation.
+
+ - alice29.txt, asyoulik.txt, plrabn12.txt and lcet10.txt are from Project
+ Gutenberg. The first three have expired copyrights and are in the public
+ domain; the latter does not have expired copyright, but is still in the
+ public domain according to the license information
+ (http://www.gutenberg.org/ebooks/53).
+
+--------------------------------------------------------------------------------
+
+3rdparty dependency gflags is statically linked in certain binary
+distributions, like the python wheels. gflags has the following license:
+
+Copyright (c) 2006, Google Inc.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+ * Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above
+copyright notice, this list of conditions and the following disclaimer
+in the documentation and/or other materials provided with the
+distribution.
+ * Neither the name of Google Inc. nor the names of its
+contributors may be used to endorse or promote products derived from
+this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+--------------------------------------------------------------------------------
+
+3rdparty dependency glog is statically linked in certain binary
+distributions, like the python wheels. glog has the following license:
+
+Copyright (c) 2008, Google Inc.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+ * Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above
+copyright notice, this list of conditions and the following disclaimer
+in the documentation and/or other materials provided with the
+distribution.
+ * Neither the name of Google Inc. nor the names of its
+contributors may be used to endorse or promote products derived from
+this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+
+A function gettimeofday in utilities.cc is based on
+
+http://www.google.com/codesearch/p?hl=en#dR3YEbitojA/COPYING&q=GetSystemTimeAsFileTime%20license:bsd
+
+The license of this code is:
+
+Copyright (c) 2003-2008, Jouni Malinen <j...@w1.fi> and contributors
+All Rights Reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+1. Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright
+ notice, this list of conditions and the following disclaimer in the
+ documentation and/or other materials provided with the distribution.
+
+3. Neither the name(s) of the above-listed copyright holder(s) nor the
+ names of its contributors may be used to endorse or promote products
+ derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+--------------------------------------------------------------------------------
+
+3rdparty dependency re2 is statically linked in certain binary
+distributions, like the python wheels. re2 has the following license:
+
+Copyright (c) 2009 The RE2 Authors. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+ * Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above
+ copyright notice, this list of conditions and the following
+ disclaimer in the documentation and/or other materials provided
+ with the distribution.
+ * Neither the name of Google Inc. nor the names of its contributors
+ may be used to endorse or promote products derived from this
+ software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+--------------------------------------------------------------------------------
+
+3rdparty dependency c-ares is statically linked in certain binary
+distributions, like the python wheels. c-ares has the following license:
+
+# c-ares license
+
+Copyright (c) 2007 - 2018, Daniel Stenberg with many contributors, see AUTHORS
+file.
+
+Copyright 1998 by the Massachusetts Institute of Technology.
+
+Permission to use, copy, modify, and distribute this software and its
+documentation for any purpose and without fee is hereby granted, provided that
+the above copyright notice appear in all copies and that both that copyright
+notice and this permission notice appear in supporting documentation, and that
+the name of M.I.T. not be used in advertising or publicity pertaining to
+distribution of the software without specific, written prior permission.
+M.I.T. makes no representations about the suitability of this software for any
+purpose. It is provided "as is" without express or implied warranty.
+
+--------------------------------------------------------------------------------
+
+3rdparty dependency zlib is redistributed as a dynamically linked shared
+library in certain binary distributions, like the python wheels. In the future
+this will likely change to static linkage. zlib has the following license:
+
+zlib.h -- interface of the 'zlib' general purpose compression library
+ version 1.2.11, January 15th, 2017
+
+ Copyright (C) 1995-2017 Jean-loup Gailly and Mark Adler
+
+ This software is provided 'as-is', without any express or implied
+ warranty. In no event will the authors be held liable for any damages
+ arising from the use of this software.
+
+ Permission is granted to anyone to use this software for any purpose,
+ including commercial applications, and to alter it and redistribute it
+ freely, subject to the following restrictions:
+
+ 1. The origin of this software must not be misrepresented; you must not
+ claim that you wrote the original software. If you use this software
+ in a product, an acknowledgment in the product documentation would be
+ appreciated but is not required.
+ 2. Altered source versions must be plainly marked as such, and must not be
+ misrepresented as being the original software.
+ 3. This notice may not be removed or altered from any source distribution.
+
+ Jean-loup Gailly Mark Adler
+ jloup@gzip.org madler@alumni.caltech.edu
+
+--------------------------------------------------------------------------------
+
+3rdparty dependency openssl is redistributed as a dynamically linked shared
+library in certain binary distributions, like the python wheels. openssl
+preceding version 3 has the following license:
+
+ LICENSE ISSUES
+ ==============
+
+ The OpenSSL toolkit stays under a double license, i.e. both the conditions of
+ the OpenSSL License and the original SSLeay license apply to the toolkit.
+ See below for the actual license texts.
+
+ OpenSSL License
+ ---------------
+
+/* ====================================================================
+ * Copyright (c) 1998-2019 The OpenSSL Project. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ *
+ * 3. All advertising materials mentioning features or use of this
+ * software must display the following acknowledgment:
+ * "This product includes software developed by the OpenSSL Project
+ * for use in the OpenSSL Toolkit. (http://www.openssl.org/)"
+ *
+ * 4. The names "OpenSSL Toolkit" and "OpenSSL Project" must not be used to
+ * endorse or promote products derived from this software without
+ * prior written permission. For written permission, please contact
+ * openssl-core@openssl.org.
+ *
+ * 5. Products derived from this software may not be called "OpenSSL"
+ * nor may "OpenSSL" appear in their names without prior written
+ * permission of the OpenSSL Project.
+ *
+ * 6. Redistributions of any form whatsoever must retain the following
+ * acknowledgment:
+ * "This product includes software developed by the OpenSSL Project
+ * for use in the OpenSSL Toolkit (http://www.openssl.org/)"
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY
+ * EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE OpenSSL PROJECT OR
+ * ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+ * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
+ * OF THE POSSIBILITY OF SUCH DAMAGE.
+ * ====================================================================
+ *
+ * This product includes cryptographic software written by Eric Young
+ * (eay@cryptsoft.com). This product includes software written by Tim
+ * Hudson (tjh@cryptsoft.com).
+ *
+ */
+
+ Original SSLeay License
+ -----------------------
+
+/* Copyright (C) 1995-1998 Eric Young (eay@cryptsoft.com)
+ * All rights reserved.
+ *
+ * This package is an SSL implementation written
+ * by Eric Young (eay@cryptsoft.com).
+ * The implementation was written so as to conform with Netscapes SSL.
+ *
+ * This library is free for commercial and non-commercial use as long as
+ * the following conditions are aheared to. The following conditions
+ * apply to all code found in this distribution, be it the RC4, RSA,
+ * lhash, DES, etc., code; not just the SSL code. The SSL documentation
+ * included with this distribution is covered by the same copyright terms
+ * except that the holder is Tim Hudson (tjh@cryptsoft.com).
+ *
+ * Copyright remains Eric Young's, and as such any Copyright notices in
+ * the code are not to be removed.
+ * If this package is used in a product, Eric Young should be given attribution
+ * as the author of the parts of the library used.
+ * This can be in the form of a textual message at program startup or
+ * in documentation (online or textual) provided with the package.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ * 3. All advertising materials mentioning features or use of this software
+ * must display the following acknowledgement:
+ * "This product includes cryptographic software written by
+ * Eric Young (eay@cryptsoft.com)"
+ * The word 'cryptographic' can be left out if the rouines from the library
+ * being used are not cryptographic related :-).
+ * 4. If you include any Windows specific code (or a derivative thereof) from
+ * the apps directory (application code) you must include an acknowledgement:
+ * "This product includes software written by Tim Hudson (tjh@cryptsoft.com)"
+ *
+ * THIS SOFTWARE IS PROVIDED BY ERIC YOUNG ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * The licence and distribution terms for any publically available version or
+ * derivative of this code cannot be changed. i.e. this code cannot simply be
+ * copied and put under another distribution licence
+ * [including the GNU Public Licence.]
+ */
+
+--------------------------------------------------------------------------------
+
+This project includes code from the rtools-backports project.
+
+* ci/scripts/PKGBUILD and ci/scripts/r_windows_build.sh are based on code
+ from the rtools-backports project.
+
+Copyright: Copyright (c) 2013 - 2019, Алексей and Jeroen Ooms.
+All rights reserved.
+Homepage: https://github.com/r-windows/rtools-backports
+License: 3-clause BSD
+
+--------------------------------------------------------------------------------
+
+Some code from pandas has been adapted for the pyarrow codebase. pandas is
+available under the 3-clause BSD license, which follows:
+
+pandas license
+==============
+
+Copyright (c) 2011-2012, Lambda Foundry, Inc. and PyData Development Team
+All rights reserved.
+
+Copyright (c) 2008-2011 AQR Capital Management, LLC
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+ * Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+
+ * Redistributions in binary form must reproduce the above
+ copyright notice, this list of conditions and the following
+ disclaimer in the documentation and/or other materials provided
+ with the distribution.
+
+ * Neither the name of the copyright holder nor the names of any
+ contributors may be used to endorse or promote products derived
+ from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+--------------------------------------------------------------------------------
+
+Some bits from DyND, in particular aspects of the build system, have been
+adapted from libdynd and dynd-python under the terms of the BSD 2-clause
+license
+
+The BSD 2-Clause License
+
+ Copyright (C) 2011-12, Dynamic NDArray Developers
+ All rights reserved.
+
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions are
+ met:
+
+ * Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+
+ * Redistributions in binary form must reproduce the above
+ copyright notice, this list of conditions and the following
+ disclaimer in the documentation and/or other materials provided
+ with the distribution.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+Dynamic NDArray Developers list:
+
+ * Mark Wiebe
+ * Continuum Analytics
+
+--------------------------------------------------------------------------------
+
+Some source code from Ibis (https://github.com/cloudera/ibis) has been adapted
+for PyArrow. Ibis is released under the Apache License, Version 2.0.
+
+--------------------------------------------------------------------------------
+
+This project includes code from the autobrew project.
+
+* r/tools/autobrew and dev/tasks/homebrew-formulae/autobrew/apache-arrow.rb
+ are based on code from the autobrew project.
+
+Copyright (c) 2019, Jeroen Ooms
+License: MIT
+Homepage: https://github.com/jeroen/autobrew
+
+--------------------------------------------------------------------------------
+
+dev/tasks/homebrew-formulae/apache-arrow.rb has the following license:
+
+BSD 2-Clause License
+
+Copyright (c) 2009-present, Homebrew contributors
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+ list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice,
+ this list of conditions and the following disclaimer in the documentation
+ and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+----------------------------------------------------------------------
+
+cpp/src/arrow/vendored/base64.cpp has the following license
+
+ZLIB License
+
+Copyright (C) 2004-2017 René Nyffenegger
+
+This source code is provided 'as-is', without any express or implied
+warranty. In no event will the author be held liable for any damages arising
+from the use of this software.
+
+Permission is granted to anyone to use this software for any purpose, including
+commercial applications, and to alter it and redistribute it freely, subject to
+the following restrictions:
+
+1. The origin of this source code must not be misrepresented; you must not
+ claim that you wrote the original source code. If you use this source code
+ in a product, an acknowledgment in the product documentation would be
+ appreciated but is not required.
+
+2. Altered source versions must be plainly marked as such, and must not be
+ misrepresented as being the original source code.
+
+3. This notice may not be removed or altered from any source distribution.
+
+René Nyffenegger rene.nyffenegger@adp-gmbh.ch
+
+--------------------------------------------------------------------------------
+
+The file cpp/src/arrow/vendored/optional.hpp has the following license
+
+Boost Software License - Version 1.0 - August 17th, 2003
+
+Permission is hereby granted, free of charge, to any person or organization
+obtaining a copy of the software and accompanying documentation covered by
+this license (the "Software") to use, reproduce, display, distribute,
+execute, and transmit the Software, and to prepare derivative works of the
+Software, and to permit third-parties to whom the Software is furnished to
+do so, all subject to the following:
+
+The copyright notices in the Software and this entire statement, including
+the above license grant, this restriction and the following disclaimer,
+must be included in all copies of the Software, in whole or in part, and
+all derivative works of the Software, unless such copies or derivative
+works are solely in the form of machine-executable object code generated by
+a source language processor.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT
+SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE
+FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE,
+ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+DEALINGS IN THE SOFTWARE.
+
+--------------------------------------------------------------------------------
+
+This project includes code from Folly.
+
+ * cpp/src/arrow/vendored/ProducerConsumerQueue.h
+
+is based on Folly's
+
+ * folly/Portability.h
+ * folly/lang/Align.h
+ * folly/ProducerConsumerQueue.h
+
+Copyright: Copyright (c) Facebook, Inc. and its affiliates.
+Home page: https://github.com/facebook/folly
+License: http://www.apache.org/licenses/LICENSE-2.0
+
+--------------------------------------------------------------------------------
+
+The file cpp/src/arrow/vendored/musl/strptime.c has the following license
+
+Copyright © 2005-2020 Rich Felker, et al.
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+--------------------------------------------------------------------------------
+
+The file cpp/cmake_modules/BuildUtils.cmake contains code from
+
+https://gist.github.com/cristianadam/ef920342939a89fae3e8a85ca9459b49
+
+which is made available under the MIT license
+
+Copyright (c) 2019 Cristian Adam
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+--------------------------------------------------------------------------------
+
+The files in cpp/src/arrow/vendored/portable-snippets/ contain code from
+
+https://github.com/nemequ/portable-snippets
+
+and have the following copyright notice:
+
+Each source file contains a preamble explaining the license situation
+for that file, which takes priority over this file. With the
+exception of some code pulled in from other repositories (such as
+µnit, an MIT-licensed project which is used for testing), the code is
+public domain, released using the CC0 1.0 Universal dedication (*).
+
+(*) https://creativecommons.org/publicdomain/zero/1.0/legalcode
+
+--------------------------------------------------------------------------------
+
+The files in cpp/src/arrow/vendored/fast_float/ contain code from
+
+https://github.com/lemire/fast_float
+
+which is made available under the Apache License 2.0.
+
+--------------------------------------------------------------------------------
+
+The file python/pyarrow/vendored/docscrape.py contains code from
+
+https://github.com/numpy/numpydoc/
+
+which is made available under the BSD 2-clause license.
+
+--------------------------------------------------------------------------------
+
+The file python/pyarrow/vendored/version.py contains code from
+
+https://github.com/pypa/packaging/
+
+which is made available under both the Apache license v2.0 and the
+BSD 2-clause license.
+
+--------------------------------------------------------------------------------
+
+The files in cpp/src/arrow/vendored/pcg contain code from
+
+https://github.com/imneme/pcg-cpp
+
+and have the following copyright notice:
+
+Copyright 2014-2019 Melissa O'Neill <on...@pcg-random.org>,
+ and the PCG Project contributors.
+
+SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+Licensed under the Apache License, Version 2.0 (provided in
+LICENSE-APACHE.txt and at http://www.apache.org/licenses/LICENSE-2.0)
+or under the MIT license (provided in LICENSE-MIT.txt and at
+http://opensource.org/licenses/MIT), at your option. This file may not
+be copied, modified, or distributed except according to those terms.
+
+Distributed on an "AS IS" BASIS, WITHOUT WARRANTY OF ANY KIND, either
+express or implied. See your chosen license for details.
+
+--------------------------------------------------------------------------------
+r/R/dplyr-count-tally.R (some portions)
+
+Some portions of this file are derived from code from
+
+https://github.com/tidyverse/dplyr/
+
+which is made available under the MIT license
+
+Copyright (c) 2013-2019 RStudio and others.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the “Software”), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+--------------------------------------------------------------------------------
+
+The file src/arrow/util/io_util.cc contains code from the CPython project
+which is made available under the Python Software Foundation License Version 2.
+
+--------------------------------------------------------------------------------
+
+3rdparty dependency opentelemetry-cpp is statically linked in certain binary
+distributions. opentelemetry-cpp is made available under the Apache License 2.0.
+
+Copyright The OpenTelemetry Authors
+SPDX-License-Identifier: Apache-2.0
+
+--------------------------------------------------------------------------------
+
+ci/conan/ is based on code from Conan Package and Dependency Manager.
+
+Copyright (c) 2019 Conan.io
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+--------------------------------------------------------------------------------
+
+3rdparty dependency UCX is redistributed as a dynamically linked shared
+library in certain binary distributions. UCX has the following license:
+
+Copyright (c) 2014-2015 UT-Battelle, LLC. All rights reserved.
+Copyright (C) 2014-2020 Mellanox Technologies Ltd. All rights reserved.
+Copyright (C) 2014-2015 The University of Houston System. All rights reserved.
+Copyright (C) 2015 The University of Tennessee and The University
+ of Tennessee Research Foundation. All rights reserved.
+Copyright (C) 2016-2020 ARM Ltd. All rights reserved.
+Copyright (c) 2016 Los Alamos National Security, LLC. All rights reserved.
+Copyright (C) 2016-2020 Advanced Micro Devices, Inc. All rights reserved.
+Copyright (C) 2019 UChicago Argonne, LLC. All rights reserved.
+Copyright (c) 2018-2020 NVIDIA CORPORATION. All rights reserved.
+Copyright (C) 2020 Huawei Technologies Co., Ltd. All rights reserved.
+Copyright (C) 2016-2020 Stony Brook University. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+
+1. Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+2. Redistributions in binary form must reproduce the above copyright
+notice, this list of conditions and the following disclaimer in the
+documentation and/or other materials provided with the distribution.
+3. Neither the name of the copyright holder nor the names of its
+contributors may be used to endorse or promote products derived from
+this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
+TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
+NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..4ba6cff
--- /dev/null
+++ b/README.md
@@ -0,0 +1,24 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# ADBC: Arrow Database Connectivity
+
+[](https://github.com/apache/arrow-adbc/blob/master/LICENSE.txt)
+
+EXPERIMENTAL. Please see the [mailing list discussion](https://lists.apache.org/thread/gnz1kz2rj3rb8rh8qz7l0mv8lvzq254w).
diff --git a/adbc.h b/adbc.h
new file mode 100644
index 0000000..1aa7b47
--- /dev/null
+++ b/adbc.h
@@ -0,0 +1,661 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <stddef.h>
+#include <stdint.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#ifndef ARROW_C_DATA_INTERFACE
+#define ARROW_C_DATA_INTERFACE
+
+#define ARROW_FLAG_DICTIONARY_ORDERED 1
+#define ARROW_FLAG_NULLABLE 2
+#define ARROW_FLAG_MAP_KEYS_SORTED 4
+
+struct ArrowSchema {
+ // Array type description
+ const char* format;
+ const char* name;
+ const char* metadata;
+ int64_t flags;
+ int64_t n_children;
+ struct ArrowSchema** children;
+ struct ArrowSchema* dictionary;
+
+ // Release callback
+ void (*release)(struct ArrowSchema*);
+ // Opaque producer-specific data
+ void* private_data;
+};
+
+struct ArrowArray {
+ // Array data description
+ int64_t length;
+ int64_t null_count;
+ int64_t offset;
+ int64_t n_buffers;
+ int64_t n_children;
+ const void** buffers;
+ struct ArrowArray** children;
+ struct ArrowArray* dictionary;
+
+ // Release callback
+ void (*release)(struct ArrowArray*);
+ // Opaque producer-specific data
+ void* private_data;
+};
+
+#endif // ARROW_C_DATA_INTERFACE
+
+// EXPERIMENTAL: C stream interface
+
+#ifndef ARROW_C_STREAM_INTERFACE
+#define ARROW_C_STREAM_INTERFACE
+
+struct ArrowArrayStream {
+ // Callback to get the stream type
+ // (will be the same for all arrays in the stream).
+ //
+ // Return value: 0 if successful, an `errno`-compatible error code otherwise.
+ //
+ // If successful, the ArrowSchema must be released independently from the stream.
+ int (*get_schema)(struct ArrowArrayStream*, struct ArrowSchema* out);
+
+ // Callback to get the next array
+ // (if no error and the array is released, the stream has ended)
+ //
+ // Return value: 0 if successful, an `errno`-compatible error code otherwise.
+ //
+ // If successful, the ArrowArray must be released independently from the stream.
+ int (*get_next)(struct ArrowArrayStream*, struct ArrowArray* out);
+
+ // Callback to get optional detailed error information.
+ // This must only be called if the last stream operation failed
+ // with a non-0 return code.
+ //
+ // Return value: pointer to a null-terminated character array describing
+ // the last error, or NULL if no description is available.
+ //
+ // The returned pointer is only valid until the next operation on this stream
+ // (including release).
+ const char* (*get_last_error)(struct ArrowArrayStream*);
+
+ // Release callback: release the stream's own resources.
+ // Note that arrays returned by `get_next` must be individually released.
+ void (*release)(struct ArrowArrayStream*);
+
+ // Opaque producer-specific data
+ void* private_data;
+};
+
+#endif // ARROW_C_STREAM_INTERFACE
+
+#ifndef ADBC
+#define ADBC
+
+/// \file ADBC: Arrow DataBase connectivity (client API)
+///
+/// Implemented by libadbc.so (provided by Arrow/C++), which in turn
+/// dynamically loads the appropriate database driver.
+///
+/// EXPERIMENTAL. Interface subject to change.
+
+/// \page object-model Object Model
+///
+/// Except where noted, objects are not thread-safe and clients should
+/// take care to serialize accesses to methods.
+
+// Forward declarations
+struct AdbcDriver;
+struct AdbcStatement;
+
+/// \defgroup adbc-error-handling Error handling primitives.
+/// ADBC uses integer error codes to signal errors. To provide more
+/// detail about errors, functions may also return an AdbcError via an
+/// optional out parameter, which can be inspected. If provided, it is
+/// the responsibility of the caller to zero-initialize the AdbcError
+/// value.
+///
+/// @{
+
+/// Error codes for operations that may fail.
+typedef uint8_t AdbcStatusCode;
+
+/// No error.
+#define ADBC_STATUS_OK 0
+/// An unknown error occurred.
+#define ADBC_STATUS_UNKNOWN 1
+/// The operation is not implemented.
+#define ADBC_STATUS_NOT_IMPLEMENTED 2
+/// An operation was attempted on an uninitialized object.
+#define ADBC_STATUS_UNINITIALIZED 3
+/// The arguments are invalid.
+#define ADBC_STATUS_INVALID_ARGUMENT 4
+/// The object is in an invalid state for the given operation.
+#define ADBC_STATUS_INTERNAL 5
+/// An I/O error occurred.
+#define ADBC_STATUS_IO 6
+
+/// \brief A detailed error message for an operation.
+struct AdbcError {
+ /// \brief The error message.
+ char* message;
+
+ /// \brief The associated driver (used by the driver manager to help
+ /// track state).
+ AdbcDriver* private_driver;
+
+ // TODO: go back to just inlining 'release' here? And remove the
+ // global AdbcErrorRelease? It would be slightly inconsistent (and
+ // would make the struct impossible to extend) but would be easier
+ // to manage between the driver manager and driver.
+};
+
+/// \brief Destroy an error message.
+void AdbcErrorRelease(struct AdbcError* error);
+
+/// \brief Get a human-readable description of a status code.
+const char* AdbcStatusCodeMessage(AdbcStatusCode code);
+
+/// }@
+
+/// \defgroup adbc-database Database initialization.
+/// Clients first initialize a database, then connect to the database
+/// (below). For client-server databases, one of these steps may be a
+/// no-op; for in-memory or otherwise non-client-server databases,
+/// this gives the implementation a place to initialize and own any
+/// common connection state.
+/// @{
+
+/// \brief A set of database options.
+struct AdbcDatabaseOptions {
+ /// \brief A driver-specific database string.
+ ///
+ /// Should be in ODBC-style format ("Key1=Value1;Key2=Value2").
+ const char* target;
+
+ /// \brief The associated driver. Required if using the driver
+ /// manager; not required if directly calling into a driver.
+ AdbcDriver* driver;
+};
+
+/// \brief An instance of a database.
+///
+/// Must be kept alive as long as any connections exist.
+struct AdbcDatabase {
+ /// \brief Opaque implementation-defined state.
+ /// This field is NULLPTR iff the connection is unintialized/freed.
+ void* private_data;
+ /// \brief The associated driver (used by the driver manager to help
+ /// track state).
+ AdbcDriver* private_driver;
+};
+
+/// \brief Initialize a new database.
+AdbcStatusCode AdbcDatabaseInit(const struct AdbcDatabaseOptions* options,
+ struct AdbcDatabase* out, struct AdbcError* error);
+
+/// \brief Destroy this database. No connections may exist.
+/// \param[in] database The database to release.
+/// \param[out] error An optional location to return an error
+/// message if necessary.
+AdbcStatusCode AdbcDatabaseRelease(struct AdbcDatabase* database,
+ struct AdbcError* error);
+
+/// }@
+
+/// \defgroup adbc-connection Connection establishment.
+/// @{
+
+/// \brief A set of connection options.
+struct AdbcConnectionOptions {
+ /// \brief The database to connect to.
+ struct AdbcDatabase* database;
+
+ /// \brief A driver-specific connection string.
+ ///
+ /// Should be in ODBC-style format ("Key1=Value1;Key2=Value2").
+ const char* target;
+};
+
+/// \brief An active database connection.
+///
+/// Provides methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// Connections are not thread-safe and clients should take care to
+/// serialize accesses to a connection.
+struct AdbcConnection {
+ /// \brief Opaque implementation-defined state.
+ /// This field is NULLPTR iff the connection is unintialized/freed.
+ void* private_data;
+ /// \brief The associated driver (used by the driver manager to help
+ /// track state).
+ AdbcDriver* private_driver;
+};
+
+/// \brief Create a new connection to a database.
+AdbcStatusCode AdbcConnectionInit(const struct AdbcConnectionOptions* options,
+ struct AdbcConnection* connection,
+ struct AdbcError* error);
+
+/// \brief Destroy this connection.
+/// \param[in] connection The connection to release.
+/// \param[out] error An optional location to return an error
+/// message if necessary.
+AdbcStatusCode AdbcConnectionRelease(struct AdbcConnection* connection,
+ struct AdbcError* error);
+
+/// \defgroup adbc-connection-sql SQL Semantics
+/// Functions for executing SQL queries, or querying SQL-related
+/// metadata. Drivers are not required to support both SQL and
+/// Substrait semantics. If they do, it may be via converting
+/// between representations internally.
+/// @{
+
+/// \brief Execute a one-shot query.
+///
+/// For queries expected to be executed repeatedly, create a
+/// prepared statement.
+///
+/// \param[in] connection The database connection.
+/// \param[in] query The query to execute.
+/// \param[in,out] statement The result set. Allocate with AdbcStatementInit.
+/// \param[out] error Error details, if an error occurs.
+AdbcStatusCode AdbcConnectionSqlExecute(struct AdbcConnection* connection,
+ const char* query,
+ struct AdbcStatement* statement,
+ struct AdbcError* error);
+
+/// \brief Prepare a query to be executed multiple times.
+///
+/// TODO: this should return AdbcPreparedStatement to disaggregate
+/// preparation and execution
+AdbcStatusCode AdbcConnectionSqlPrepare(struct AdbcConnection* connection,
+ const char* query,
+ struct AdbcStatement* statement,
+ struct AdbcError* error);
+
+/// }@
+
+/// \defgroup adbc-connection-substrait Substrait Semantics
+/// Functions for executing Substrait plans, or querying
+/// Substrait-related metadata. Drivers are not required to support
+/// both SQL and Substrait semantics. If they do, it may be via
+/// converting between representations internally.
+/// @{
+
+// TODO: not yet defined
+
+/// }@
+
+/// \defgroup adbc-connection-partition Partitioned Results
+/// Some databases may internally partition the results. These
+/// partitions are exposed to clients who may wish to integrate them
+/// with a threaded or distributed execution model, where partitions
+/// can be divided among threads or machines for processing.
+///
+/// Drivers are not required to support partitioning.
+///
+/// Partitions are not ordered. If the result set is sorted,
+/// implementations should return a single partition.
+///
+/// @{
+
+/// \brief Construct a statement for a partition of a query. The
+/// statement can then be read independently.
+///
+/// A partition can be retrieved from AdbcStatementGetPartitionDesc.
+AdbcStatusCode AdbcConnectionDeserializePartitionDesc(struct AdbcConnection* connection,
+ const uint8_t* serialized_partition,
+ size_t serialized_length,
+ struct AdbcStatement* statement,
+ struct AdbcError* error);
+
+/// }@
+
+/// \defgroup adbc-connection-metadata Metadata
+/// Functions for retrieving metadata about the database.
+///
+/// Generally, these functions return an AdbcStatement that can be evaluated to
+/// get the metadata as Arrow data. The returned metadata has an expected
+/// schema given in the function docstring. Schema fields are nullable unless
+/// otherwise marked.
+///
+/// Some functions accept a "search pattern" argument, which is a string that
+/// can contain the special character "%" to match zero or more characters, or
+/// "_" to match exactly one character. (See the documentation of
+/// DatabaseMetaData in JDBC or "Pattern Value Arguments" in the ODBC
+/// documentation.)
+///
+/// TODO: escaping in search patterns?
+///
+/// @{
+
+/// \brief Get a list of catalogs in the database.
+///
+/// The result is an Arrow dataset with the following schema:
+///
+/// Field Name | Field Type
+/// ---------------|--------------
+/// catalog_name | utf8 not null
+///
+/// \param[in] connection The database connection.
+/// \param[out] statement The result set.
+/// \param[out] error Error details, if an error occurs.
+AdbcStatusCode AdbcConnectionGetCatalogs(struct AdbcConnection* connection,
+ struct AdbcStatement* statement,
+ struct AdbcError* error);
+
+/// \brief Get a list of schemas in the database.
+///
+/// The result is an Arrow dataset with the following schema:
+///
+/// Field Name | Field Type
+/// ---------------|--------------
+/// catalog_name | utf8
+/// db_schema_name | utf8 not null
+///
+/// \param[in] connection The database connection.
+/// \param[out] statement The result set.
+/// \param[out] error Error details, if an error occurs.
+AdbcStatusCode AdbcConnectionGetDbSchemas(struct AdbcConnection* connection,
+ struct AdbcStatement* statement,
+ struct AdbcError* error);
+
+/// \brief Get a list of table types in the database.
+///
+/// The result is an Arrow dataset with the following schema:
+///
+/// Field Name | Field Type
+/// ---------------|--------------
+/// table_type | utf8 not null
+///
+/// \param[in] connection The database connection.
+/// \param[out] statement The result set.
+/// \param[out] error Error details, if an error occurs.
+AdbcStatusCode AdbcConnectionGetTableTypes(struct AdbcConnection* connection,
+ struct AdbcStatement* statement,
+ struct AdbcError* error);
+
+/// \brief Get a list of tables matching the given criteria.
+///
+/// The result is an Arrow dataset with the following schema:
+///
+/// Field Name | Field Type
+/// ---------------|--------------
+/// catalog_name | utf8
+/// db_schema_name | utf8
+/// table_name | utf8 not null
+/// table_type | utf8 not null
+///
+/// \param[in] connection The database connection.
+/// \param[in] catalog Only show tables in the given catalog. If NULL, do not
+/// filter by catalog. If an empty string, only show tables without a
+/// catalog.
+/// \param[in] db_schema Only show tables in the given database schema. If
+/// NULL, do not filter by database schema. If an empty string, only show
+/// tables without a database schema. May be a search pattern (see section
+/// documentation).
+/// \param[in] table_name Only show tables with the given name. If NULL, do not
+/// filter by name. May be a search pattern (see section documentation).
+/// \param[in] table_types Only show tables matching one of the given table
+/// types. If NULL, show tables of any type. Valid table types can be fetched
+/// from get_table_types.
+/// \param[out] statement The result set.
+/// \param[out] error Error details, if an error occurs.
+AdbcStatusCode AdbcConnectionGetTables(struct AdbcConnection* connection,
+ const char* catalog, const char* db_schema,
+ const char* table_name, const char** table_types,
+ struct AdbcStatement* statement,
+ struct AdbcError* error);
+/// }@
+
+/// }@
+
+/// \defgroup adbc-statement Managing statements.
+/// Applications should first initialize and configure a statement with
+/// AdbcStatementInit and the AdbcStatementSetOption functions, then use the
+/// statement with a function like AdbcConnectionSqlExecute.
+/// @{
+
+/// \brief An instance of a database query, from parameters set before
+/// execution to the result of execution.
+///
+/// Statements are not thread-safe and clients should take care to
+/// serialize access.
+struct AdbcStatement {
+ /// \brief Opaque implementation-defined state.
+ /// This field is NULLPTR iff the connection is unintialized/freed.
+ void* private_data;
+
+ /// \brief The associated driver (used by the driver manager to help
+ /// track state).
+ AdbcDriver* private_driver;
+};
+
+/// \brief Create a new statement for a given connection.
+AdbcStatusCode AdbcStatementInit(struct AdbcConnection* connection,
+ struct AdbcStatement* statement,
+ struct AdbcError* error);
+
+/// \brief Set an integer option on a statement.
+AdbcStatusCode AdbcStatementSetOptionInt64(struct AdbcStatement* statement,
+ struct AdbcError* error);
+
+/// \brief Destroy a statement.
+/// \param[in] statement The statement to release.
+/// \param[out] error An optional location to return an error
+/// message if necessary.
+AdbcStatusCode AdbcStatementRelease(struct AdbcStatement* statement,
+ struct AdbcError* error);
+
+/// \brief Bind parameter values for parameterized statements.
+/// \param[in] statement The statement to bind to.
+/// \param[in] values The values to bind. The driver will not call the
+/// release callback itself, although it may not do this until the
+/// statement is released.
+/// \param[in] schema The schema of the values to bind.
+/// \param[out] error An optional location to return an error message
+/// if necessary.
+AdbcStatusCode AdbcStatementBind(struct AdbcStatement* statement,
+ struct ArrowArray* values, struct ArrowSchema* schema,
+ struct AdbcError* error);
+
+/// \brief Execute a statement.
+///
+/// Not called for one-shot queries (e.g. AdbcConnectionSqlExecute).
+AdbcStatusCode AdbcStatementExecute(struct AdbcStatement* statement,
+ struct AdbcError* error);
+
+/// \brief Read the result of a statement.
+///
+/// This method can be called only once per execution of the
+/// statement. It may not be called if any of the partitioning methods
+/// have been called (see below).
+///
+/// \return out A stream of Arrow data. The stream itself must be
+/// released before the statement is released.
+AdbcStatusCode AdbcStatementGetStream(struct AdbcStatement* statement,
+ struct ArrowArrayStream* out,
+ struct AdbcError* error);
+
+// TODO: methods to get a particular result set from the statement,
+// etc. especially for prepared statements with parameter batches
+
+/// \defgroup adbc-statement-partition Partitioned Results
+/// Some backends may internally partition the results. These
+/// partitions are exposed to clients who may wish to integrate them
+/// with a threaded or distributed execution model, where partitions
+/// can be divided among threads or machines. Partitions are exposed
+/// as an iterator.
+///
+/// Drivers are not required to support partitioning. In this case,
+/// num_partitions will return 0. They are required to support
+/// AdbcStatementGetStream.
+///
+/// If any of the partitioning methods are called,
+/// AdbcStatementGetStream may not be called, and vice versa.
+///
+/// @{
+
+/// \brief Get the length of the serialized descriptor for the current
+/// partition.
+///
+/// This method must be called first, before calling other partitioning
+/// methods. This method may block and perform I/O.
+/// \param[in] statement The statement.
+/// \param[out] length The length of the serialized partition, or 0 if there
+/// are no more partitions.
+/// \param[out] error An optional location to return an error message if
+/// necessary.
+AdbcStatusCode AdbcStatementGetPartitionDescSize(struct AdbcStatement* statement,
+ size_t* length, struct AdbcError* error);
+
+/// \brief Get the serialized descriptor for the current partition, and advance
+/// the iterator.
+///
+/// This method may block and perform I/O.
+///
+/// A partition can be turned back into a statement via
+/// AdbcConnectionDeserializePartitionDesc. Effectively, this means AdbcStatement
+/// is similar to arrow::flight::FlightInfo in Flight/Flight SQL and
+/// get_partitions is similar to getting the arrow::flight::Ticket.
+///
+/// \param[in] statement The statement.
+/// \param[out] partition_desc A caller-allocated buffer, to which the
+/// serialized partition will be written. The length to allocate can be
+/// queried with AdbcStatementGetPartitionDescSize.
+/// \param[out] error An optional location to return an error message if
+/// necessary.
+AdbcStatusCode AdbcStatementGetPartitionDesc(struct AdbcStatement* statement,
+ uint8_t* partition_desc,
+ struct AdbcError* error);
+
+/// }@
+
+/// }@
+
+/// \defgroup adbc-driver Driver initialization.
+/// @{
+
+/// \brief An instance of an initialized database driver.
+///
+/// This provides a common interface for implementation-specific
+/// driver initialization routines. Drivers should populate this
+/// struct, and applications can call ADBC functions through this
+/// struct, without worrying about multiple definitions of the same
+/// symbol.
+struct AdbcDriver {
+ /// \brief Opaque implementation-defined state.
+ /// This field is NULLPTR if the driver is unintialized/freed (but
+ /// it need not have a value even if the driver is initialized).
+ void* private_data;
+ // TODO: DriverRelease
+
+ void (*ErrorRelease)(struct AdbcError*);
+ const char* (*StatusCodeMessage)(AdbcStatusCode);
+
+ AdbcStatusCode (*DatabaseInit)(const struct AdbcDatabaseOptions*, struct AdbcDatabase*,
+ struct AdbcError*);
+ AdbcStatusCode (*DatabaseRelease)(struct AdbcDatabase*, struct AdbcError*);
+
+ AdbcStatusCode (*ConnectionInit)(const struct AdbcConnectionOptions*,
+ struct AdbcConnection*, struct AdbcError*);
+ AdbcStatusCode (*ConnectionRelease)(struct AdbcConnection*, struct AdbcError*);
+ AdbcStatusCode (*ConnectionSqlExecute)(struct AdbcConnection*, const char*,
+ struct AdbcStatement*, struct AdbcError*);
+ AdbcStatusCode (*ConnectionSqlPrepare)(struct AdbcConnection*, const char*,
+ struct AdbcStatement*, struct AdbcError*);
+ AdbcStatusCode (*ConnectionDeserializePartitionDesc)(struct AdbcConnection*,
+ const uint8_t*, size_t,
+ struct AdbcStatement*,
+ struct AdbcError*);
+
+ AdbcStatusCode (*ConnectionGetCatalogs)(struct AdbcConnection*, struct AdbcStatement*,
+ struct AdbcError*);
+ AdbcStatusCode (*ConnectionGetDbSchemas)(struct AdbcConnection*, struct AdbcStatement*,
+ struct AdbcError*);
+ AdbcStatusCode (*ConnectionGetTableTypes)(struct AdbcConnection*, struct AdbcStatement*,
+ struct AdbcError*);
+ AdbcStatusCode (*ConnectionGetTables)(struct AdbcConnection*, const char*, const char*,
+ const char*, const char**, struct AdbcStatement*,
+ struct AdbcError*);
+
+ AdbcStatusCode (*StatementInit)(struct AdbcConnection*, struct AdbcStatement*,
+ struct AdbcError*);
+ AdbcStatusCode (*StatementSetOptionInt64)(struct AdbcStatement*, struct AdbcError*);
+ AdbcStatusCode (*StatementRelease)(struct AdbcStatement*, struct AdbcError*);
+ AdbcStatusCode (*StatementBind)(struct AdbcStatement*, struct ArrowArray*,
+ struct ArrowSchema*, struct AdbcError*);
+ AdbcStatusCode (*StatementExecute)(struct AdbcStatement*, struct AdbcError*);
+ AdbcStatusCode (*StatementGetStream)(struct AdbcStatement*, struct ArrowArrayStream*,
+ struct AdbcError*);
+ AdbcStatusCode (*StatementGetPartitionDescSize)(struct AdbcStatement*, size_t*,
+ struct AdbcError*);
+ AdbcStatusCode (*StatementGetPartitionDesc)(struct AdbcStatement*, uint8_t*,
+ struct AdbcError*);
+ // Do not edit fields. New fields can only be appended to the end.
+};
+
+/// \brief Common entry point for drivers via the driver manager
+/// (which uses dlopen(3)/LoadLibrary). The driver manager is told
+/// to load a library and call a function of this type to load the
+/// driver.
+///
+/// \param[in] count The number of entries to initialize. Provides
+/// backwards compatibility if the struct definition is changed.
+/// \param[out] driver The table of function pointers to initialize.
+/// \param[out] initialized How much of the table was actually
+/// initialized (can be less than count).
+typedef AdbcStatusCode (*AdbcDriverInitFunc)(size_t count, struct AdbcDriver* driver,
+ size_t* initialized);
+// TODO: how best to report errors here?
+// TODO: use sizeof() instead of count, or version the
+// struct/entrypoint instead?
+
+// For use with count
+#define ADBC_VERSION_0_0_1 21
+
+/// }@
+
+/// \page typical-usage Typical Usage Patterns
+/// (TODO: describe request sequences)
+
+/// \page decoder-ring Decoder Ring
+///
+/// ADBC - Flight SQL - JDBC - ODBC
+///
+/// AdbcConnection - FlightClient - Connection - Connection handle
+///
+/// AdbcStatement - FlightInfo - Statement - Statement handle
+///
+/// ArrowArrayStream - FlightStream (Java)/RecordBatchReader (C++) -
+/// ResultSet - Statement handle
+
+/// \page compatibility Backwards and Forwards Compatibility
+
+#endif // ADBC
+
+#ifdef __cplusplus
+}
+#endif
diff --git a/adbc_driver_manager/CMakeLists.txt b/adbc_driver_manager/CMakeLists.txt
new file mode 100644
index 0000000..ed41526
--- /dev/null
+++ b/adbc_driver_manager/CMakeLists.txt
@@ -0,0 +1,47 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+cmake_minimum_required(VERSION 3.10)
+
+set(ARROW_VERSION "9.0.0-SNAPSHOT")
+set(ARROW_BASE_VERSION "9.0.0")
+set(ARROW_VERSION_MAJOR "9")
+set(ARROW_VERSION_MINOR "0")
+set(ARROW_VERSION_PATCH "0")
+
+math(EXPR ARROW_SO_VERSION "${ARROW_VERSION_MAJOR} * 100 + ${ARROW_VERSION_MINOR}")
+set(ARROW_FULL_SO_VERSION "${ARROW_SO_VERSION}.${ARROW_VERSION_PATCH}.0")
+
+project(adbc_driver_manager VERSION "${ARROW_BASE_VERSION}" LANGUAGES CXX)
+
+# Common CMake utilities
+get_filename_component(REPOSITORY_ROOT ${CMAKE_SOURCE_DIR} DIRECTORY)
+list(APPEND CMAKE_MODULE_PATH "${REPOSITORY_ROOT}/cmake_modules/")
+include(BuildUtils)
+include(DefineOptions)
+
+# Driver manager library
+add_arrow_lib(adbc_driver_manager
+ SOURCES
+ adbc_driver_manager.cc)
+include_directories(adbc_driver_manager SYSTEM
+ PRIVATE ${REPOSITORY_ROOT})
+
+# Tests
+
+validate_config()
+config_summary_message()
diff --git a/adbc_driver_manager/adbc_driver_manager.cc b/adbc_driver_manager/adbc_driver_manager.cc
new file mode 100644
index 0000000..998c672
--- /dev/null
+++ b/adbc_driver_manager/adbc_driver_manager.cc
@@ -0,0 +1,248 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "adbc_driver_manager.h"
+
+#include <dlfcn.h>
+#include <algorithm>
+#include <string>
+#include <unordered_map>
+
+namespace {
+std::unordered_map<std::string, std::string> ParseConnectionString(
+ const std::string& target) {
+ // TODO: this does not properly implement the ODBC connection string format.
+ std::unordered_map<std::string, std::string> option_pairs;
+ size_t cur = 0;
+
+ while (cur < target.size()) {
+ auto divider = target.find('=', cur);
+ if (divider == std::string::npos) break;
+
+ std::string key = target.substr(cur, divider - cur);
+ cur = divider + 1;
+ auto end = target.find(';', cur);
+ if (end == std::string::npos) {
+ option_pairs.insert({std::move(key), target.substr(cur)});
+ break;
+ } else {
+ option_pairs.insert({std::string(key), target.substr(cur, end - cur)});
+ cur = end + 1;
+ }
+ }
+ return option_pairs;
+}
+
+// Default stubs
+AdbcStatusCode ConnectionSqlPrepare(struct AdbcConnection*, const char*,
+ struct AdbcStatement*, struct AdbcError* error) {
+ return ADBC_STATUS_NOT_IMPLEMENTED;
+}
+
+AdbcStatusCode StatementBind(struct AdbcStatement*, struct ArrowArray*,
+ struct ArrowSchema*, struct AdbcError* error) {
+ return ADBC_STATUS_NOT_IMPLEMENTED;
+}
+
+AdbcStatusCode StatementExecute(struct AdbcStatement*, struct AdbcError* error) {
+ return ADBC_STATUS_NOT_IMPLEMENTED;
+}
+} // namespace
+
+#define FILL_DEFAULT(DRIVER, STUB) \
+ if (!DRIVER->STUB) { \
+ DRIVER->STUB = &STUB; \
+ }
+
+// Direct implementations of API methods
+
+void AdbcErrorRelease(struct AdbcError* error) {
+ if (!error->message) return;
+ // TODO: assert
+ error->private_driver->ErrorRelease(error);
+}
+
+AdbcStatusCode AdbcDatabaseInit(const struct AdbcDatabaseOptions* options,
+ struct AdbcDatabase* out, struct AdbcError* error) {
+ if (!options->driver) {
+ // TODO: set error
+ return ADBC_STATUS_INVALID_ARGUMENT;
+ }
+ auto status = options->driver->DatabaseInit(options, out, error);
+ out->private_driver = options->driver;
+ return status;
+}
+
+AdbcStatusCode AdbcDatabaseRelease(struct AdbcDatabase* database,
+ struct AdbcError* error) {
+ if (!database->private_driver) {
+ return ADBC_STATUS_UNINITIALIZED;
+ }
+ auto status = database->private_driver->DatabaseRelease(database, error);
+ database->private_driver = nullptr;
+ return status;
+}
+
+AdbcStatusCode AdbcConnectionInit(const struct AdbcConnectionOptions* options,
+ struct AdbcConnection* out, struct AdbcError* error) {
+ if (!options->database->private_driver) {
+ // TODO: set error
+ return ADBC_STATUS_INVALID_ARGUMENT;
+ }
+ auto status = options->database->private_driver->ConnectionInit(options, out, error);
+ out->private_driver = options->database->private_driver;
+ return status;
+}
+
+AdbcStatusCode AdbcConnectionRelease(struct AdbcConnection* connection,
+ struct AdbcError* error) {
+ if (!connection->private_driver) {
+ return ADBC_STATUS_UNINITIALIZED;
+ }
+ auto status = connection->private_driver->ConnectionRelease(connection, error);
+ connection->private_driver = nullptr;
+ return status;
+}
+
+AdbcStatusCode AdbcConnectionSqlExecute(struct AdbcConnection* connection,
+ const char* query,
+ struct AdbcStatement* statement,
+ struct AdbcError* error) {
+ if (!connection->private_driver) {
+ return ADBC_STATUS_UNINITIALIZED;
+ }
+ return connection->private_driver->ConnectionSqlExecute(connection, query, statement,
+ error);
+}
+
+AdbcStatusCode AdbcConnectionSqlPrepare(struct AdbcConnection* connection,
+ const char* query,
+ struct AdbcStatement* statement,
+ struct AdbcError* error) {
+ if (!connection->private_driver) {
+ return ADBC_STATUS_UNINITIALIZED;
+ }
+ return connection->private_driver->ConnectionSqlPrepare(connection, query, statement,
+ error);
+}
+
+AdbcStatusCode AdbcStatementInit(struct AdbcConnection* connection,
+ struct AdbcStatement* statement,
+ struct AdbcError* error) {
+ if (!connection->private_driver) {
+ // TODO: set error
+ return ADBC_STATUS_INVALID_ARGUMENT;
+ }
+ auto status = connection->private_driver->StatementInit(connection, statement, error);
+ statement->private_driver = connection->private_driver;
+ return status;
+}
+
+AdbcStatusCode AdbcStatementRelease(struct AdbcStatement* statement,
+ struct AdbcError* error) {
+ if (!statement->private_driver) {
+ return ADBC_STATUS_UNINITIALIZED;
+ }
+ auto status = statement->private_driver->StatementRelease(statement, error);
+ statement->private_driver = nullptr;
+ return status;
+}
+
+AdbcStatusCode AdbcStatementBind(struct AdbcStatement* statement,
+ struct ArrowArray* values, struct ArrowSchema* schema,
+ struct AdbcError* error) {
+ if (!statement->private_driver) {
+ return ADBC_STATUS_UNINITIALIZED;
+ }
+ return statement->private_driver->StatementBind(statement, values, schema, error);
+}
+
+AdbcStatusCode AdbcStatementExecute(struct AdbcStatement* statement,
+ struct AdbcError* error) {
+ if (!statement->private_driver) {
+ return ADBC_STATUS_UNINITIALIZED;
+ }
+ return statement->private_driver->StatementExecute(statement, error);
+}
+
+AdbcStatusCode AdbcStatementGetStream(struct AdbcStatement* statement,
+ struct ArrowArrayStream* out,
+ struct AdbcError* error) {
+ if (!statement->private_driver) {
+ return ADBC_STATUS_UNINITIALIZED;
+ }
+ return statement->private_driver->StatementGetStream(statement, out, error);
+}
+
+const char* AdbcStatusCodeMessage(AdbcStatusCode code) {
+#define STRINGIFY(s) #s
+#define STRINGIFY_VALUE(s) STRINGIFY(s)
+#define CASE(CONSTANT) \
+ case CONSTANT: \
+ return STRINGIFY(CONSTANT) " (" STRINGIFY_VALUE(CONSTANT) ")";
+
+ switch (code) {
+ CASE(ADBC_STATUS_OK)
+ CASE(ADBC_STATUS_UNKNOWN)
+ CASE(ADBC_STATUS_NOT_IMPLEMENTED)
+ CASE(ADBC_STATUS_UNINITIALIZED)
+ CASE(ADBC_STATUS_INVALID_ARGUMENT)
+ CASE(ADBC_STATUS_INTERNAL)
+ CASE(ADBC_STATUS_IO)
+ default:
+ return "(invalid code)";
+ }
+#undef CASE
+#undef STRINGIFY_VALUE
+#undef STRINGIFY
+}
+
+AdbcStatusCode AdbcLoadDriver(const char* connection, size_t count,
+ struct AdbcDriver* driver, size_t* initialized) {
+ auto params = ParseConnectionString(connection);
+
+ auto driver_str = params.find("Driver");
+ if (driver_str == params.end()) {
+ return ADBC_STATUS_INVALID_ARGUMENT;
+ }
+
+ auto entrypoint_str = params.find("Entrypoint");
+ if (entrypoint_str == params.end()) {
+ return ADBC_STATUS_INVALID_ARGUMENT;
+ }
+
+ void* handle = dlopen(driver_str->second.c_str(), RTLD_NOW | RTLD_LOCAL);
+ if (!handle) {
+ return ADBC_STATUS_UNKNOWN;
+ }
+
+ void* load_handle = dlsym(handle, entrypoint_str->second.c_str());
+ auto* load = reinterpret_cast<AdbcDriverInitFunc>(load_handle);
+ if (!load) {
+ return ADBC_STATUS_INTERNAL;
+ }
+
+ auto result = load(count, driver, initialized);
+ if (result != ADBC_STATUS_OK) {
+ return result;
+ }
+
+ FILL_DEFAULT(driver, ConnectionSqlPrepare);
+ FILL_DEFAULT(driver, StatementBind);
+ FILL_DEFAULT(driver, StatementExecute);
+ return ADBC_STATUS_OK;
+}
diff --git a/adbc_driver_manager/adbc_driver_manager.h b/adbc_driver_manager/adbc_driver_manager.h
new file mode 100644
index 0000000..074cb52
--- /dev/null
+++ b/adbc_driver_manager/adbc_driver_manager.h
@@ -0,0 +1,50 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <adbc.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#ifndef ADBC_DRIVER_MANAGER_H
+#define ADBC_DRIVER_MANAGER_H
+
+/// \brief Common entry point for drivers via the driver manager.
+///
+/// The driver manager can fill in default implementations of some
+/// ADBC functions for drivers. Drivers must implement a minimum level
+/// of functionality for this to be possible, however, and some
+/// functions must be implemented by the driver.
+///
+/// \param[in] connection The driver to initialize. Should be in
+/// ODBC-style format ("Key1=Value1;Key2=Value2").
+/// \param[in] count The number of entries to initialize. Provides
+/// backwards compatibility if the struct definition is changed.
+/// \param[out] driver The table of function pointers to initialize.
+/// \param[out] initialized How much of the table was actually
+/// initialized (can be less than count).
+AdbcStatusCode AdbcLoadDriver(const char* connection, size_t count,
+ struct AdbcDriver* driver, size_t* initialized);
+
+#endif // ADBC_DRIVER_MANAGER_H
+
+#ifdef __cplusplus
+}
+#endif
diff --git a/adbc_driver_manager/adbc_driver_manager_test.cc b/adbc_driver_manager/adbc_driver_manager_test.cc
new file mode 100644
index 0000000..cc77958
--- /dev/null
+++ b/adbc_driver_manager/adbc_driver_manager_test.cc
@@ -0,0 +1,103 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Test the driver, but using the driver manager's stubs instead of
+// the function pointer table.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "adbc/adbc.h"
+#include "adbc/adbc_driver_manager.h"
+#include "adbc/test_sqlite_internal.h"
+#include "adbc/test_util.h"
+#include "arrow/c/bridge.h"
+#include "arrow/record_batch.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/matchers.h"
+
+namespace adbc {
+
+using arrow::PointeesEqual;
+
+TEST(AdbcDriverManager, Basics) {
+ AdbcDatabase database;
+ AdbcConnection connection;
+ AdbcError error = {};
+
+ AdbcDriver driver;
+ ASSERT_NO_FATAL_FAILURE(InitSqlite(&driver));
+
+ {
+ AdbcDatabaseOptions options;
+ std::memset(&options, 0, sizeof(options));
+ options.driver = &driver;
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ AdbcDatabaseInit(&options, &database, &error));
+ ASSERT_NE(database.private_data, nullptr);
+ }
+
+ {
+ AdbcConnectionOptions options;
+ std::memset(&options, 0, sizeof(options));
+ options.database = &database;
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ AdbcConnectionInit(&options, &connection, &error));
+ ASSERT_NE(connection.private_data, nullptr);
+ }
+
+ {
+ std::string query = "SELECT 1";
+ AdbcStatement statement;
+ std::memset(&statement, 0, sizeof(statement));
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ AdbcStatementInit(&connection, &statement, &error));
+ ADBC_ASSERT_OK_WITH_ERROR(
+ &driver, error,
+ AdbcConnectionSqlExecute(&connection, query.c_str(), &statement, &error));
+
+ arrow::RecordBatchVector batches;
+ ArrowArrayStream stream;
+
+ ADBC_ASSERT_OK(AdbcStatementGetStream(&statement, &stream, &error));
+ ASSERT_OK_AND_ASSIGN(auto reader, arrow::ImportRecordBatchReader(&stream));
+
+ auto schema = reader->schema();
+ while (true) {
+ ASSERT_OK_AND_ASSIGN(auto batch, reader->Next());
+ if (!batch) break;
+ batches.push_back(std::move(batch));
+ }
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error, AdbcStatementRelease(&statement, &error));
+
+ arrow::AssertSchemaEqual(*schema,
+ *arrow::schema({arrow::field("1", arrow::int64())}));
+ EXPECT_THAT(batches,
+ ::testing::UnorderedPointwise(
+ PointeesEqual(), {
+ arrow::RecordBatchFromJSON(schema, "[[1]]"),
+ }));
+ }
+
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error, AdbcConnectionRelease(&connection, &error));
+ ASSERT_EQ(connection.private_data, nullptr);
+
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error, AdbcDatabaseRelease(&database, &error));
+ ASSERT_EQ(database.private_data, nullptr);
+}
+
+} // namespace adbc
diff --git a/cmake_modules/BuildUtils.cmake b/cmake_modules/BuildUtils.cmake
new file mode 100644
index 0000000..c153f3a
--- /dev/null
+++ b/cmake_modules/BuildUtils.cmake
@@ -0,0 +1,920 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Common path suffixes to be searched by find_library or find_path.
+# Windows artifacts may be found under "<root>/Library", so
+# search there as well.
+set(ARROW_LIBRARY_PATH_SUFFIXES
+ "${CMAKE_LIBRARY_ARCHITECTURE}"
+ "lib/${CMAKE_LIBRARY_ARCHITECTURE}"
+ "lib64"
+ "lib32"
+ "lib"
+ "bin"
+ "Library"
+ "Library/lib"
+ "Library/bin")
+set(ARROW_INCLUDE_PATH_SUFFIXES "include" "Library" "Library/include")
+
+function(add_thirdparty_lib LIB_NAME LIB_TYPE LIB)
+ set(options)
+ set(one_value_args)
+ set(multi_value_args DEPS INCLUDE_DIRECTORIES)
+ cmake_parse_arguments(ARG
+ "${options}"
+ "${one_value_args}"
+ "${multi_value_args}"
+ ${ARGN})
+ if(ARG_UNPARSED_ARGUMENTS)
+ message(SEND_ERROR "Error: unrecognized arguments: ${ARG_UNPARSED_ARGUMENTS}")
+ endif()
+
+ add_library(${LIB_NAME} ${LIB_TYPE} IMPORTED)
+ if(${LIB_TYPE} STREQUAL "STATIC")
+ set_target_properties(${LIB_NAME} PROPERTIES IMPORTED_LOCATION "${LIB}")
+ message(STATUS "Added static library dependency ${LIB_NAME}: ${LIB}")
+ else()
+ if(WIN32)
+ # Mark the ".lib" location as part of a Windows DLL
+ set_target_properties(${LIB_NAME} PROPERTIES IMPORTED_IMPLIB "${LIB}")
+ else()
+ set_target_properties(${LIB_NAME} PROPERTIES IMPORTED_LOCATION "${LIB}")
+ endif()
+ message(STATUS "Added shared library dependency ${LIB_NAME}: ${LIB}")
+ endif()
+ if(ARG_DEPS)
+ set_target_properties(${LIB_NAME} PROPERTIES INTERFACE_LINK_LIBRARIES "${ARG_DEPS}")
+ endif()
+ if(ARG_INCLUDE_DIRECTORIES)
+ set_target_properties(${LIB_NAME} PROPERTIES INTERFACE_INCLUDE_DIRECTORIES
+ "${ARG_INCLUDE_DIRECTORIES}")
+ endif()
+endfunction()
+
+function(REUSE_PRECOMPILED_HEADER_LIB TARGET_NAME LIB_NAME)
+ if(ARROW_USE_PRECOMPILED_HEADERS)
+ target_precompile_headers(${TARGET_NAME} REUSE_FROM ${LIB_NAME})
+ endif()
+endfunction()
+
+# Based on MIT-licensed
+# https://gist.github.com/cristianadam/ef920342939a89fae3e8a85ca9459b49
+function(create_merged_static_lib output_target)
+ set(options)
+ set(one_value_args NAME ROOT)
+ set(multi_value_args TO_MERGE)
+ cmake_parse_arguments(ARG
+ "${options}"
+ "${one_value_args}"
+ "${multi_value_args}"
+ ${ARGN})
+ if(ARG_UNPARSED_ARGUMENTS)
+ message(SEND_ERROR "Error: unrecognized arguments: ${ARG_UNPARSED_ARGUMENTS}")
+ endif()
+
+ set(output_lib_path
+ ${BUILD_OUTPUT_ROOT_DIRECTORY}${CMAKE_STATIC_LIBRARY_PREFIX}${ARG_NAME}${CMAKE_STATIC_LIBRARY_SUFFIX}
+ )
+
+ set(all_library_paths $<TARGET_FILE:${ARG_ROOT}>)
+ foreach(lib ${ARG_TO_MERGE})
+ list(APPEND all_library_paths $<TARGET_FILE:${lib}>)
+ endforeach()
+
+ if(APPLE)
+ set(BUNDLE_COMMAND "libtool" "-no_warning_for_no_symbols" "-static" "-o"
+ ${output_lib_path} ${all_library_paths})
+ elseif(CMAKE_CXX_COMPILER_ID MATCHES "^(Clang|GNU|Intel)$")
+ set(ar_script_path ${CMAKE_BINARY_DIR}/${ARG_NAME}.ar)
+
+ file(WRITE ${ar_script_path}.in "CREATE ${output_lib_path}\n")
+ file(APPEND ${ar_script_path}.in "ADDLIB $<TARGET_FILE:${ARG_ROOT}>\n")
+
+ foreach(lib ${ARG_TO_MERGE})
+ file(APPEND ${ar_script_path}.in "ADDLIB $<TARGET_FILE:${lib}>\n")
+ endforeach()
+
+ file(APPEND ${ar_script_path}.in "SAVE\nEND\n")
+ file(GENERATE
+ OUTPUT ${ar_script_path}
+ INPUT ${ar_script_path}.in)
+ set(ar_tool ${CMAKE_AR})
+
+ if(CMAKE_INTERPROCEDURAL_OPTIMIZATION)
+ set(ar_tool ${CMAKE_CXX_COMPILER_AR})
+ endif()
+
+ set(BUNDLE_COMMAND ${ar_tool} -M < ${ar_script_path})
+
+ elseif(MSVC)
+ if(NOT CMAKE_LIBTOOL)
+ find_program(lib_tool lib HINTS "${CMAKE_CXX_COMPILER}/..")
+ if("${lib_tool}" STREQUAL "lib_tool-NOTFOUND")
+ message(FATAL_ERROR "Cannot locate libtool to bundle libraries")
+ endif()
+ else()
+ set(${lib_tool} ${CMAKE_LIBTOOL})
+ endif()
+ set(BUNDLE_TOOL ${lib_tool})
+ set(BUNDLE_COMMAND ${BUNDLE_TOOL} /NOLOGO /OUT:${output_lib_path}
+ ${all_library_paths})
+ else()
+ message(FATAL_ERROR "Unknown bundle scenario!")
+ endif()
+
+ add_custom_command(COMMAND ${BUNDLE_COMMAND}
+ OUTPUT ${output_lib_path}
+ COMMENT "Bundling ${output_lib_path}"
+ VERBATIM)
+
+ message(STATUS "Creating bundled static library target ${output_target} at ${output_lib_path}"
+ )
+
+ add_custom_target(${output_target} ALL DEPENDS ${output_lib_path})
+ add_dependencies(${output_target} ${ARG_ROOT} ${ARG_TO_MERGE})
+ install(FILES ${output_lib_path} DESTINATION ${CMAKE_INSTALL_LIBDIR})
+endfunction()
+
+# \arg OUTPUTS list to append built targets to
+function(ADD_ARROW_LIB LIB_NAME)
+ set(options)
+ set(one_value_args
+ BUILD_SHARED
+ BUILD_STATIC
+ CMAKE_PACKAGE_NAME
+ PKG_CONFIG_NAME
+ SHARED_LINK_FLAGS
+ PRECOMPILED_HEADER_LIB)
+ set(multi_value_args
+ SOURCES
+ PRECOMPILED_HEADERS
+ OUTPUTS
+ STATIC_LINK_LIBS
+ SHARED_LINK_LIBS
+ SHARED_PRIVATE_LINK_LIBS
+ EXTRA_INCLUDES
+ PRIVATE_INCLUDES
+ DEPENDENCIES
+ SHARED_INSTALL_INTERFACE_LIBS
+ STATIC_INSTALL_INTERFACE_LIBS
+ OUTPUT_PATH)
+ cmake_parse_arguments(ARG
+ "${options}"
+ "${one_value_args}"
+ "${multi_value_args}"
+ ${ARGN})
+ if(ARG_UNPARSED_ARGUMENTS)
+ message(SEND_ERROR "Error: unrecognized arguments: ${ARG_UNPARSED_ARGUMENTS}")
+ endif()
+
+ if(ARG_OUTPUTS)
+ set(${ARG_OUTPUTS})
+ endif()
+
+ # Allow overriding ARROW_BUILD_SHARED and ARROW_BUILD_STATIC
+ if(DEFINED ARG_BUILD_SHARED)
+ set(BUILD_SHARED ${ARG_BUILD_SHARED})
+ else()
+ set(BUILD_SHARED ${ARROW_BUILD_SHARED})
+ endif()
+ if(DEFINED ARG_BUILD_STATIC)
+ set(BUILD_STATIC ${ARG_BUILD_STATIC})
+ else()
+ set(BUILD_STATIC ${ARROW_BUILD_STATIC})
+ endif()
+ if(ARG_OUTPUT_PATH)
+ set(OUTPUT_PATH ${ARG_OUTPUT_PATH})
+ else()
+ set(OUTPUT_PATH ${BUILD_OUTPUT_ROOT_DIRECTORY})
+ endif()
+
+ if(WIN32
+ OR (CMAKE_GENERATOR STREQUAL Xcode)
+ OR CMAKE_VERSION VERSION_LESS 3.12)
+ # We need to compile C++ separately for each library kind (shared and static)
+ # because of dllexport declarations on Windows.
+ # The Xcode generator doesn't reliably work with Xcode as target names are not
+ # guessed correctly.
+ # We can't use target for object library with CMake 3.11 or earlier.
+ # See also: Object Libraries:
+ # https://cmake.org/cmake/help/latest/command/add_library.html#object-libraries
+ set(USE_OBJLIB OFF)
+ else()
+ set(USE_OBJLIB ON)
+ endif()
+
+ if(USE_OBJLIB)
+ # Generate a single "objlib" from all C++ modules and link
+ # that "objlib" into each library kind, to avoid compiling twice
+ add_library(${LIB_NAME}_objlib OBJECT ${ARG_SOURCES})
+ # Necessary to make static linking into other shared libraries work properly
+ set_property(TARGET ${LIB_NAME}_objlib PROPERTY POSITION_INDEPENDENT_CODE 1)
+ if(ARG_DEPENDENCIES)
+ add_dependencies(${LIB_NAME}_objlib ${ARG_DEPENDENCIES})
+ endif()
+ if(ARG_PRECOMPILED_HEADER_LIB)
+ reuse_precompiled_header_lib(${LIB_NAME}_objlib ${ARG_PRECOMPILED_HEADER_LIB})
+ endif()
+ if(ARG_PRECOMPILED_HEADERS AND ARROW_USE_PRECOMPILED_HEADERS)
+ target_precompile_headers(${LIB_NAME}_objlib PRIVATE ${ARG_PRECOMPILED_HEADERS})
+ endif()
+ set(LIB_DEPS $<TARGET_OBJECTS:${LIB_NAME}_objlib>)
+ set(LIB_INCLUDES)
+ set(EXTRA_DEPS)
+
+ if(ARG_OUTPUTS)
+ list(APPEND ${ARG_OUTPUTS} ${LIB_NAME}_objlib)
+ endif()
+
+ if(ARG_EXTRA_INCLUDES)
+ target_include_directories(${LIB_NAME}_objlib SYSTEM PUBLIC ${ARG_EXTRA_INCLUDES})
+ endif()
+ if(ARG_PRIVATE_INCLUDES)
+ target_include_directories(${LIB_NAME}_objlib PRIVATE ${ARG_PRIVATE_INCLUDES})
+ endif()
+ target_link_libraries(${LIB_NAME}_objlib
+ PRIVATE ${ARG_SHARED_LINK_LIBS} ${ARG_SHARED_PRIVATE_LINK_LIBS}
+ ${ARG_STATIC_LINK_LIBS})
+ else()
+ # Prepare arguments for separate compilation of static and shared libs below
+ # TODO: add PCH directives
+ set(LIB_DEPS ${ARG_SOURCES})
+ set(EXTRA_DEPS ${ARG_DEPENDENCIES})
+
+ if(ARG_EXTRA_INCLUDES)
+ set(LIB_INCLUDES ${ARG_EXTRA_INCLUDES})
+ endif()
+ endif()
+
+ set(RUNTIME_INSTALL_DIR bin)
+
+ if(BUILD_SHARED)
+ add_library(${LIB_NAME}_shared SHARED ${LIB_DEPS})
+ if(EXTRA_DEPS)
+ add_dependencies(${LIB_NAME}_shared ${EXTRA_DEPS})
+ endif()
+
+ if(ARG_PRECOMPILED_HEADER_LIB)
+ reuse_precompiled_header_lib(${LIB_NAME}_shared ${ARG_PRECOMPILED_HEADER_LIB})
+ endif()
+
+ if(ARG_OUTPUTS)
+ list(APPEND ${ARG_OUTPUTS} ${LIB_NAME}_shared)
+ endif()
+
+ if(LIB_INCLUDES)
+ target_include_directories(${LIB_NAME}_shared SYSTEM PUBLIC ${ARG_EXTRA_INCLUDES})
+ endif()
+
+ if(ARG_PRIVATE_INCLUDES)
+ target_include_directories(${LIB_NAME}_shared PRIVATE ${ARG_PRIVATE_INCLUDES})
+ endif()
+
+ # On iOS, specifying -undefined conflicts with enabling bitcode
+ if(APPLE
+ AND NOT IOS
+ AND NOT DEFINED ENV{EMSCRIPTEN})
+ # On OS X, you can avoid linking at library load time and instead
+ # expecting that the symbols have been loaded separately. This happens
+ # with libpython* where there can be conflicts between system Python and
+ # the Python from a thirdparty distribution
+ #
+ # When running with the Emscripten Compiler, we need not worry about
+ # python, and the Emscripten Compiler does not support this option.
+ set(ARG_SHARED_LINK_FLAGS "-undefined dynamic_lookup ${ARG_SHARED_LINK_FLAGS}")
+ endif()
+
+ set_target_properties(${LIB_NAME}_shared
+ PROPERTIES LIBRARY_OUTPUT_DIRECTORY "${OUTPUT_PATH}"
+ RUNTIME_OUTPUT_DIRECTORY "${OUTPUT_PATH}"
+ PDB_OUTPUT_DIRECTORY "${OUTPUT_PATH}"
+ LINK_FLAGS "${ARG_SHARED_LINK_FLAGS}"
+ OUTPUT_NAME ${LIB_NAME}
+ VERSION "${ARROW_FULL_SO_VERSION}"
+ SOVERSION "${ARROW_SO_VERSION}")
+
+ target_link_libraries(${LIB_NAME}_shared
+ LINK_PUBLIC
+ "$<BUILD_INTERFACE:${ARG_SHARED_LINK_LIBS}>"
+ "$<INSTALL_INTERFACE:${ARG_SHARED_INSTALL_INTERFACE_LIBS}>"
+ LINK_PRIVATE
+ ${ARG_SHARED_PRIVATE_LINK_LIBS})
+
+ if(USE_OBJLIB)
+ # Ensure that dependencies are built before compilation of objects in
+ # object library, rather than only before the final link step
+ foreach(SHARED_LINK_LIB ${ARG_SHARED_LINK_LIBS})
+ if(TARGET ${SHARED_LINK_LIB})
+ add_dependencies(${LIB_NAME}_objlib ${SHARED_LINK_LIB})
+ endif()
+ endforeach()
+ endif()
+
+ if(ARROW_RPATH_ORIGIN)
+ if(APPLE)
+ set(_lib_install_rpath "@loader_path")
+ else()
+ set(_lib_install_rpath "\$ORIGIN")
+ endif()
+ set_target_properties(${LIB_NAME}_shared PROPERTIES INSTALL_RPATH
+ ${_lib_install_rpath})
+ endif()
+
+ if(APPLE)
+ if(ARROW_INSTALL_NAME_RPATH)
+ set(_lib_install_name "@rpath")
+ else()
+ set(_lib_install_name "${CMAKE_INSTALL_PREFIX}/${CMAKE_INSTALL_LIBDIR}")
+ endif()
+ set_target_properties(${LIB_NAME}_shared
+ PROPERTIES BUILD_WITH_INSTALL_RPATH ON INSTALL_NAME_DIR
+ "${_lib_install_name}")
+ endif()
+
+ install(TARGETS ${LIB_NAME}_shared ${INSTALL_IS_OPTIONAL}
+ EXPORT ${LIB_NAME}_targets
+ RUNTIME DESTINATION ${RUNTIME_INSTALL_DIR}
+ LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
+ ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR}
+ INCLUDES
+ DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})
+ endif()
+
+ if(BUILD_STATIC)
+ add_library(${LIB_NAME}_static STATIC ${LIB_DEPS})
+ if(EXTRA_DEPS)
+ add_dependencies(${LIB_NAME}_static ${EXTRA_DEPS})
+ endif()
+
+ if(ARG_PRECOMPILED_HEADER_LIB)
+ reuse_precompiled_header_lib(${LIB_NAME}_static ${ARG_PRECOMPILED_HEADER_LIB})
+ endif()
+
+ if(ARG_OUTPUTS)
+ list(APPEND ${ARG_OUTPUTS} ${LIB_NAME}_static)
+ endif()
+
+ if(LIB_INCLUDES)
+ target_include_directories(${LIB_NAME}_static SYSTEM PUBLIC ${ARG_EXTRA_INCLUDES})
+ endif()
+
+ if(ARG_PRIVATE_INCLUDES)
+ target_include_directories(${LIB_NAME}_static PRIVATE ${ARG_PRIVATE_INCLUDES})
+ endif()
+
+ if(MSVC_TOOLCHAIN)
+ set(LIB_NAME_STATIC ${LIB_NAME}_static)
+ else()
+ set(LIB_NAME_STATIC ${LIB_NAME})
+ endif()
+
+ if(WIN32)
+ target_compile_definitions(${LIB_NAME}_static PUBLIC ARROW_STATIC)
+ target_compile_definitions(${LIB_NAME}_static PUBLIC ARROW_FLIGHT_STATIC)
+ endif()
+
+ set_target_properties(${LIB_NAME}_static
+ PROPERTIES LIBRARY_OUTPUT_DIRECTORY "${OUTPUT_PATH}"
+ OUTPUT_NAME ${LIB_NAME_STATIC})
+
+ if(ARG_STATIC_INSTALL_INTERFACE_LIBS)
+ target_link_libraries(${LIB_NAME}_static LINK_PUBLIC
+ "$<INSTALL_INTERFACE:${ARG_STATIC_INSTALL_INTERFACE_LIBS}>")
+ endif()
+
+ if(ARG_STATIC_LINK_LIBS)
+ target_link_libraries(${LIB_NAME}_static LINK_PRIVATE
+ "$<BUILD_INTERFACE:${ARG_STATIC_LINK_LIBS}>")
+ if(USE_OBJLIB)
+ # Ensure that dependencies are built before compilation of objects in
+ # object library, rather than only before the final link step
+ foreach(STATIC_LINK_LIB ${ARG_STATIC_LINK_LIBS})
+ if(TARGET ${STATIC_LINK_LIB})
+ add_dependencies(${LIB_NAME}_objlib ${STATIC_LINK_LIB})
+ endif()
+ endforeach()
+ endif()
+ endif()
+
+ install(TARGETS ${LIB_NAME}_static ${INSTALL_IS_OPTIONAL}
+ EXPORT ${LIB_NAME}_targets
+ RUNTIME DESTINATION ${RUNTIME_INSTALL_DIR}
+ LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
+ ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR}
+ INCLUDES
+ DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})
+ endif()
+
+ if(ARG_CMAKE_PACKAGE_NAME)
+ arrow_install_cmake_find_module("${ARG_CMAKE_PACKAGE_NAME}")
+
+ set(TARGETS_CMAKE "${ARG_CMAKE_PACKAGE_NAME}Targets.cmake")
+ install(EXPORT ${LIB_NAME}_targets
+ FILE "${TARGETS_CMAKE}"
+ DESTINATION "${ARROW_CMAKE_INSTALL_DIR}")
+
+ set(CONFIG_CMAKE "${ARG_CMAKE_PACKAGE_NAME}Config.cmake")
+ set(BUILT_CONFIG_CMAKE "${CMAKE_CURRENT_BINARY_DIR}/${CONFIG_CMAKE}")
+ configure_package_config_file("${CONFIG_CMAKE}.in" "${BUILT_CONFIG_CMAKE}"
+ INSTALL_DESTINATION "${ARROW_CMAKE_INSTALL_DIR}")
+ install(FILES "${BUILT_CONFIG_CMAKE}" DESTINATION "${ARROW_CMAKE_INSTALL_DIR}")
+
+ set(CONFIG_VERSION_CMAKE "${ARG_CMAKE_PACKAGE_NAME}ConfigVersion.cmake")
+ set(BUILT_CONFIG_VERSION_CMAKE "${CMAKE_CURRENT_BINARY_DIR}/${CONFIG_VERSION_CMAKE}")
+ write_basic_package_version_file(
+ "${BUILT_CONFIG_VERSION_CMAKE}"
+ VERSION ${${PROJECT_NAME}_VERSION}
+ COMPATIBILITY AnyNewerVersion)
+ install(FILES "${BUILT_CONFIG_VERSION_CMAKE}"
+ DESTINATION "${ARROW_CMAKE_INSTALL_DIR}")
+ endif()
+
+ if(ARG_PKG_CONFIG_NAME)
+ arrow_add_pkg_config("${ARG_PKG_CONFIG_NAME}")
+ endif()
+
+ # Modify variable in calling scope
+ if(ARG_OUTPUTS)
+ set(${ARG_OUTPUTS}
+ ${${ARG_OUTPUTS}}
+ PARENT_SCOPE)
+ endif()
+endfunction()
+
+#
+# Benchmarking
+#
+# Add a new micro benchmark, with or without an executable that should be built.
+# If benchmarks are enabled then they will be run along side unit tests with ctest.
+# 'make benchmark' and 'make unittest' to build/run only benchmark or unittests,
+# respectively.
+#
+# REL_BENCHMARK_NAME is the name of the benchmark app. It may be a single component
+# (e.g. monotime-benchmark) or contain additional components (e.g.
+# net/net_util-benchmark). Either way, the last component must be a globally
+# unique name.
+
+# The benchmark will registered as unit test with ctest with a label
+# of 'benchmark'.
+#
+# Arguments after the test name will be passed to set_tests_properties().
+#
+# \arg PREFIX a string to append to the name of the benchmark executable. For
+# example, if you have src/arrow/foo/bar-benchmark.cc, then PREFIX "foo" will
+# create test executable foo-bar-benchmark
+# \arg LABELS the benchmark label or labels to assign the unit tests to. By
+# default, benchmarks will go in the "benchmark" group. Custom targets for the
+# group names must exist
+function(ADD_BENCHMARK REL_BENCHMARK_NAME)
+ set(options)
+ set(one_value_args)
+ set(multi_value_args
+ EXTRA_LINK_LIBS
+ STATIC_LINK_LIBS
+ DEPENDENCIES
+ SOURCES
+ PREFIX
+ LABELS)
+ cmake_parse_arguments(ARG
+ "${options}"
+ "${one_value_args}"
+ "${multi_value_args}"
+ ${ARGN})
+ if(ARG_UNPARSED_ARGUMENTS)
+ message(SEND_ERROR "Error: unrecognized arguments: ${ARG_UNPARSED_ARGUMENTS}")
+ endif()
+
+ if(NO_BENCHMARKS)
+ return()
+ endif()
+ get_filename_component(BENCHMARK_NAME ${REL_BENCHMARK_NAME} NAME_WE)
+
+ if(ARG_PREFIX)
+ set(BENCHMARK_NAME "${ARG_PREFIX}-${BENCHMARK_NAME}")
+ endif()
+
+ if(ARG_SOURCES)
+ set(SOURCES ${ARG_SOURCES})
+ else()
+ set(SOURCES "${REL_BENCHMARK_NAME}.cc")
+ endif()
+
+ # Make sure the executable name contains only hyphens, not underscores
+ string(REPLACE "_" "-" BENCHMARK_NAME ${BENCHMARK_NAME})
+
+ if(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/${REL_BENCHMARK_NAME}.cc)
+ # This benchmark has a corresponding .cc file, set it up as an executable.
+ set(BENCHMARK_PATH "${EXECUTABLE_OUTPUT_PATH}/${BENCHMARK_NAME}")
+ add_executable(${BENCHMARK_NAME} ${SOURCES})
+
+ if(ARG_STATIC_LINK_LIBS)
+ # Customize link libraries
+ target_link_libraries(${BENCHMARK_NAME} PRIVATE ${ARG_STATIC_LINK_LIBS})
+ else()
+ target_link_libraries(${BENCHMARK_NAME} PRIVATE ${ARROW_BENCHMARK_LINK_LIBS})
+ endif()
+ add_dependencies(benchmark ${BENCHMARK_NAME})
+ set(NO_COLOR "--color_print=false")
+
+ if(ARG_EXTRA_LINK_LIBS)
+ target_link_libraries(${BENCHMARK_NAME} PRIVATE ${ARG_EXTRA_LINK_LIBS})
+ endif()
+ else()
+ # No executable, just invoke the benchmark (probably a script) directly.
+ set(BENCHMARK_PATH ${CMAKE_CURRENT_SOURCE_DIR}/${REL_BENCHMARK_NAME})
+ set(NO_COLOR "")
+ endif()
+
+ # With OSX and conda, we need to set the correct RPATH so that dependencies
+ # are found. The installed libraries with conda have an RPATH that matches
+ # for executables and libraries lying in $ENV{CONDA_PREFIX}/bin or
+ # $ENV{CONDA_PREFIX}/lib but our test libraries and executables are not
+ # installed there.
+ if(NOT "$ENV{CONDA_PREFIX}" STREQUAL "" AND APPLE)
+ set_target_properties(${BENCHMARK_NAME}
+ PROPERTIES BUILD_WITH_INSTALL_RPATH TRUE
+ INSTALL_RPATH_USE_LINK_PATH TRUE
+ INSTALL_RPATH
+ "$ENV{CONDA_PREFIX}/lib;${EXECUTABLE_OUTPUT_PATH}")
+ endif()
+
+ # Add test as dependency of relevant label targets
+ add_dependencies(all-benchmarks ${BENCHMARK_NAME})
+ foreach(TARGET ${ARG_LABELS})
+ add_dependencies(${TARGET} ${BENCHMARK_NAME})
+ endforeach()
+
+ if(ARG_DEPENDENCIES)
+ add_dependencies(${BENCHMARK_NAME} ${ARG_DEPENDENCIES})
+ endif()
+
+ if(ARG_LABELS)
+ set(ARG_LABELS "benchmark;${ARG_LABELS}")
+ else()
+ set(ARG_LABELS benchmark)
+ endif()
+
+ if(ARROW_BUILD_DETAILED_BENCHMARKS)
+ target_compile_definitions(${BENCHMARK_NAME} PRIVATE ARROW_BUILD_DETAILED_BENCHMARKS)
+ endif()
+
+ add_test(${BENCHMARK_NAME}
+ ${BUILD_SUPPORT_DIR}/run-test.sh
+ ${CMAKE_BINARY_DIR}
+ benchmark
+ ${BENCHMARK_PATH}
+ ${NO_COLOR})
+ set_property(TEST ${BENCHMARK_NAME}
+ APPEND
+ PROPERTY LABELS ${ARG_LABELS})
+endfunction()
+
+#
+# Testing
+#
+# Add a new test case, with or without an executable that should be built.
+#
+# REL_TEST_NAME is the name of the test. It may be a single component
+# (e.g. monotime-test) or contain additional components (e.g.
+# net/net_util-test). Either way, the last component must be a globally
+# unique name.
+#
+# If given, SOURCES is the list of C++ source files to compile into the test
+# executable. Otherwise, "REL_TEST_NAME.cc" is used.
+#
+# The unit test is added with a label of "unittest" to support filtering with
+# ctest.
+#
+# Arguments after the test name will be passed to set_tests_properties().
+#
+# \arg ENABLED if passed, add this unit test even if ARROW_BUILD_TESTS is off
+# \arg PREFIX a string to append to the name of the test executable. For
+# example, if you have src/arrow/foo/bar-test.cc, then PREFIX "foo" will create
+# test executable foo-bar-test
+# \arg LABELS the unit test label or labels to assign the unit tests
+# to. By default, unit tests will go in the "unittest" group, but if we have
+# multiple unit tests in some subgroup, you can assign a test to multiple
+# groups use the syntax unittest;GROUP2;GROUP3. Custom targets for the group
+# names must exist
+function(ADD_TEST_CASE REL_TEST_NAME)
+ set(options NO_VALGRIND ENABLED)
+ set(one_value_args PRECOMPILED_HEADER_LIB)
+ set(multi_value_args
+ SOURCES
+ PRECOMPILED_HEADERS
+ STATIC_LINK_LIBS
+ EXTRA_LINK_LIBS
+ EXTRA_INCLUDES
+ EXTRA_DEPENDENCIES
+ LABELS
+ EXTRA_LABELS
+ TEST_ARGUMENTS
+ PREFIX)
+ cmake_parse_arguments(ARG
+ "${options}"
+ "${one_value_args}"
+ "${multi_value_args}"
+ ${ARGN})
+ if(ARG_UNPARSED_ARGUMENTS)
+ message(SEND_ERROR "Error: unrecognized arguments: ${ARG_UNPARSED_ARGUMENTS}")
+ endif()
+
+ if(NO_TESTS AND NOT ARG_ENABLED)
+ return()
+ endif()
+ get_filename_component(TEST_NAME ${REL_TEST_NAME} NAME_WE)
+
+ if(ARG_PREFIX)
+ set(TEST_NAME "${ARG_PREFIX}-${TEST_NAME}")
+ endif()
+
+ if(ARG_SOURCES)
+ set(SOURCES ${ARG_SOURCES})
+ else()
+ set(SOURCES "${REL_TEST_NAME}.cc")
+ endif()
+
+ # Make sure the executable name contains only hyphens, not underscores
+ string(REPLACE "_" "-" TEST_NAME ${TEST_NAME})
+
+ set(TEST_PATH "${EXECUTABLE_OUTPUT_PATH}/${TEST_NAME}")
+ add_executable(${TEST_NAME} ${SOURCES})
+
+ # With OSX and conda, we need to set the correct RPATH so that dependencies
+ # are found. The installed libraries with conda have an RPATH that matches
+ # for executables and libraries lying in $ENV{CONDA_PREFIX}/bin or
+ # $ENV{CONDA_PREFIX}/lib but our test libraries and executables are not
+ # installed there.
+ if(NOT "$ENV{CONDA_PREFIX}" STREQUAL "" AND APPLE)
+ set_target_properties(${TEST_NAME}
+ PROPERTIES BUILD_WITH_INSTALL_RPATH TRUE
+ INSTALL_RPATH_USE_LINK_PATH TRUE
+ INSTALL_RPATH
+ "${EXECUTABLE_OUTPUT_PATH};$ENV{CONDA_PREFIX}/lib")
+ endif()
+
+ if(ARG_STATIC_LINK_LIBS)
+ # Customize link libraries
+ target_link_libraries(${TEST_NAME} PRIVATE ${ARG_STATIC_LINK_LIBS})
+ else()
+ target_link_libraries(${TEST_NAME} PRIVATE ${ARROW_TEST_LINK_LIBS})
+ endif()
+
+ if(ARG_PRECOMPILED_HEADER_LIB)
+ reuse_precompiled_header_lib(${TEST_NAME} ${ARG_PRECOMPILED_HEADER_LIB})
+ endif()
+
+ if(ARG_PRECOMPILED_HEADERS AND ARROW_USE_PRECOMPILED_HEADERS)
+ target_precompile_headers(${TEST_NAME} PRIVATE ${ARG_PRECOMPILED_HEADERS})
+ endif()
+
+ if(ARG_EXTRA_LINK_LIBS)
+ target_link_libraries(${TEST_NAME} PRIVATE ${ARG_EXTRA_LINK_LIBS})
+ endif()
+
+ if(ARG_EXTRA_INCLUDES)
+ target_include_directories(${TEST_NAME} SYSTEM PUBLIC ${ARG_EXTRA_INCLUDES})
+ endif()
+
+ if(ARG_EXTRA_DEPENDENCIES)
+ add_dependencies(${TEST_NAME} ${ARG_EXTRA_DEPENDENCIES})
+ endif()
+
+ if(ARROW_TEST_MEMCHECK AND NOT ARG_NO_VALGRIND)
+ add_test(${TEST_NAME}
+ bash
+ -c
+ "cd '${CMAKE_SOURCE_DIR}'; \
+ valgrind --suppressions=valgrind.supp --tool=memcheck --gen-suppressions=all \
+ --num-callers=500 --leak-check=full --leak-check-heuristics=stdstring \
+ --error-exitcode=1 ${TEST_PATH} ${ARG_TEST_ARGUMENTS}")
+ elseif(WIN32)
+ add_test(${TEST_NAME} ${TEST_PATH} ${ARG_TEST_ARGUMENTS})
+ else()
+ add_test(${TEST_NAME}
+ ${BUILD_SUPPORT_DIR}/run-test.sh
+ ${CMAKE_BINARY_DIR}
+ test
+ ${TEST_PATH}
+ ${ARG_TEST_ARGUMENTS})
+ endif()
+
+ # Add test as dependency of relevant targets
+ add_dependencies(all-tests ${TEST_NAME})
+ foreach(TARGET ${ARG_LABELS})
+ add_dependencies(${TARGET} ${TEST_NAME})
+ endforeach()
+
+ set(LABELS)
+ list(APPEND LABELS "unittest")
+ if(ARG_LABELS)
+ list(APPEND LABELS ${ARG_LABELS})
+ endif()
+ # EXTRA_LABELS don't create their own dependencies, they are only used
+ # to ease running certain test categories.
+ if(ARG_EXTRA_LABELS)
+ list(APPEND LABELS ${ARG_EXTRA_LABELS})
+ endif()
+
+ foreach(LABEL ${ARG_LABELS})
+ # ensure there is a cmake target which exercises tests with this LABEL
+ set(LABEL_TEST_NAME "test-${LABEL}")
+ if(NOT TARGET ${LABEL_TEST_NAME})
+ add_custom_target(${LABEL_TEST_NAME}
+ ctest -L "${LABEL}" --output-on-failure
+ USES_TERMINAL)
+ endif()
+ # ensure the test is (re)built before the LABEL test runs
+ add_dependencies(${LABEL_TEST_NAME} ${TEST_NAME})
+ endforeach()
+
+ set_property(TEST ${TEST_NAME}
+ APPEND
+ PROPERTY LABELS ${LABELS})
+endfunction()
+
+#
+# Examples
+#
+# Add a new example, with or without an executable that should be built.
+# If examples are enabled then they will be run along side unit tests with ctest.
+# 'make runexample' to build/run only examples.
+#
+# REL_EXAMPLE_NAME is the name of the example app. It may be a single component
+# (e.g. monotime-example) or contain additional components (e.g.
+# net/net_util-example). Either way, the last component must be a globally
+# unique name.
+
+# The example will registered as unit test with ctest with a label
+# of 'example'.
+#
+# Arguments after the test name will be passed to set_tests_properties().
+#
+# \arg PREFIX a string to append to the name of the example executable. For
+# example, if you have src/arrow/foo/bar-example.cc, then PREFIX "foo" will
+# create test executable foo-bar-example
+function(ADD_ARROW_EXAMPLE REL_EXAMPLE_NAME)
+ set(options)
+ set(one_value_args)
+ set(multi_value_args
+ EXTRA_INCLUDES
+ EXTRA_LINK_LIBS
+ EXTRA_SOURCES
+ DEPENDENCIES
+ PREFIX)
+ cmake_parse_arguments(ARG
+ "${options}"
+ "${one_value_args}"
+ "${multi_value_args}"
+ ${ARGN})
+ if(ARG_UNPARSED_ARGUMENTS)
+ message(SEND_ERROR "Error: unrecognized arguments: ${ARG_UNPARSED_ARGUMENTS}")
+ endif()
+
+ if(NO_EXAMPLES)
+ return()
+ endif()
+ get_filename_component(EXAMPLE_NAME ${REL_EXAMPLE_NAME} NAME_WE)
+
+ if(ARG_PREFIX)
+ set(EXAMPLE_NAME "${ARG_PREFIX}-${EXAMPLE_NAME}")
+ endif()
+
+ # Make sure the executable name contains only hyphens, not underscores
+ string(REPLACE "_" "-" EXAMPLE_NAME ${EXAMPLE_NAME})
+
+ if(EXISTS ${CMAKE_SOURCE_DIR}/examples/arrow/${REL_EXAMPLE_NAME}.cc)
+ # This example has a corresponding .cc file, set it up as an executable.
+ set(EXAMPLE_PATH "${EXECUTABLE_OUTPUT_PATH}/${EXAMPLE_NAME}")
+ add_executable(${EXAMPLE_NAME} "${REL_EXAMPLE_NAME}.cc" ${ARG_EXTRA_SOURCES})
+ target_link_libraries(${EXAMPLE_NAME} ${ARROW_EXAMPLE_LINK_LIBS})
+ add_dependencies(runexample ${EXAMPLE_NAME})
+ set(NO_COLOR "--color_print=false")
+
+ if(ARG_EXTRA_LINK_LIBS)
+ target_link_libraries(${EXAMPLE_NAME} ${ARG_EXTRA_LINK_LIBS})
+ endif()
+ endif()
+
+ if(ARG_DEPENDENCIES)
+ add_dependencies(${EXAMPLE_NAME} ${ARG_DEPENDENCIES})
+ endif()
+
+ if(ARG_EXTRA_INCLUDES)
+ target_include_directories(${EXAMPLE_NAME} SYSTEM PUBLIC ${ARG_EXTRA_INCLUDES})
+ endif()
+
+ add_test(${EXAMPLE_NAME} ${EXAMPLE_PATH})
+ set_tests_properties(${EXAMPLE_NAME} PROPERTIES LABELS "example")
+endfunction()
+
+#
+# Fuzzing
+#
+# Add new fuzz target executable.
+#
+# The single source file must define a function:
+# extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
+#
+# No main function must be present within the source file!
+#
+function(ADD_FUZZ_TARGET REL_FUZZING_NAME)
+ set(options)
+ set(one_value_args PREFIX)
+ set(multi_value_args LINK_LIBS)
+ cmake_parse_arguments(ARG
+ "${options}"
+ "${one_value_args}"
+ "${multi_value_args}"
+ ${ARGN})
+ if(ARG_UNPARSED_ARGUMENTS)
+ message(SEND_ERROR "Error: unrecognized arguments: ${ARG_UNPARSED_ARGUMENTS}")
+ endif()
+
+ if(NO_FUZZING)
+ return()
+ endif()
+
+ get_filename_component(FUZZING_NAME ${REL_FUZZING_NAME} NAME_WE)
+
+ # Make sure the executable name contains only hyphens, not underscores
+ string(REPLACE "_" "-" FUZZING_NAME ${FUZZING_NAME})
+
+ if(ARG_PREFIX)
+ set(FUZZING_NAME "${ARG_PREFIX}-${FUZZING_NAME}")
+ endif()
+
+ # For OSS-Fuzz
+ # (https://google.github.io/oss-fuzz/advanced-topics/ideal-integration/)
+ if(DEFINED ENV{LIB_FUZZING_ENGINE})
+ set(FUZZ_LDFLAGS $ENV{LIB_FUZZING_ENGINE})
+ else()
+ set(FUZZ_LDFLAGS "-fsanitize=fuzzer")
+ endif()
+
+ add_executable(${FUZZING_NAME} "${REL_FUZZING_NAME}.cc")
+ target_link_libraries(${FUZZING_NAME} ${LINK_LIBS})
+ target_compile_options(${FUZZING_NAME} PRIVATE ${FUZZ_LDFLAGS})
+ set_target_properties(${FUZZING_NAME} PROPERTIES LINK_FLAGS ${FUZZ_LDFLAGS} LABELS
+ "fuzzing")
+endfunction()
+
+function(ARROW_INSTALL_ALL_HEADERS PATH)
+ set(options)
+ set(one_value_args)
+ set(multi_value_args PATTERN)
+ cmake_parse_arguments(ARG
+ "${options}"
+ "${one_value_args}"
+ "${multi_value_args}"
+ ${ARGN})
+ if(NOT ARG_PATTERN)
+ # The .hpp extension is used by some vendored libraries
+ set(ARG_PATTERN "*.h" "*.hpp")
+ endif()
+ file(GLOB CURRENT_DIRECTORY_HEADERS ${ARG_PATTERN})
+
+ set(PUBLIC_HEADERS)
+ foreach(HEADER ${CURRENT_DIRECTORY_HEADERS})
+ get_filename_component(HEADER_BASENAME ${HEADER} NAME)
+ if(HEADER_BASENAME MATCHES "internal")
+ continue()
+ endif()
+ list(APPEND PUBLIC_HEADERS ${HEADER})
+ endforeach()
+ install(FILES ${PUBLIC_HEADERS} DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}/${PATH}")
+endfunction()
+
+function(ARROW_ADD_PKG_CONFIG MODULE)
+ configure_file(${MODULE}.pc.in "${CMAKE_CURRENT_BINARY_DIR}/${MODULE}.pc" @ONLY)
+ install(FILES "${CMAKE_CURRENT_BINARY_DIR}/${MODULE}.pc"
+ DESTINATION "${CMAKE_INSTALL_LIBDIR}/pkgconfig/")
+endfunction()
+
+function(ARROW_INSTALL_CMAKE_FIND_MODULE MODULE)
+ install(FILES "${ARROW_SOURCE_DIR}/cmake_modules/Find${MODULE}.cmake"
+ DESTINATION "${ARROW_CMAKE_INSTALL_DIR}")
+endfunction()
+
+# Implementations of lisp "car" and "cdr" functions
+macro(ARROW_CAR var)
+ set(${var} ${ARGV1})
+endmacro()
+
+macro(ARROW_CDR var rest)
+ set(${var} ${ARGN})
+endmacro()
diff --git a/cmake_modules/DefineOptions.cmake b/cmake_modules/DefineOptions.cmake
new file mode 100644
index 0000000..394f76d
--- /dev/null
+++ b/cmake_modules/DefineOptions.cmake
@@ -0,0 +1,348 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+macro(set_option_category name)
+ set(ARROW_OPTION_CATEGORY ${name})
+ list(APPEND "ARROW_OPTION_CATEGORIES" ${name})
+endmacro()
+
+function(check_description_length name description)
+ foreach(description_line ${description})
+ string(LENGTH ${description_line} line_length)
+ if(${line_length} GREATER 80)
+ message(FATAL_ERROR "description for ${name} contained a\n\
+ line ${line_length} characters long!\n\
+ (max is 80). Split it into more lines with semicolons")
+ endif()
+ endforeach()
+endfunction()
+
+function(list_join lst glue out)
+ if("${${lst}}" STREQUAL "")
+ set(${out}
+ ""
+ PARENT_SCOPE)
+ return()
+ endif()
+
+ list(GET ${lst} 0 joined)
+ list(REMOVE_AT ${lst} 0)
+ foreach(item ${${lst}})
+ set(joined "${joined}${glue}${item}")
+ endforeach()
+ set(${out}
+ ${joined}
+ PARENT_SCOPE)
+endfunction()
+
+macro(define_option name description default)
+ check_description_length(${name} ${description})
+ list_join(description "\n" multiline_description)
+
+ option(${name} "${multiline_description}" ${default})
+
+ list(APPEND "ARROW_${ARROW_OPTION_CATEGORY}_OPTION_NAMES" ${name})
+ set("${name}_OPTION_DESCRIPTION" ${description})
+ set("${name}_OPTION_DEFAULT" ${default})
+ set("${name}_OPTION_TYPE" "bool")
+endmacro()
+
+macro(define_option_string name description default)
+ check_description_length(${name} ${description})
+ list_join(description "\n" multiline_description)
+
+ set(${name}
+ ${default}
+ CACHE STRING "${multiline_description}")
+
+ list(APPEND "ARROW_${ARROW_OPTION_CATEGORY}_OPTION_NAMES" ${name})
+ set("${name}_OPTION_DESCRIPTION" ${description})
+ set("${name}_OPTION_DEFAULT" "\"${default}\"")
+ set("${name}_OPTION_TYPE" "string")
+ set("${name}_OPTION_POSSIBLE_VALUES" ${ARGN})
+
+ list_join("${name}_OPTION_POSSIBLE_VALUES" "|" "${name}_OPTION_ENUM")
+ if(NOT ("${${name}_OPTION_ENUM}" STREQUAL ""))
+ set_property(CACHE ${name} PROPERTY STRINGS "${name}_OPTION_POSSIBLE_VALUES")
+ endif()
+endmacro()
+
+# Top level cmake dir
+if("${CMAKE_SOURCE_DIR}" STREQUAL "${CMAKE_CURRENT_SOURCE_DIR}")
+ #----------------------------------------------------------------------
+ set_option_category("Compile and link")
+
+ define_option_string(ARROW_CXXFLAGS "Compiler flags to append when compiling Arrow" "")
+
+ define_option(ARROW_BUILD_STATIC "Build static libraries" ON)
+
+ define_option(ARROW_BUILD_SHARED "Build shared libraries" ON)
+
+ define_option_string(ARROW_PACKAGE_KIND
+ "Arbitrary string that identifies the kind of package;\
+(for informational purposes)" "")
+
+ define_option_string(ARROW_GIT_ID "The Arrow git commit id (if any)" "")
+
+ define_option_string(ARROW_GIT_DESCRIPTION "The Arrow git commit description (if any)"
+ "")
+
+ define_option(ARROW_NO_DEPRECATED_API "Exclude deprecated APIs from build" OFF)
+
+ define_option(ARROW_USE_CCACHE "Use ccache when compiling (if available)" ON)
+
+ define_option(ARROW_USE_PRECOMPILED_HEADERS "Use precompiled headers when compiling"
+ OFF)
+
+ # Arm64 architectures and extensions can lead to exploding combinations.
+ # So set it directly through cmake command line.
+ #
+ # If you change this, you need to change the definition in
+ # python/CMakeLists.txt too.
+ define_option_string(ARROW_ARMV8_ARCH
+ "Arm64 arch and extensions"
+ "armv8-a" # Default
+ "armv8-a"
+ "armv8-a+crc+crypto")
+
+ define_option(ARROW_ALTIVEC "Build with Altivec if compiler has support" ON)
+
+ define_option(ARROW_RPATH_ORIGIN "Build Arrow libraries with RATH set to \$ORIGIN" OFF)
+
+ define_option(ARROW_INSTALL_NAME_RPATH
+ "Build Arrow libraries with install_name set to @rpath" ON)
+
+ define_option(ARROW_GGDB_DEBUG "Pass -ggdb flag to debug builds" ON)
+
+ #----------------------------------------------------------------------
+ set_option_category("Test and benchmark")
+
+ define_option(ARROW_BUILD_EXAMPLES "Build the Arrow examples" OFF)
+
+ define_option(ARROW_BUILD_TESTS "Build the Arrow googletest unit tests" OFF)
+
+ define_option(ARROW_BUILD_INTEGRATION "Build the Arrow integration test executables"
+ OFF)
+
+ define_option(ARROW_BUILD_BENCHMARKS "Build the Arrow micro benchmarks" OFF)
+
+ if(ARROW_BUILD_SHARED)
+ set(ARROW_TEST_LINKAGE_DEFAULT "shared")
+ else()
+ set(ARROW_TEST_LINKAGE_DEFAULT "static")
+ endif()
+
+ define_option_string(ARROW_TEST_LINKAGE
+ "Linkage of Arrow libraries with unit tests executables."
+ "${ARROW_TEST_LINKAGE_DEFAULT}"
+ "shared"
+ "static")
+
+ #----------------------------------------------------------------------
+ set_option_category("Lint")
+
+ define_option(ARROW_GENERATE_COVERAGE "Build with C++ code coverage enabled" OFF)
+
+ #----------------------------------------------------------------------
+ set_option_category("Checks")
+
+ define_option(ARROW_TEST_MEMCHECK "Run the test suite using valgrind --tool=memcheck"
+ OFF)
+
+ define_option(ARROW_USE_ASAN "Enable Address Sanitizer checks" OFF)
+
+ define_option(ARROW_USE_TSAN "Enable Thread Sanitizer checks" OFF)
+
+ define_option(ARROW_USE_UBSAN "Enable Undefined Behavior sanitizer checks" OFF)
+
+ #----------------------------------------------------------------------
+ set_option_category("Thirdparty toolchain")
+
+ # Determine how we will look for dependencies
+ # * AUTO: Guess which packaging systems we're running in and pull the
+ # dependencies from there. Build any missing ones through the
+ # ExternalProject setup. This is the default unless the CONDA_PREFIX
+ # environment variable is set, in which case the CONDA method is used
+ # * BUNDLED: Build dependencies through CMake's ExternalProject facility. If
+ # you wish to build individual dependencies from source instead of using
+ # one of the other methods, pass -D$NAME_SOURCE=BUNDLED
+ # * SYSTEM: Use CMake's find_package and find_library without any custom
+ # paths. If individual packages are on non-default locations, you can pass
+ # $NAME_ROOT arguments to CMake, or set environment variables for the same
+ # with CMake 3.11 and higher. If your system packages are in a non-default
+ # location, or if you are using a non-standard toolchain, you can also pass
+ # ARROW_PACKAGE_PREFIX to set the *_ROOT variables to look in that
+ # directory
+ # * CONDA: Same as SYSTEM but set all *_ROOT variables to
+ # ENV{CONDA_PREFIX}. If this is run within an active conda environment,
+ # then ENV{CONDA_PREFIX} will be used for dependencies unless
+ # ARROW_DEPENDENCY_SOURCE is set explicitly to one of the other options
+ # * VCPKG: Searches for dependencies installed by vcpkg.
+ # * BREW: Use SYSTEM but search for select packages with brew.
+ if(NOT "$ENV{CONDA_PREFIX}" STREQUAL "")
+ set(ARROW_DEPENDENCY_SOURCE_DEFAULT "CONDA")
+ else()
+ set(ARROW_DEPENDENCY_SOURCE_DEFAULT "AUTO")
+ endif()
+ define_option_string(ARROW_DEPENDENCY_SOURCE
+ "Method to use for acquiring arrow's build dependencies"
+ "${ARROW_DEPENDENCY_SOURCE_DEFAULT}"
+ "AUTO"
+ "BUNDLED"
+ "SYSTEM"
+ "CONDA"
+ "VCPKG"
+ "BREW")
+
+ #----------------------------------------------------------------------
+ if(MSVC_TOOLCHAIN)
+ set_option_category("MSVC")
+
+ define_option(MSVC_LINK_VERBOSE
+ "Pass verbose linking options when linking libraries and executables"
+ OFF)
+
+ define_option(ARROW_USE_STATIC_CRT "Build Arrow with statically linked CRT" OFF)
+ endif()
+
+ #----------------------------------------------------------------------
+ set_option_category("Advanced developer")
+
+ option(ARROW_BUILD_CONFIG_SUMMARY_JSON "Summarize build configuration in a JSON file"
+ ON)
+endif()
+
+macro(validate_config)
+ foreach(category ${ARROW_OPTION_CATEGORIES})
+ set(option_names ${ARROW_${category}_OPTION_NAMES})
+
+ foreach(name ${option_names})
+ set(possible_values ${${name}_OPTION_POSSIBLE_VALUES})
+ set(value "${${name}}")
+ if(possible_values)
+ if(NOT "${value}" IN_LIST possible_values)
+ message(FATAL_ERROR "Configuration option ${name} got invalid value '${value}'. "
+ "Allowed values: ${${name}_OPTION_ENUM}.")
+ endif()
+ endif()
+ endforeach()
+
+ endforeach()
+endmacro()
+
+macro(config_summary_message)
+ message(STATUS "---------------------------------------------------------------------")
+ message(STATUS "Arrow version: ${ARROW_VERSION}")
+ message(STATUS)
+ message(STATUS "Build configuration summary:")
+
+ message(STATUS " Generator: ${CMAKE_GENERATOR}")
+ message(STATUS " Build type: ${CMAKE_BUILD_TYPE}")
+ message(STATUS " Source directory: ${CMAKE_CURRENT_SOURCE_DIR}")
+ message(STATUS " Install prefix: ${CMAKE_INSTALL_PREFIX}")
+ if(${CMAKE_EXPORT_COMPILE_COMMANDS})
+ message(STATUS " Compile commands: ${CMAKE_CURRENT_BINARY_DIR}/compile_commands.json"
+ )
+ endif()
+
+ foreach(category ${ARROW_OPTION_CATEGORIES})
+
+ message(STATUS)
+ message(STATUS "${category} options:")
+ message(STATUS)
+
+ set(option_names ${ARROW_${category}_OPTION_NAMES})
+
+ foreach(name ${option_names})
+ set(value "${${name}}")
+ if("${value}" STREQUAL "")
+ set(value "\"\"")
+ endif()
+
+ set(description ${${name}_OPTION_DESCRIPTION})
+
+ if(NOT ("${${name}_OPTION_ENUM}" STREQUAL ""))
+ set(summary "=${value} [default=${${name}_OPTION_ENUM}]")
+ else()
+ set(summary "=${value} [default=${${name}_OPTION_DEFAULT}]")
+ endif()
+
+ message(STATUS " ${name}${summary}")
+ foreach(description_line ${description})
+ message(STATUS " ${description_line}")
+ endforeach()
+ endforeach()
+
+ endforeach()
+
+endmacro()
+
+macro(config_summary_json)
+ set(summary "${CMAKE_CURRENT_BINARY_DIR}/cmake_summary.json")
+ message(STATUS " Outputting build configuration summary to ${summary}")
+ file(WRITE ${summary} "{\n")
+
+ foreach(category ${ARROW_OPTION_CATEGORIES})
+ foreach(name ${ARROW_${category}_OPTION_NAMES})
+ file(APPEND ${summary} "\"${name}\": \"${${name}}\",\n")
+ endforeach()
+ endforeach()
+
+ file(APPEND ${summary} "\"generator\": \"${CMAKE_GENERATOR}\",\n")
+ file(APPEND ${summary} "\"build_type\": \"${CMAKE_BUILD_TYPE}\",\n")
+ file(APPEND ${summary} "\"source_dir\": \"${CMAKE_CURRENT_SOURCE_DIR}\",\n")
+ if(${CMAKE_EXPORT_COMPILE_COMMANDS})
+ file(APPEND ${summary} "\"compile_commands\": "
+ "\"${CMAKE_CURRENT_BINARY_DIR}/compile_commands.json\",\n")
+ endif()
+ file(APPEND ${summary} "\"install_prefix\": \"${CMAKE_INSTALL_PREFIX}\",\n")
+ file(APPEND ${summary} "\"arrow_version\": \"${ARROW_VERSION}\"\n")
+ file(APPEND ${summary} "}\n")
+endmacro()
+
+macro(config_summary_cmake_setters path)
+ file(WRITE ${path} "# Options used to build arrow:")
+
+ foreach(category ${ARROW_OPTION_CATEGORIES})
+ file(APPEND ${path} "\n\n## ${category} options:")
+ foreach(name ${ARROW_${category}_OPTION_NAMES})
+ set(description ${${name}_OPTION_DESCRIPTION})
+ foreach(description_line ${description})
+ file(APPEND ${path} "\n### ${description_line}")
+ endforeach()
+ file(APPEND ${path} "\nset(${name} \"${${name}}\")")
+ endforeach()
+ endforeach()
+
+endmacro()
+
+#----------------------------------------------------------------------
+# Compute default values for omitted variables
+
+if(NOT ARROW_GIT_ID)
+ execute_process(COMMAND "git" "log" "-n1" "--format=%H"
+ WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
+ OUTPUT_VARIABLE ARROW_GIT_ID
+ OUTPUT_STRIP_TRAILING_WHITESPACE)
+endif()
+if(NOT ARROW_GIT_DESCRIPTION)
+ execute_process(COMMAND "git" "describe" "--tags" "--dirty"
+ WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
+ ERROR_QUIET
+ OUTPUT_VARIABLE ARROW_GIT_DESCRIPTION
+ OUTPUT_STRIP_TRAILING_WHITESPACE)
+endif()
diff --git a/cmake_modules/README.md b/cmake_modules/README.md
new file mode 100644
index 0000000..9e726c4
--- /dev/null
+++ b/cmake_modules/README.md
@@ -0,0 +1,6 @@
+# CMake Build Utilities
+
+Common CMake utilities taken from Apache Arrow. These should be kept
+in sync with Arrow upstream without modification. Exceptions:
+
+- DefineOptions.cmake has different flags.
diff --git a/drivers/flight_sql/flight_sql.cc b/drivers/flight_sql/flight_sql.cc
new file mode 100644
index 0000000..03adf3f
--- /dev/null
+++ b/drivers/flight_sql/flight_sql.cc
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <mutex>
+#include <string>
+
+#include "adbc/adbc.h"
+#include "adbc/driver/util.h"
+#include "arrow/c/bridge.h"
+#include "arrow/flight/client.h"
+#include "arrow/flight/sql/client.h"
+#include "arrow/record_batch.h"
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/util/string_builder.h"
+#include "arrow/util/string_view.h"
+
+namespace flight = arrow::flight;
+namespace flightsql = arrow::flight::sql;
+using arrow::Status;
+
+namespace {
+
+void SetError(const Status& status, struct AdbcError* error) {
+ if (!error) return;
+ std::string message = arrow::util::StringBuilder("[Flight SQL] ", status.ToString());
+ if (error->message) {
+ message.reserve(message.size() + 1 + std::strlen(error->message));
+ message.append(1, '\n');
+ message.append(error->message);
+ delete[] error->message;
+ }
+ error->message = new char[message.size() + 1];
+ message.copy(error->message, message.size());
+ error->message[message.size()] = '\0';
+}
+
+template <typename... Args>
+void SetError(struct AdbcError* error, Args&&... args) {
+ if (!error) return;
+ std::string message =
+ arrow::util::StringBuilder("[Flight SQL] ", std::forward<Args>(args)...);
+ if (error->message) {
+ message.reserve(message.size() + 1 + std::strlen(error->message));
+ message.append(1, '\n');
+ message.append(error->message);
+ delete[] error->message;
+ }
+ error->message = new char[message.size() + 1];
+ message.copy(error->message, message.size());
+ error->message[message.size()] = '\0';
+}
+
+class FlightSqlDatabaseImpl {
+ public:
+ explicit FlightSqlDatabaseImpl(std::unique_ptr<flightsql::FlightSqlClient> client)
+ : client_(std::move(client)), connection_count_(0) {}
+
+ flightsql::FlightSqlClient* Connect() {
+ std::lock_guard<std::mutex> guard(mutex_);
+ ++connection_count_;
+ return client_.get();
+ }
+
+ AdbcStatusCode Disconnect(struct AdbcError* error) {
+ std::lock_guard<std::mutex> guard(mutex_);
+ if (--connection_count_ < 0) {
+ SetError(error, "Connection count underflow");
+ return ADBC_STATUS_INTERNAL;
+ }
+ return ADBC_STATUS_OK;
+ }
+
+ AdbcStatusCode Release(struct AdbcError* error) {
+ std::lock_guard<std::mutex> guard(mutex_);
+
+ if (connection_count_ > 0) {
+ SetError(error, "Cannot release database with ", connection_count_,
+ " open connections");
+ return ADBC_STATUS_INTERNAL;
+ }
+
+ auto status = client_->Close();
+ if (!status.ok()) {
+ SetError(status, error);
+ return ADBC_STATUS_IO;
+ }
+ return ADBC_STATUS_OK;
+ }
+
+ private:
+ std::unique_ptr<flightsql::FlightSqlClient> client_;
+ int connection_count_;
+ std::mutex mutex_;
+};
+
+class FlightSqlStatementImpl : public arrow::RecordBatchReader {
+ public:
+ FlightSqlStatementImpl() : client_(nullptr), info_() {}
+
+ void Init(flightsql::FlightSqlClient* client,
+ std::unique_ptr<flight::FlightInfo> info) {
+ client_ = client;
+ info_ = std::move(info);
+ }
+
+ //----------------------------------------------------------
+ // arrow::RecordBatchReader Methods
+ //----------------------------------------------------------
+
+ std::shared_ptr<arrow::Schema> schema() const override { return schema_; }
+
+ Status ReadNext(std::shared_ptr<arrow::RecordBatch>* batch) override {
+ flight::FlightStreamChunk chunk;
+ while (current_stream_ && !chunk.data) {
+ RETURN_NOT_OK(current_stream_->Next(&chunk));
+ if (chunk.data) {
+ *batch = chunk.data;
+ break;
+ }
+ if (!chunk.data && !chunk.app_metadata) {
+ RETURN_NOT_OK(NextStream());
+ }
+ }
+ return Status::OK();
+ }
+
+ //----------------------------------------------------------
+ // Common Functions
+ //----------------------------------------------------------
+
+ // TODO: a lot of these could be implemented in a common mixin
+ AdbcStatusCode Close(struct AdbcError* error) { return ADBC_STATUS_OK; }
+
+ //----------------------------------------------------------
+ // Statement Functions
+ //----------------------------------------------------------
+
+ AdbcStatusCode GetStream(const std::shared_ptr<FlightSqlStatementImpl>& self,
+ struct ArrowArrayStream* out, struct AdbcError* error) {
+ if (!client_) {
+ SetError(error, "Statement has not yet been executed");
+ return ADBC_STATUS_UNINITIALIZED;
+ }
+
+ auto status = NextStream();
+ if (!status.ok()) {
+ SetError(status, error);
+ return ADBC_STATUS_IO;
+ }
+ if (!schema_) {
+ return ADBC_STATUS_UNKNOWN;
+ }
+
+ status = arrow::ExportRecordBatchReader(self, out);
+ if (!status.ok()) {
+ SetError(status, error);
+ return ADBC_STATUS_UNKNOWN;
+ }
+ return ADBC_STATUS_OK;
+ }
+
+ AdbcStatusCode GetPartitionDescSize(size_t* length, struct AdbcError* error) const {
+ if (!client_) {
+ SetError(error, "Statement has not yet been executed");
+ return ADBC_STATUS_UNINITIALIZED;
+ }
+
+ // TODO: we're only encoding the ticket, not the actual locations
+ if (next_endpoint_ >= info_->endpoints().size()) {
+ *length = 0;
+ return ADBC_STATUS_OK;
+ }
+ *length = info_->endpoints()[next_endpoint_].ticket.ticket.size();
+ return ADBC_STATUS_OK;
+ }
+
+ AdbcStatusCode GetPartitionDesc(uint8_t* partition_desc, struct AdbcError* error) {
+ if (!client_) {
+ SetError(error, "Statement has not yet been executed");
+ return ADBC_STATUS_UNINITIALIZED;
+ }
+
+ if (next_endpoint_ >= info_->endpoints().size()) {
+ return ADBC_STATUS_INVALID_ARGUMENT;
+ }
+ const std::string& ticket = info_->endpoints()[next_endpoint_].ticket.ticket;
+ std::memcpy(partition_desc, ticket.data(), ticket.size());
+ next_endpoint_++;
+ return ADBC_STATUS_OK;
+ }
+
+ private:
+ Status NextStream() {
+ if (next_endpoint_ >= info_->endpoints().size()) {
+ current_stream_ = nullptr;
+ return Status::OK();
+ }
+ // TODO: this needs to account for location
+ flight::FlightCallOptions call_options;
+ ARROW_ASSIGN_OR_RAISE(
+ current_stream_,
+ client_->DoGet(call_options, info_->endpoints()[next_endpoint_].ticket));
+ next_endpoint_++;
+ if (!schema_) {
+ ARROW_ASSIGN_OR_RAISE(schema_, current_stream_->GetSchema());
+ }
+ return Status::OK();
+ }
+
+ flightsql::FlightSqlClient* client_;
+ std::unique_ptr<flight::FlightInfo> info_;
+ std::shared_ptr<arrow::Schema> schema_;
+ size_t next_endpoint_ = 0;
+ std::unique_ptr<flight::FlightStreamReader> current_stream_;
+};
+
+class AdbcFlightSqlImpl {
+ public:
+ explicit AdbcFlightSqlImpl(std::shared_ptr<FlightSqlDatabaseImpl> database)
+ : database_(std::move(database)), client_(database_->Connect()) {}
+
+ //----------------------------------------------------------
+ // Common Functions
+ //----------------------------------------------------------
+
+ AdbcStatusCode Close(struct AdbcError* error) { return database_->Disconnect(error); }
+
+ //----------------------------------------------------------
+ // Metadata
+ //----------------------------------------------------------
+
+ AdbcStatusCode GetTableTypes(struct AdbcStatement* out, struct AdbcError* error) {
+ if (!out->private_data) {
+ SetError(error, "Statement is uninitialized, use AdbcStatementInit");
+ return ADBC_STATUS_UNINITIALIZED;
+ }
+ auto* ptr =
+ reinterpret_cast<std::shared_ptr<FlightSqlStatementImpl>*>(out->private_data);
+ auto* impl = ptr->get();
+
+ flight::FlightCallOptions call_options;
+ std::unique_ptr<flight::FlightInfo> flight_info;
+ auto status = client_->GetTableTypes(call_options).Value(&flight_info);
+ if (!status.ok()) {
+ SetError(status, error);
+ return ADBC_STATUS_IO;
+ }
+ impl->Init(client_, std::move(flight_info));
+ return ADBC_STATUS_OK;
+ }
+
+ //----------------------------------------------------------
+ // SQL Semantics
+ //----------------------------------------------------------
+
+ AdbcStatusCode SqlExecute(const char* query, struct AdbcStatement* out,
+ struct AdbcError* error) {
+ if (!out->private_data) {
+ SetError(error, "Statement is uninitialized, use AdbcStatementInit");
+ return ADBC_STATUS_UNINITIALIZED;
+ }
+ auto* ptr =
+ reinterpret_cast<std::shared_ptr<FlightSqlStatementImpl>*>(out->private_data);
+ auto* impl = ptr->get();
+
+ flight::FlightCallOptions call_options;
+ std::unique_ptr<flight::FlightInfo> flight_info;
+ auto status = client_->Execute(call_options, std::string(query)).Value(&flight_info);
+ if (!status.ok()) {
+ SetError(status, error);
+ return ADBC_STATUS_IO;
+ }
+ impl->Init(client_, std::move(flight_info));
+ return ADBC_STATUS_OK;
+ }
+
+ //----------------------------------------------------------
+ // Partitioned Results
+ //----------------------------------------------------------
+
+ AdbcStatusCode DeserializePartitionDesc(const uint8_t* serialized_partition,
+ size_t serialized_length,
+ struct AdbcStatement* out,
+ struct AdbcError* error) {
+ if (!out->private_data) {
+ SetError(error, "Statement is uninitialized, use AdbcStatementInit");
+ return ADBC_STATUS_UNINITIALIZED;
+ }
+ auto* ptr =
+ reinterpret_cast<std::shared_ptr<FlightSqlStatementImpl>*>(out->private_data);
+ auto* impl = ptr->get();
+
+ std::vector<flight::FlightEndpoint> endpoints(1);
+ endpoints[0].ticket.ticket = std::string(
+ reinterpret_cast<const char*>(serialized_partition), serialized_length);
+ auto maybe_info = flight::FlightInfo::Make(
+ *arrow::schema({}), flight::FlightDescriptor::Command(""), endpoints,
+ /*total_records=*/-1, /*total_bytes=*/-1);
+ if (!maybe_info.ok()) {
+ SetError(maybe_info.status(), error);
+ return ADBC_STATUS_INVALID_ARGUMENT;
+ }
+ std::unique_ptr<flight::FlightInfo> flight_info(
+ new flight::FlightInfo(maybe_info.MoveValueUnsafe()));
+
+ impl->Init(client_, std::move(flight_info));
+ return ADBC_STATUS_OK;
+ }
+
+ private:
+ std::shared_ptr<FlightSqlDatabaseImpl> database_;
+ flightsql::FlightSqlClient* client_;
+};
+
+} // namespace
+
+ADBC_DRIVER_EXPORT
+void AdbcErrorRelease(struct AdbcError* error) {
+ delete[] error->message;
+ error->message = nullptr;
+}
+
+ADBC_DRIVER_EXPORT
+AdbcStatusCode AdbcDatabaseInit(const struct AdbcDatabaseOptions* options,
+ struct AdbcDatabase* out, struct AdbcError* error) {
+ std::unordered_map<std::string, std::string> option_pairs;
+ auto status = adbc::ParseConnectionString(arrow::util::string_view(options->target))
+ .Value(&option_pairs);
+ if (!status.ok()) {
+ SetError(status, error);
+ return ADBC_STATUS_INVALID_ARGUMENT;
+ }
+ auto location_it = option_pairs.find("Location");
+ if (location_it == option_pairs.end()) {
+ SetError(Status::Invalid("Must provide Location option"), error);
+ return ADBC_STATUS_INVALID_ARGUMENT;
+ }
+
+ flight::Location location;
+ status = flight::Location::Parse(location_it->second, &location);
+ if (!status.ok()) {
+ SetError(status, error);
+ return ADBC_STATUS_INVALID_ARGUMENT;
+ }
+
+ std::unique_ptr<flight::FlightClient> flight_client;
+ status = flight::FlightClient::Connect(location, &flight_client);
+ if (!status.ok()) {
+ SetError(status, error);
+ return ADBC_STATUS_IO;
+ }
+ std::unique_ptr<flightsql::FlightSqlClient> client(
+ new flightsql::FlightSqlClient(std::move(flight_client)));
+
+ auto impl = std::make_shared<FlightSqlDatabaseImpl>(std::move(client));
+ out->private_data = new std::shared_ptr<FlightSqlDatabaseImpl>(impl);
+ return ADBC_STATUS_OK;
+}
+
+ADBC_DRIVER_EXPORT
+AdbcStatusCode AdbcDatabaseRelease(struct AdbcDatabase* database,
+ struct AdbcError* error) {
+ if (!database->private_data) return ADBC_STATUS_UNINITIALIZED;
+ auto ptr =
+ reinterpret_cast<std::shared_ptr<FlightSqlDatabaseImpl>*>(database->private_data);
+ AdbcStatusCode status = (*ptr)->Release(error);
+ delete ptr;
+ database->private_data = nullptr;
+ return status;
+}
+
+ADBC_DRIVER_EXPORT
+AdbcStatusCode AdbcConnectionDeserializePartitionDesc(struct AdbcConnection* connection,
+ const uint8_t* serialized_partition,
+ size_t serialized_length,
+ struct AdbcStatement* statement,
+ struct AdbcError* error) {
+ if (!connection->private_data) return ADBC_STATUS_UNINITIALIZED;
+ auto* ptr =
+ reinterpret_cast<std::shared_ptr<AdbcFlightSqlImpl>*>(connection->private_data);
+ return (*ptr)->DeserializePartitionDesc(serialized_partition, serialized_length,
+ statement, error);
+}
+
+ADBC_DRIVER_EXPORT
+AdbcStatusCode AdbcConnectionGetTableTypes(struct AdbcConnection* connection,
+ struct AdbcStatement* statement,
+ struct AdbcError* error) {
+ if (!connection->private_data) return ADBC_STATUS_UNINITIALIZED;
+ auto* ptr =
+ reinterpret_cast<std::shared_ptr<AdbcFlightSqlImpl>*>(connection->private_data);
+ return (*ptr)->GetTableTypes(statement, error);
+}
+
+ADBC_DRIVER_EXPORT
+AdbcStatusCode AdbcConnectionInit(const struct AdbcConnectionOptions* options,
+ struct AdbcConnection* out, struct AdbcError* error) {
+ if (!options->database || !options->database->private_data) {
+ SetError(error, "Must provide database");
+ return ADBC_STATUS_INVALID_ARGUMENT;
+ }
+ auto ptr = reinterpret_cast<std::shared_ptr<FlightSqlDatabaseImpl>*>(
+ options->database->private_data);
+ auto impl = std::make_shared<AdbcFlightSqlImpl>(*ptr);
+ out->private_data = new std::shared_ptr<AdbcFlightSqlImpl>(impl);
+ return ADBC_STATUS_OK;
+}
+
+ADBC_DRIVER_EXPORT
+AdbcStatusCode AdbcConnectionRelease(struct AdbcConnection* connection,
+ struct AdbcError* error) {
+ if (!connection->private_data) return ADBC_STATUS_UNINITIALIZED;
+ auto* ptr =
+ reinterpret_cast<std::shared_ptr<AdbcFlightSqlImpl>*>(connection->private_data);
+ auto status = (*ptr)->Close(error);
+ delete ptr;
+ connection->private_data = nullptr;
+ return status;
+}
+
+ADBC_DRIVER_EXPORT
+AdbcStatusCode AdbcConnectionSqlExecute(struct AdbcConnection* connection,
+ const char* query, struct AdbcStatement* out,
+ struct AdbcError* error) {
+ if (!connection->private_data) return ADBC_STATUS_UNINITIALIZED;
+ auto* ptr =
+ reinterpret_cast<std::shared_ptr<AdbcFlightSqlImpl>*>(connection->private_data);
+ return (*ptr)->SqlExecute(query, out, error);
+}
+
+ADBC_DRIVER_EXPORT
+AdbcStatusCode AdbcStatementGetPartitionDesc(struct AdbcStatement* statement,
+ uint8_t* partition_desc,
+ struct AdbcError* error) {
+ if (!statement->private_data) return ADBC_STATUS_UNINITIALIZED;
+ auto* ptr =
+ reinterpret_cast<std::shared_ptr<FlightSqlStatementImpl>*>(statement->private_data);
+ return (*ptr)->GetPartitionDesc(partition_desc, error);
+}
+
+ADBC_DRIVER_EXPORT
+AdbcStatusCode AdbcStatementGetPartitionDescSize(struct AdbcStatement* statement,
+ size_t* length,
+ struct AdbcError* error) {
+ if (!statement->private_data) return ADBC_STATUS_UNINITIALIZED;
+ auto* ptr =
+ reinterpret_cast<std::shared_ptr<FlightSqlStatementImpl>*>(statement->private_data);
+ return (*ptr)->GetPartitionDescSize(length, error);
+}
+
+ADBC_DRIVER_EXPORT
+AdbcStatusCode AdbcStatementGetStream(struct AdbcStatement* statement,
+ struct ArrowArrayStream* out,
+ struct AdbcError* error) {
+ if (!statement->private_data) return ADBC_STATUS_UNINITIALIZED;
+ auto* ptr =
+ reinterpret_cast<std::shared_ptr<FlightSqlStatementImpl>*>(statement->private_data);
+ return (*ptr)->GetStream(*ptr, out, error);
+}
+
+ADBC_DRIVER_EXPORT
+AdbcStatusCode AdbcStatementInit(struct AdbcConnection* connection,
+ struct AdbcStatement* statement,
+ struct AdbcError* error) {
+ auto impl = std::make_shared<FlightSqlStatementImpl>();
+ statement->private_data = new std::shared_ptr<FlightSqlStatementImpl>(impl);
+ return ADBC_STATUS_OK;
+}
+
+ADBC_DRIVER_EXPORT
+AdbcStatusCode AdbcStatementRelease(struct AdbcStatement* statement,
+ struct AdbcError* error) {
+ if (!statement->private_data) return ADBC_STATUS_UNINITIALIZED;
+ auto* ptr =
+ reinterpret_cast<std::shared_ptr<FlightSqlStatementImpl>*>(statement->private_data);
+ auto status = (*ptr)->Close(error);
+ delete ptr;
+ statement->private_data = nullptr;
+ return status;
+}
+
+extern "C" {
+ARROW_EXPORT
+AdbcStatusCode AdbcFlightSqlDriverInit(size_t count, struct AdbcDriver* driver,
+ size_t* initialized) {
+ if (count < ADBC_VERSION_0_0_1) return ADBC_STATUS_NOT_IMPLEMENTED;
+
+ std::memset(driver, 0, sizeof(*driver));
+ driver->ErrorRelease = AdbcErrorRelease;
+ driver->DatabaseInit = AdbcDatabaseInit;
+ driver->DatabaseRelease = AdbcDatabaseRelease;
+ driver->ConnectionInit = AdbcConnectionInit;
+ driver->ConnectionRelease = AdbcConnectionRelease;
+ driver->ConnectionSqlExecute = AdbcConnectionSqlExecute;
+ driver->ConnectionDeserializePartitionDesc = AdbcConnectionDeserializePartitionDesc;
+ driver->ConnectionGetTableTypes = AdbcConnectionGetTableTypes;
+ driver->StatementGetPartitionDesc = AdbcStatementGetPartitionDesc;
+ driver->StatementGetPartitionDescSize = AdbcStatementGetPartitionDescSize;
+ driver->StatementGetStream = AdbcStatementGetStream;
+ driver->StatementInit = AdbcStatementInit;
+ driver->StatementRelease = AdbcStatementRelease;
+ *initialized = ADBC_VERSION_0_0_1;
+ return ADBC_STATUS_OK;
+}
+}
diff --git a/drivers/flight_sql/flight_sql_test.cc b/drivers/flight_sql/flight_sql_test.cc
new file mode 100644
index 0000000..de4379e
--- /dev/null
+++ b/drivers/flight_sql/flight_sql_test.cc
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "adbc/adbc.h"
+#include "adbc/adbc_driver_manager.h"
+#include "adbc/test_util.h"
+#include "arrow/flight/sql/example/sqlite_server.h"
+#include "arrow/record_batch.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/matchers.h"
+#include "arrow/util/logging.h"
+
+namespace adbc {
+
+using arrow::PointeesEqual;
+
+class AdbcFlightSqlTest : public ::testing::Test {
+ public:
+ void SetUp() override {
+ {
+ arrow::flight::Location location;
+ ASSERT_OK(arrow::flight::Location::ForGrpcTcp("localhost", 0, &location));
+ arrow::flight::FlightServerOptions options(location);
+ ASSERT_OK_AND_ASSIGN(server,
+ arrow::flight::sql::example::SQLiteFlightSqlServer::Create());
+ ASSERT_OK(server->Init(options));
+ ASSERT_GT(server->port(), 0);
+ }
+
+ {
+ size_t initialized = 0;
+ ADBC_ASSERT_OK(
+ AdbcLoadDriver("Driver=libadbc_driver_flight_sql.so;"
+ "Entrypoint=AdbcFlightSqlDriverInit",
+ ADBC_VERSION_0_0_1, &driver, &initialized));
+ }
+
+ {
+ AdbcDatabaseOptions options;
+ std::string target = "Location=grpc://localhost:" + std::to_string(server->port());
+ options.target = target.c_str();
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.DatabaseInit(&options, &database, &error));
+ }
+
+ {
+ AdbcConnectionOptions options;
+ options.database = &database;
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.ConnectionInit(&options, &connection, &error));
+ }
+ }
+
+ void TearDown() override {
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.ConnectionRelease(&connection, &error));
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error, driver.DatabaseRelease(&database, &error));
+ ASSERT_OK(server->Shutdown());
+ }
+
+ protected:
+ std::shared_ptr<arrow::flight::sql::FlightSqlServerBase> server;
+ AdbcDriver driver;
+ AdbcDatabase database;
+ AdbcConnection connection;
+ AdbcError error = {};
+};
+
+TEST_F(AdbcFlightSqlTest, Metadata) {
+ {
+ AdbcStatement statement;
+ std::memset(&statement, 0, sizeof(statement));
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.StatementInit(&connection, &statement, &error));
+ ADBC_ASSERT_OK_WITH_ERROR(
+ &driver, error, driver.ConnectionGetTableTypes(&connection, &statement, &error));
+
+ std::shared_ptr<arrow::Schema> schema;
+ arrow::RecordBatchVector batches;
+ ReadStatement(&driver, &statement, &schema, &batches);
+ arrow::AssertSchemaEqual(
+ *schema,
+ *arrow::schema({arrow::field("table_type", arrow::utf8(), /*nullable=*/false)}));
+ EXPECT_THAT(batches, ::testing::UnorderedPointwise(
+ PointeesEqual(),
+ {
+ arrow::RecordBatchFromJSON(schema, R"([["table"]])"),
+ }));
+ }
+}
+
+TEST_F(AdbcFlightSqlTest, SqlExecute) {
+ std::string query = "SELECT 1";
+ AdbcStatement statement;
+ std::memset(&statement, 0, sizeof(statement));
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.StatementInit(&connection, &statement, &error));
+ ADBC_ASSERT_OK_WITH_ERROR(
+ &driver, error,
+ driver.ConnectionSqlExecute(&connection, query.c_str(), &statement, &error));
+
+ std::shared_ptr<arrow::Schema> schema;
+ arrow::RecordBatchVector batches;
+ ReadStatement(&driver, &statement, &schema, &batches);
+ arrow::AssertSchemaEqual(*schema, *arrow::schema({arrow::field("1", arrow::int64())}));
+ EXPECT_THAT(batches,
+ ::testing::UnorderedPointwise(
+ PointeesEqual(), {
+ arrow::RecordBatchFromJSON(schema, "[[1]]"),
+ }));
+}
+
+TEST_F(AdbcFlightSqlTest, Partitions) {
+ // Serialize the query result handle into a partition so it can be
+ // retrieved separately. (With multiple partitions we could
+ // distribute them across multiple machines or fetch data in
+ // parallel.)
+ std::string query = "SELECT 42";
+ AdbcStatement statement;
+ std::memset(&statement, 0, sizeof(statement));
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.StatementInit(&connection, &statement, &error));
+ ADBC_ASSERT_OK_WITH_ERROR(
+ &driver, error,
+ driver.ConnectionSqlExecute(&connection, query.c_str(), &statement, &error));
+
+ std::vector<std::vector<uint8_t>> descs;
+
+ while (true) {
+ size_t length = 0;
+ ADBC_ASSERT_OK_WITH_ERROR(
+ &driver, error,
+ driver.StatementGetPartitionDescSize(&statement, &length, &error));
+ if (length == 0) break;
+ descs.emplace_back(length);
+ ADBC_ASSERT_OK_WITH_ERROR(
+ &driver, error,
+ driver.StatementGetPartitionDesc(&statement, descs.back().data(), &error));
+ }
+
+ ASSERT_EQ(descs.size(), 1);
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error, driver.StatementRelease(&statement, &error));
+
+ // Reconstruct the partition
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.StatementInit(&connection, &statement, &error));
+ ADBC_ASSERT_OK_WITH_ERROR(
+ &driver, error,
+ driver.ConnectionDeserializePartitionDesc(&connection, descs.back().data(),
+ descs.back().size(), &statement, &error));
+
+ std::shared_ptr<arrow::Schema> schema;
+ arrow::RecordBatchVector batches;
+ ReadStatement(&driver, &statement, &schema, &batches);
+ arrow::AssertSchemaEqual(*schema, *arrow::schema({arrow::field("42", arrow::int64())}));
+ EXPECT_THAT(batches,
+ ::testing::UnorderedPointwise(
+ PointeesEqual(), {
+ arrow::RecordBatchFromJSON(schema, "[[42]]"),
+ }));
+}
+
+TEST_F(AdbcFlightSqlTest, InvalidSql) {
+ std::string query = "INVALID";
+ AdbcStatement statement;
+ std::memset(&statement, 0, sizeof(statement));
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.StatementInit(&connection, &statement, &error));
+ ASSERT_NE(driver.ConnectionSqlExecute(&connection, query.c_str(), &statement, &error),
+ ADBC_STATUS_OK);
+ ADBC_ASSERT_ERROR_THAT(&driver, error, ::testing::HasSubstr("syntax error"));
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error, driver.StatementRelease(&statement, &error));
+}
+
+} // namespace adbc
diff --git a/drivers/sqlite/sqlite.cc b/drivers/sqlite/sqlite.cc
new file mode 100644
index 0000000..b6a1cbc
--- /dev/null
+++ b/drivers/sqlite/sqlite.cc
@@ -0,0 +1,616 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sqlite3.h>
+
+#include <cstring>
+#include <memory>
+#include <mutex>
+#include <string>
+
+#include "adbc/adbc.h"
+#include "adbc/driver/util.h"
+#include "arrow/builder.h"
+#include "arrow/c/bridge.h"
+#include "arrow/record_batch.h"
+#include "arrow/status.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/string_builder.h"
+
+namespace {
+
+using arrow::Status;
+
+void SetError(sqlite3* db, const std::string& source, struct AdbcError* error) {
+ if (!error) return;
+ std::string message =
+ arrow::util::StringBuilder("[SQLite3] ", source, ": ", sqlite3_errmsg(db));
+ if (error->message) {
+ message.reserve(message.size() + 1 + std::strlen(error->message));
+ message.append(1, '\n');
+ message.append(error->message);
+ delete[] error->message;
+ }
+ error->message = new char[message.size() + 1];
+ message.copy(error->message, message.size());
+ error->message[message.size()] = '\0';
+}
+
+template <typename... Args>
+void SetError(struct AdbcError* error, Args&&... args) {
+ if (!error) return;
+ std::string message =
+ arrow::util::StringBuilder("[SQLite3] ", std::forward<Args>(args)...);
+ if (error->message) {
+ message.reserve(message.size() + 1 + std::strlen(error->message));
+ message.append(1, '\n');
+ message.append(error->message);
+ delete[] error->message;
+ }
+ error->message = new char[message.size() + 1];
+ message.copy(error->message, message.size());
+ error->message[message.size()] = '\0';
+}
+
+std::shared_ptr<arrow::Schema> StatementToSchema(sqlite3_stmt* stmt) {
+ const int num_columns = sqlite3_column_count(stmt);
+ arrow::FieldVector fields(num_columns);
+ for (int i = 0; i < num_columns; i++) {
+ const char* column_name = sqlite3_column_name(stmt, i);
+ const int column_type = sqlite3_column_type(stmt, i);
+ std::shared_ptr<arrow::DataType> arrow_type = nullptr;
+ switch (column_type) {
+ case SQLITE_INTEGER:
+ arrow_type = arrow::int64();
+ break;
+ case SQLITE_FLOAT:
+ arrow_type = arrow::float64();
+ break;
+ case SQLITE_BLOB:
+ arrow_type = arrow::binary();
+ break;
+ case SQLITE_TEXT:
+ arrow_type = arrow::utf8();
+ break;
+ case SQLITE_NULL:
+ default:
+ arrow_type = arrow::null();
+ break;
+ }
+ fields[i] = arrow::field(column_name, std::move(arrow_type));
+ }
+ return arrow::schema(std::move(fields));
+}
+
+class SqliteDatabaseImpl {
+ public:
+ explicit SqliteDatabaseImpl(sqlite3* db) : db_(db), connection_count_(0) {}
+
+ sqlite3* Connect() {
+ std::lock_guard<std::mutex> guard(mutex_);
+ ++connection_count_;
+ return db_;
+ }
+
+ AdbcStatusCode Disconnect(struct AdbcError* error) {
+ std::lock_guard<std::mutex> guard(mutex_);
+ if (--connection_count_ < 0) {
+ SetError(error, "Connection count underflow");
+ return ADBC_STATUS_INTERNAL;
+ }
+ return ADBC_STATUS_OK;
+ }
+
+ AdbcStatusCode Release(struct AdbcError* error) {
+ std::lock_guard<std::mutex> guard(mutex_);
+
+ if (connection_count_ > 0) {
+ SetError(error, "Cannot release database with ", connection_count_,
+ " open connections");
+ return ADBC_STATUS_INTERNAL;
+ }
+
+ auto status = sqlite3_close(db_);
+ if (status != SQLITE_OK) {
+ // TODO:
+ return ADBC_STATUS_UNKNOWN;
+ }
+ return ADBC_STATUS_OK;
+ }
+
+ private:
+ sqlite3* db_;
+ int connection_count_;
+ std::mutex mutex_;
+};
+
+class SqliteStatementImpl : public arrow::RecordBatchReader {
+ public:
+ SqliteStatementImpl()
+ : db_(nullptr), stmt_(nullptr), schema_(nullptr), bind_index_(0), done_(false) {}
+
+ void Init(sqlite3* db, sqlite3_stmt* stmt) {
+ db_ = db;
+ stmt_ = stmt;
+ }
+
+ //----------------------------------------------------------
+ // arrow::RecordBatchReader
+ //----------------------------------------------------------
+
+ std::shared_ptr<arrow::Schema> schema() const override {
+ DCHECK(schema_);
+ return schema_;
+ }
+
+ Status ReadNext(std::shared_ptr<arrow::RecordBatch>* batch) override {
+ constexpr int64_t kBatchSize = 1024;
+
+ if (done_) {
+ *batch = nullptr;
+ return Status::OK();
+ }
+
+ std::vector<std::unique_ptr<arrow::ArrayBuilder>> builders(schema_->num_fields());
+ for (int i = 0; static_cast<size_t>(i) < builders.size(); i++) {
+ // TODO: allow overriding memory pool
+ ARROW_RETURN_NOT_OK(arrow::MakeBuilder(arrow::default_memory_pool(),
+ schema_->field(i)->type(), &builders[i]));
+ }
+
+ // The statement was stepped once at the start, so step at the end of the loop
+ int64_t num_rows = 0;
+ for (int64_t row = 0; row < kBatchSize; row++) {
+ for (int col = 0; col < schema_->num_fields(); col++) {
+ const auto& field = schema_->field(col);
+ switch (field->type()->id()) {
+ case arrow::Type::INT64: {
+ const sqlite3_int64 value = sqlite3_column_int64(stmt_, col);
+ ARROW_RETURN_NOT_OK(
+ dynamic_cast<arrow::Int64Builder*>(builders[col].get())->Append(value));
+ break;
+ }
+ case arrow::Type::STRING: {
+ const char* value =
+ reinterpret_cast<const char*>(sqlite3_column_text(stmt_, col));
+ if (!value) {
+ ARROW_RETURN_NOT_OK(
+ dynamic_cast<arrow::StringBuilder*>(builders[col].get())->AppendNull());
+ } else {
+ const arrow::util::string_view view(value, std::strlen(value));
+ ARROW_RETURN_NOT_OK(dynamic_cast<arrow::StringBuilder*>(builders[col].get())
+ ->Append(value));
+ }
+ break;
+ }
+ default:
+ return Status::NotImplemented("[SQLite3] Cannot read field '", field->name(),
+ "' of type ", field->type()->ToString());
+ }
+ }
+ num_rows++;
+
+ int status = sqlite3_step(stmt_);
+ if (status == SQLITE_ROW) {
+ continue;
+ } else if (status == SQLITE_DONE) {
+ if (bind_parameters_ && bind_index_ < bind_parameters_->num_rows()) {
+ status = sqlite3_reset(stmt_);
+ if (status != SQLITE_OK) {
+ return Status::IOError("[SQLite3] sqlite3_reset: ", sqlite3_errmsg(db_));
+ }
+ RETURN_NOT_OK(BindNext());
+ status = sqlite3_step(stmt_);
+ if (status == SQLITE_ROW) continue;
+ } else {
+ done_ = true;
+ bind_parameters_.reset();
+ }
+ break;
+ }
+ return Status::IOError("[SQLite3] sqlite3_step: ", sqlite3_errmsg(db_));
+ }
+
+ arrow::ArrayVector arrays(builders.size());
+ for (size_t i = 0; i < builders.size(); i++) {
+ ARROW_RETURN_NOT_OK(builders[i]->Finish(&arrays[i]));
+ }
+ *batch = arrow::RecordBatch::Make(schema_, num_rows, std::move(arrays));
+ return Status::OK();
+ }
+
+ //----------------------------------------------------------
+ // Common Functions
+ //----------------------------------------------------------
+
+ AdbcStatusCode Close(struct AdbcError* error) {
+ if (stmt_) {
+ auto status = sqlite3_finalize(stmt_);
+ if (status != SQLITE_OK) {
+ SetError(db_, "sqlite3_finalize", error);
+ return ADBC_STATUS_UNKNOWN;
+ }
+ stmt_ = nullptr;
+ bind_parameters_.reset();
+ }
+ return ADBC_STATUS_OK;
+ }
+
+ //----------------------------------------------------------
+ // Statement Functions
+ //----------------------------------------------------------
+
+ AdbcStatusCode Bind(const std::shared_ptr<SqliteStatementImpl>& self,
+ struct ArrowArray* values, struct ArrowSchema* schema,
+ struct AdbcError* error) {
+ auto status = arrow::ImportRecordBatch(values, schema).Value(&bind_parameters_);
+ if (!status.ok()) {
+ SetError(error, status);
+ return ADBC_STATUS_INVALID_ARGUMENT;
+ }
+ return ADBC_STATUS_OK;
+ }
+
+ AdbcStatusCode Execute(const std::shared_ptr<SqliteStatementImpl>& self,
+ struct AdbcError* error) {
+ int rc = 0;
+ if (schema_) {
+ rc = sqlite3_clear_bindings(stmt_);
+ if (rc != SQLITE_OK) {
+ SetError(db_, "sqlite3_reset_bindings", error);
+ rc = sqlite3_finalize(stmt_);
+ if (rc != SQLITE_OK) {
+ SetError(db_, "sqlite3_finalize", error);
+ }
+ return ADBC_STATUS_IO;
+ }
+
+ rc = sqlite3_reset(stmt_);
+ if (rc != SQLITE_OK) {
+ SetError(db_, "sqlite3_reset", error);
+ rc = sqlite3_finalize(stmt_);
+ if (rc != SQLITE_OK) {
+ SetError(db_, "sqlite3_finalize", error);
+ }
+ return ADBC_STATUS_IO;
+ }
+ }
+ // Step the statement and get the schema (SQLite doesn't
+ // necessarily know the schema until it begins to execute it)
+ auto status = BindNext().Value(&rc);
+ // XXX: with parameters, inferring the schema from the first
+ // argument is inaccurate (what if one is null?). Is there a way
+ // to hint to SQLite the real type?
+ if (!status.ok()) {
+ // TODO: map Arrow codes to ADBC codes
+ SetError(error, status);
+ return ADBC_STATUS_IO;
+ } else if (rc != SQLITE_OK) {
+ SetError(db_, "sqlite3_bind", error);
+ rc = sqlite3_finalize(stmt_);
+ if (rc != SQLITE_OK) {
+ SetError(db_, "sqlite3_finalize", error);
+ }
+ return ADBC_STATUS_IO;
+ }
+ rc = sqlite3_step(stmt_);
+ if (rc == SQLITE_ERROR) {
+ SetError(db_, "sqlite3_step", error);
+ rc = sqlite3_finalize(stmt_);
+ if (rc != SQLITE_OK) {
+ SetError(db_, "sqlite3_finalize", error);
+ }
+ return ADBC_STATUS_IO;
+ }
+ schema_ = StatementToSchema(stmt_);
+ done_ = rc != SQLITE_ROW;
+ return ADBC_STATUS_OK;
+ }
+
+ AdbcStatusCode GetStream(const std::shared_ptr<SqliteStatementImpl>& self,
+ struct ArrowArrayStream* out, struct AdbcError* error) {
+ if (!stmt_ || !schema_) {
+ SetError(error, "Statement has not yet been executed");
+ return ADBC_STATUS_UNINITIALIZED;
+ }
+ auto status = arrow::ExportRecordBatchReader(self, out);
+ if (!status.ok()) {
+ SetError(error, "Could not initialize result reader: ", status);
+ return ADBC_STATUS_UNKNOWN;
+ }
+ return ADBC_STATUS_OK;
+ }
+
+ private:
+ arrow::Result<int> BindNext() {
+ if (!bind_parameters_ || bind_index_ >= bind_parameters_->num_rows()) {
+ return SQLITE_OK;
+ }
+
+ int col_index = 1;
+ // TODO: multiple output rows
+ const int bind_index = bind_index_++;
+ for (const auto& column : bind_parameters_->columns()) {
+ if (column->IsNull(bind_index)) {
+ const int rc = sqlite3_bind_null(stmt_, col_index);
+ if (rc != SQLITE_OK) return rc;
+ } else {
+ switch (column->type()->id()) {
+ case arrow::Type::INT64: {
+ const int rc = sqlite3_bind_int64(
+ stmt_, col_index,
+ static_cast<const arrow::Int64Array&>(*column).Value(bind_index));
+ if (rc != SQLITE_OK) return rc;
+ break;
+ }
+ case arrow::Type::STRING: {
+ const auto& strings = static_cast<const arrow::StringArray&>(*column);
+ const int rc = sqlite3_bind_text64(
+ stmt_, col_index, strings.Value(bind_index).data(),
+ strings.value_length(bind_index), SQLITE_STATIC, SQLITE_UTF8);
+ if (rc != SQLITE_OK) return rc;
+ break;
+ }
+ default:
+ return arrow::Status::NotImplemented("Binding parameter of type ",
+ *column->type());
+ }
+ }
+ col_index++;
+ }
+
+ return SQLITE_OK;
+ }
+
+ sqlite3* db_;
+ sqlite3_stmt* stmt_;
+ std::shared_ptr<arrow::Schema> schema_;
+ std::shared_ptr<arrow::RecordBatch> bind_parameters_;
+ int64_t bind_index_;
+ bool done_;
+};
+
+class SqliteConnectionImpl {
+ public:
+ explicit SqliteConnectionImpl(std::shared_ptr<SqliteDatabaseImpl> database)
+ : database_(std::move(database)), db_(database_->Connect()) {}
+
+ //----------------------------------------------------------
+ // Common Functions
+ //----------------------------------------------------------
+
+ AdbcStatusCode Release(struct AdbcError* error) { return database_->Disconnect(error); }
+
+ //----------------------------------------------------------
+ // SQL Semantics
+ //----------------------------------------------------------
+
+ AdbcStatusCode SqlExecute(const char* query, struct AdbcStatement* out,
+ struct AdbcError* error) {
+ auto status = SqlPrepare(query, out, error);
+ if (status != ADBC_STATUS_OK) return status;
+
+ return AdbcStatementExecute(out, error);
+ }
+
+ AdbcStatusCode SqlPrepare(const char* query, struct AdbcStatement* out,
+ struct AdbcError* error) {
+ if (!out->private_data) {
+ SetError(error, "Statement is uninitialized, use AdbcStatementInit");
+ return ADBC_STATUS_UNINITIALIZED;
+ }
+ auto* ptr =
+ reinterpret_cast<std::shared_ptr<SqliteStatementImpl>*>(out->private_data);
+ auto* impl = ptr->get();
+
+ sqlite3_stmt* stmt = nullptr;
+ auto rc = sqlite3_prepare_v2(db_, query, static_cast<int>(std::strlen(query)), &stmt,
+ /*pzTail=*/nullptr);
+ if (rc != SQLITE_OK) {
+ if (stmt) {
+ rc = sqlite3_finalize(stmt);
+ if (rc != SQLITE_OK) {
+ SetError(db_, "sqlite3_finalize", error);
+ return ADBC_STATUS_UNKNOWN;
+ }
+ }
+ SetError(db_, "sqlite3_prepare_v2", error);
+ return ADBC_STATUS_UNKNOWN;
+ }
+
+ impl->Init(db_, stmt);
+ return ADBC_STATUS_OK;
+ }
+
+ private:
+ std::shared_ptr<SqliteDatabaseImpl> database_;
+ sqlite3* db_;
+};
+
+} // namespace
+
+ADBC_DRIVER_EXPORT
+void AdbcErrorRelease(struct AdbcError* error) {
+ delete[] error->message;
+ error->message = nullptr;
+}
+
+ADBC_DRIVER_EXPORT
+AdbcStatusCode AdbcDatabaseInit(const struct AdbcDatabaseOptions* options,
+ struct AdbcDatabase* out, struct AdbcError* error) {
+ sqlite3* db = nullptr;
+ auto status = sqlite3_open_v2(
+ ":memory:", &db, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, /*zVfs=*/nullptr);
+ if (status != SQLITE_OK) {
+ if (db) {
+ SetError(db, "sqlite3_open_v2", error);
+ }
+ return ADBC_STATUS_UNKNOWN;
+ }
+
+ auto impl = std::make_shared<SqliteDatabaseImpl>(db);
+ out->private_data = new std::shared_ptr<SqliteDatabaseImpl>(impl);
+ return ADBC_STATUS_OK;
+}
+
+ADBC_DRIVER_EXPORT
+AdbcStatusCode AdbcDatabaseRelease(struct AdbcDatabase* database,
+ struct AdbcError* error) {
+ if (!database->private_data) return ADBC_STATUS_UNINITIALIZED;
+ auto ptr =
+ reinterpret_cast<std::shared_ptr<SqliteDatabaseImpl>*>(database->private_data);
+ AdbcStatusCode status = (*ptr)->Release(error);
+ delete ptr;
+ database->private_data = nullptr;
+ return status;
+}
+
+ADBC_DRIVER_EXPORT
+AdbcStatusCode AdbcConnectionInit(const struct AdbcConnectionOptions* options,
+ struct AdbcConnection* out, struct AdbcError* error) {
+ if (!options->database || !options->database->private_data) {
+ SetError(error, "Must provide database");
+ return ADBC_STATUS_INVALID_ARGUMENT;
+ }
+ auto ptr = reinterpret_cast<std::shared_ptr<SqliteDatabaseImpl>*>(
+ options->database->private_data);
+ auto impl = std::make_shared<SqliteConnectionImpl>(*ptr);
+ out->private_data = new std::shared_ptr<SqliteConnectionImpl>(impl);
+ return ADBC_STATUS_OK;
+}
+
+ADBC_DRIVER_EXPORT
+AdbcStatusCode AdbcConnectionRelease(struct AdbcConnection* connection,
+ struct AdbcError* error) {
+ if (!connection->private_data) return ADBC_STATUS_UNINITIALIZED;
+ auto ptr =
+ reinterpret_cast<std::shared_ptr<SqliteConnectionImpl>*>(connection->private_data);
+ AdbcStatusCode status = (*ptr)->Release(error);
+ delete ptr;
+ connection->private_data = nullptr;
+ return status;
+}
+
+ADBC_DRIVER_EXPORT
+AdbcStatusCode AdbcConnectionSqlExecute(struct AdbcConnection* connection,
+ const char* query, struct AdbcStatement* out,
+ struct AdbcError* error) {
+ if (!connection->private_data) return ADBC_STATUS_UNINITIALIZED;
+ auto* ptr =
+ reinterpret_cast<std::shared_ptr<SqliteConnectionImpl>*>(connection->private_data);
+ return (*ptr)->SqlExecute(query, out, error);
+}
+
+ADBC_DRIVER_EXPORT
+AdbcStatusCode AdbcConnectionSqlPrepare(struct AdbcConnection* connection,
+ const char* query, struct AdbcStatement* out,
+ struct AdbcError* error) {
+ if (!connection->private_data) return ADBC_STATUS_UNINITIALIZED;
+ auto* ptr =
+ reinterpret_cast<std::shared_ptr<SqliteConnectionImpl>*>(connection->private_data);
+ return (*ptr)->SqlPrepare(query, out, error);
+}
+
+ADBC_DRIVER_EXPORT
+AdbcStatusCode AdbcStatementGetPartitionDesc(struct AdbcStatement* statement,
+ uint8_t* partition_desc,
+ struct AdbcError* error) {
+ return ADBC_STATUS_NOT_IMPLEMENTED;
+}
+
+ADBC_DRIVER_EXPORT
+AdbcStatusCode AdbcStatementGetPartitionDescSize(struct AdbcStatement* statement,
+ size_t* length,
+ struct AdbcError* error) {
+ return ADBC_STATUS_NOT_IMPLEMENTED;
+}
+
+ADBC_DRIVER_EXPORT
+AdbcStatusCode AdbcStatementBind(struct AdbcStatement* statement,
+ struct ArrowArray* values, struct ArrowSchema* schema,
+ struct AdbcError* error) {
+ if (!statement->private_data) return ADBC_STATUS_UNINITIALIZED;
+ auto* ptr =
+ reinterpret_cast<std::shared_ptr<SqliteStatementImpl>*>(statement->private_data);
+ return (*ptr)->Bind(*ptr, values, schema, error);
+}
+
+ADBC_DRIVER_EXPORT
+AdbcStatusCode AdbcStatementExecute(struct AdbcStatement* statement,
+ struct AdbcError* error) {
+ if (!statement->private_data) return ADBC_STATUS_UNINITIALIZED;
+ auto* ptr =
+ reinterpret_cast<std::shared_ptr<SqliteStatementImpl>*>(statement->private_data);
+ return (*ptr)->Execute(*ptr, error);
+}
+
+ADBC_DRIVER_EXPORT
+AdbcStatusCode AdbcStatementGetStream(struct AdbcStatement* statement,
+ struct ArrowArrayStream* out,
+ struct AdbcError* error) {
+ if (!statement->private_data) return ADBC_STATUS_UNINITIALIZED;
+ auto* ptr =
+ reinterpret_cast<std::shared_ptr<SqliteStatementImpl>*>(statement->private_data);
+ return (*ptr)->GetStream(*ptr, out, error);
+}
+
+ADBC_DRIVER_EXPORT
+AdbcStatusCode AdbcStatementInit(struct AdbcConnection* connection,
+ struct AdbcStatement* statement,
+ struct AdbcError* error) {
+ auto impl = std::make_shared<SqliteStatementImpl>();
+ statement->private_data = new std::shared_ptr<SqliteStatementImpl>(impl);
+ return ADBC_STATUS_OK;
+}
+
+ADBC_DRIVER_EXPORT
+AdbcStatusCode AdbcStatementRelease(struct AdbcStatement* statement,
+ struct AdbcError* error) {
+ if (!statement->private_data) return ADBC_STATUS_UNINITIALIZED;
+ auto* ptr =
+ reinterpret_cast<std::shared_ptr<SqliteStatementImpl>*>(statement->private_data);
+ auto status = (*ptr)->Close(error);
+ delete ptr;
+ statement->private_data = nullptr;
+ return status;
+}
+
+extern "C" {
+ARROW_EXPORT
+AdbcStatusCode AdbcSqliteDriverInit(size_t count, struct AdbcDriver* driver,
+ size_t* initialized) {
+ if (count < ADBC_VERSION_0_0_1) return ADBC_STATUS_NOT_IMPLEMENTED;
+
+ std::memset(driver, 0, sizeof(*driver));
+ driver->ErrorRelease = AdbcErrorRelease;
+ driver->DatabaseInit = AdbcDatabaseInit;
+ driver->DatabaseRelease = AdbcDatabaseRelease;
+ driver->ConnectionInit = AdbcConnectionInit;
+ driver->ConnectionRelease = AdbcConnectionRelease;
+ driver->ConnectionSqlExecute = AdbcConnectionSqlExecute;
+ driver->ConnectionSqlPrepare = AdbcConnectionSqlPrepare;
+ driver->StatementBind = AdbcStatementBind;
+ driver->StatementExecute = AdbcStatementExecute;
+ driver->StatementGetPartitionDesc = AdbcStatementGetPartitionDesc;
+ driver->StatementGetPartitionDescSize = AdbcStatementGetPartitionDescSize;
+ driver->StatementGetStream = AdbcStatementGetStream;
+ driver->StatementInit = AdbcStatementInit;
+ driver->StatementRelease = AdbcStatementRelease;
+ *initialized = ADBC_VERSION_0_0_1;
+ return ADBC_STATUS_OK;
+}
+}
diff --git a/drivers/sqlite/sqlite_test.cc b/drivers/sqlite/sqlite_test.cc
new file mode 100644
index 0000000..9f088e0
--- /dev/null
+++ b/drivers/sqlite/sqlite_test.cc
@@ -0,0 +1,303 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "adbc/adbc.h"
+#include "adbc/adbc_driver_manager.h"
+#include "adbc/test_sqlite_internal.h"
+#include "adbc/test_util.h"
+#include "arrow/c/bridge.h"
+#include "arrow/record_batch.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/matchers.h"
+
+namespace adbc {
+
+using arrow::PointeesEqual;
+
+TEST(Adbc, Basics) {
+ AdbcDatabase database;
+ AdbcConnection connection;
+ AdbcError error = {};
+
+ AdbcDriver driver;
+ ASSERT_NO_FATAL_FAILURE(InitSqlite(&driver));
+
+ {
+ AdbcDatabaseOptions options;
+ std::memset(&options, 0, sizeof(options));
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.DatabaseInit(&options, &database, &error));
+ ASSERT_NE(database.private_data, nullptr);
+ }
+
+ {
+ AdbcConnectionOptions options;
+ std::memset(&options, 0, sizeof(options));
+ options.database = &database;
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.ConnectionInit(&options, &connection, &error));
+ ASSERT_NE(connection.private_data, nullptr);
+ }
+
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.ConnectionRelease(&connection, &error));
+ ASSERT_EQ(connection.private_data, nullptr);
+
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error, driver.DatabaseRelease(&database, &error));
+ ASSERT_EQ(database.private_data, nullptr);
+}
+
+TEST(Adbc, Errors) {
+ AdbcDriver driver;
+ size_t initialized = 0;
+ ASSERT_NE(ADBC_STATUS_OK, AdbcLoadDriver("Driver=libadbc_driver_fake.so",
+ ADBC_VERSION_0_0_1, &driver, &initialized));
+}
+
+TEST(AdbcSqlite, SqlExecute) {
+ // Execute a query with the SQLite example driver.
+ AdbcDatabase database;
+ AdbcConnection connection;
+ AdbcError error = {};
+
+ AdbcDriver driver;
+ ASSERT_NO_FATAL_FAILURE(InitSqlite(&driver));
+
+ {
+ AdbcDatabaseOptions options;
+ std::memset(&options, 0, sizeof(options));
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.DatabaseInit(&options, &database, &error));
+ }
+ {
+ AdbcConnectionOptions options;
+ std::memset(&options, 0, sizeof(options));
+ options.database = &database;
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.ConnectionInit(&options, &connection, &error));
+ }
+
+ {
+ std::string query = "SELECT 1";
+ AdbcStatement statement;
+ std::memset(&statement, 0, sizeof(statement));
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.StatementInit(&connection, &statement, &error));
+ ADBC_ASSERT_OK_WITH_ERROR(
+ &driver, error,
+ driver.ConnectionSqlExecute(&connection, query.c_str(), &statement, &error));
+
+ std::shared_ptr<arrow::Schema> schema;
+ arrow::RecordBatchVector batches;
+ ReadStatement(&driver, &statement, &schema, &batches);
+ arrow::AssertSchemaEqual(*schema,
+ *arrow::schema({arrow::field("1", arrow::int64())}));
+ EXPECT_THAT(batches,
+ ::testing::UnorderedPointwise(
+ PointeesEqual(), {
+ arrow::RecordBatchFromJSON(schema, "[[1]]"),
+ }));
+ }
+
+ {
+ std::string query = "INVALID";
+ AdbcStatement statement;
+ std::memset(&statement, 0, sizeof(statement));
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.StatementInit(&connection, &statement, &error));
+ ASSERT_NE(driver.ConnectionSqlExecute(&connection, query.c_str(), &statement, &error),
+ ADBC_STATUS_OK);
+ ADBC_ASSERT_ERROR_THAT(
+ &driver, error,
+ ::testing::AllOf(::testing::HasSubstr("[SQLite3] sqlite3_prepare_v2:"),
+ ::testing::HasSubstr("syntax error")));
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.StatementRelease(&statement, &error));
+ }
+
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.ConnectionRelease(&connection, &error));
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error, driver.DatabaseRelease(&database, &error));
+}
+
+TEST(AdbcSqlite, SqlPrepare) {
+ AdbcDatabase database;
+ AdbcConnection connection;
+ AdbcError error = {};
+
+ AdbcDriver driver;
+ ASSERT_NO_FATAL_FAILURE(InitSqlite(&driver));
+
+ {
+ AdbcDatabaseOptions options;
+ std::memset(&options, 0, sizeof(options));
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.DatabaseInit(&options, &database, &error));
+ }
+ {
+ AdbcConnectionOptions options;
+ std::memset(&options, 0, sizeof(options));
+ options.database = &database;
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.ConnectionInit(&options, &connection, &error));
+ }
+
+ {
+ std::string query = "SELECT 1";
+ AdbcStatement statement;
+ std::memset(&statement, 0, sizeof(statement));
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.StatementInit(&connection, &statement, &error));
+ ADBC_ASSERT_OK_WITH_ERROR(
+ &driver, error,
+ driver.ConnectionSqlPrepare(&connection, query.c_str(), &statement, &error));
+
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.StatementExecute(&statement, &error));
+
+ std::shared_ptr<arrow::Schema> schema;
+ arrow::RecordBatchVector batches;
+ ASSERT_NO_FATAL_FAILURE(ReadStatement(&driver, &statement, &schema, &batches));
+ arrow::AssertSchemaEqual(*schema,
+ *arrow::schema({arrow::field("1", arrow::int64())}));
+ EXPECT_THAT(batches,
+ ::testing::UnorderedPointwise(
+ PointeesEqual(), {
+ arrow::RecordBatchFromJSON(schema, "[[1]]"),
+ }));
+ }
+
+ {
+ auto param_schema = arrow::schema(
+ {arrow::field("1", arrow::int64()), arrow::field("2", arrow::utf8())});
+ std::string query = "SELECT ?, ?";
+ AdbcStatement statement;
+ ArrowArray export_params;
+ ArrowSchema export_schema;
+ std::memset(&statement, 0, sizeof(statement));
+
+ ASSERT_OK(ExportRecordBatch(
+ *arrow::RecordBatchFromJSON(param_schema, R"([[1, "foo"], [2, "bar"]])"),
+ &export_params));
+ ASSERT_OK(ExportSchema(*param_schema, &export_schema));
+
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.StatementInit(&connection, &statement, &error));
+ ADBC_ASSERT_OK_WITH_ERROR(
+ &driver, error,
+ driver.ConnectionSqlPrepare(&connection, query.c_str(), &statement, &error));
+
+ ADBC_ASSERT_OK_WITH_ERROR(
+ &driver, error,
+ driver.StatementBind(&statement, &export_params, &export_schema, &error));
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.StatementExecute(&statement, &error));
+
+ std::shared_ptr<arrow::Schema> schema;
+ arrow::RecordBatchVector batches;
+ ASSERT_NO_FATAL_FAILURE(ReadStatement(&driver, &statement, &schema, &batches));
+ arrow::AssertSchemaEqual(*schema, *arrow::schema({arrow::field("?", arrow::int64()),
+ arrow::field("?", arrow::utf8())}));
+ EXPECT_THAT(batches,
+ ::testing::UnorderedPointwise(
+ PointeesEqual(),
+ {
+ arrow::RecordBatchFromJSON(schema, R"([[1, "foo"], [2, "bar"]])"),
+ }));
+ }
+
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.ConnectionRelease(&connection, &error));
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error, driver.DatabaseRelease(&database, &error));
+}
+
+TEST(AdbcSqlite, MultipleConnections) {
+ // Execute a query with the SQLite example driver.
+ AdbcDatabase database;
+ AdbcConnection connection1, connection2;
+ AdbcError error = {};
+
+ AdbcDriver driver;
+ ASSERT_NO_FATAL_FAILURE(InitSqlite(&driver));
+
+ {
+ AdbcDatabaseOptions options;
+ std::memset(&options, 0, sizeof(options));
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.DatabaseInit(&options, &database, &error));
+ }
+ {
+ AdbcConnectionOptions options;
+ std::memset(&options, 0, sizeof(options));
+ options.database = &database;
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.ConnectionInit(&options, &connection1, &error));
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.ConnectionInit(&options, &connection2, &error));
+ }
+
+ {
+ std::string query = "CREATE TABLE foo (bar INTEGER)";
+ AdbcStatement statement;
+ std::memset(&statement, 0, sizeof(statement));
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.StatementInit(&connection1, &statement, &error));
+ ADBC_ASSERT_OK_WITH_ERROR(
+ &driver, error,
+ driver.ConnectionSqlExecute(&connection1, query.c_str(), &statement, &error));
+
+ std::shared_ptr<arrow::Schema> schema;
+ arrow::RecordBatchVector batches;
+ ASSERT_NO_FATAL_FAILURE(ReadStatement(&driver, &statement, &schema, &batches));
+ arrow::AssertSchemaEqual(*schema, *arrow::schema({}));
+ EXPECT_THAT(batches,
+ ::testing::UnorderedPointwise(
+ PointeesEqual(), std::vector<std::shared_ptr<arrow::RecordBatch>>{}));
+ }
+
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.ConnectionRelease(&connection1, &error));
+
+ {
+ std::string query = "SELECT * FROM foo";
+ AdbcStatement statement;
+ std::memset(&statement, 0, sizeof(statement));
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.StatementInit(&connection2, &statement, &error));
+ ADBC_ASSERT_OK_WITH_ERROR(
+ &driver, error,
+ driver.ConnectionSqlExecute(&connection2, query.c_str(), &statement, &error));
+
+ std::shared_ptr<arrow::Schema> schema;
+ arrow::RecordBatchVector batches;
+ ReadStatement(&driver, &statement, &schema, &batches);
+ arrow::AssertSchemaEqual(*schema,
+ *arrow::schema({arrow::field("bar", arrow::null())}));
+ EXPECT_THAT(batches,
+ ::testing::UnorderedPointwise(
+ PointeesEqual(), std::vector<std::shared_ptr<arrow::RecordBatch>>{}));
+ }
+
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error,
+ driver.ConnectionRelease(&connection2, &error));
+ ADBC_ASSERT_OK_WITH_ERROR(&driver, error, driver.DatabaseRelease(&database, &error));
+}
+
+} // namespace adbc
diff --git a/drivers/sqlite/test_sqlite_internal.cc b/drivers/sqlite/test_sqlite_internal.cc
new file mode 100644
index 0000000..8252df5
--- /dev/null
+++ b/drivers/sqlite/test_sqlite_internal.cc
@@ -0,0 +1,30 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "adbc/test_sqlite_internal.h"
+
+namespace adbc {
+
+void InitSqlite(AdbcDriver* driver) {
+ size_t initialized = 0;
+ ADBC_ASSERT_OK(
+ AdbcLoadDriver("Driver=libadbc_driver_sqlite.so;Entrypoint=AdbcSqliteDriverInit",
+ ADBC_VERSION_0_0_1, driver, &initialized));
+ ASSERT_EQ(initialized, ADBC_VERSION_0_0_1);
+}
+
+} // namespace adbc
diff --git a/drivers/sqlite/test_sqlite_internal.h b/drivers/sqlite/test_sqlite_internal.h
new file mode 100644
index 0000000..3adaef5
--- /dev/null
+++ b/drivers/sqlite/test_sqlite_internal.h
@@ -0,0 +1,32 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <gtest/gtest.h>
+
+#include "adbc/adbc.h"
+#include "adbc/adbc_driver_manager.h"
+#include "adbc/test_util.h"
+#include "arrow/util/visibility.h"
+
+namespace adbc {
+
+ARROW_EXPORT
+void InitSqlite(AdbcDriver* driver);
+
+} // namespace adbc
diff --git a/drivers/test_util.h b/drivers/test_util.h
new file mode 100644
index 0000000..adf4abb
--- /dev/null
+++ b/drivers/test_util.h
@@ -0,0 +1,74 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <memory>
+
+#include "adbc/adbc.h"
+#include "arrow/c/bridge.h"
+#include "arrow/record_batch.h"
+#include "arrow/result.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/util/macros.h"
+
+namespace adbc {
+
+#define ADBC_ASSERT_OK(expr) \
+ do { \
+ auto code_ = (expr); \
+ ASSERT_EQ(code_, ADBC_STATUS_OK); \
+ } while (false)
+
+#define ADBC_ASSERT_OK_WITH_ERROR(DRIVER, ERROR, EXPR) \
+ do { \
+ auto code_ = (EXPR); \
+ if (code_ != ADBC_STATUS_OK) { \
+ std::string errmsg_ = ERROR.message ? ERROR.message : "(unknown error)"; \
+ (DRIVER)->ErrorRelease(&ERROR); \
+ ASSERT_EQ(code_, ADBC_STATUS_OK) << errmsg_; \
+ } \
+ } while (false)
+
+#define ADBC_ASSERT_ERROR_THAT(DRIVER, ERROR, PATTERN) \
+ do { \
+ ASSERT_NE(ERROR.message, nullptr); \
+ std::string errmsg_ = ERROR.message ? ERROR.message : "(unknown error)"; \
+ (DRIVER)->ErrorRelease(&ERROR); \
+ ASSERT_THAT(errmsg_, PATTERN) << errmsg_; \
+ } while (false)
+
+static inline void ReadStatement(AdbcDriver* driver, AdbcStatement* statement,
+ std::shared_ptr<arrow::Schema>* schema,
+ arrow::RecordBatchVector* batches) {
+ AdbcError error = {};
+ ArrowArrayStream stream;
+ ADBC_ASSERT_OK_WITH_ERROR(driver, error,
+ driver->StatementGetStream(statement, &stream, &error));
+ ASSERT_OK_AND_ASSIGN(auto reader, arrow::ImportRecordBatchReader(&stream));
+
+ *schema = reader->schema();
+
+ while (true) {
+ ASSERT_OK_AND_ASSIGN(auto batch, reader->Next());
+ if (!batch) break;
+ batches->push_back(std::move(batch));
+ }
+ ADBC_ASSERT_OK_WITH_ERROR(driver, error, driver->StatementRelease(statement, &error));
+}
+
+} // namespace adbc
diff --git a/drivers/util.cc b/drivers/util.cc
new file mode 100644
index 0000000..fe71727
--- /dev/null
+++ b/drivers/util.cc
@@ -0,0 +1,59 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "adbc/driver/util.h"
+#include "arrow/status.h"
+#include "arrow/util/string_view.h"
+
+namespace adbc {
+
+using arrow::util::string_view;
+
+arrow::Result<std::unordered_map<std::string, std::string>> ParseConnectionString(
+ string_view target) {
+ // TODO: this does not properly implement the ODBC connection string format.
+ std::unordered_map<std::string, std::string> option_pairs;
+ string_view cur(target);
+
+ while (true) {
+ auto divider = cur.find_first_of('=');
+ if (divider == string_view::npos) {
+ if (!cur.empty()) {
+ return arrow::Status::Invalid("Trailing characters: ", cur);
+ }
+ break;
+ }
+ string_view key = cur.substr(0, divider);
+ cur = cur.substr(divider + 1);
+ auto end = cur.find_first_of(';');
+ string_view value;
+ if (end == string_view::npos) {
+ value = cur;
+ cur = string_view();
+ } else {
+ value = cur.substr(0, end);
+ cur = cur.substr(end + 1);
+ }
+
+ if (!option_pairs.insert({std::string(key), std::string(value)}).second) {
+ return arrow::Status::Invalid("Duplicate option: ", key);
+ }
+ }
+ return option_pairs;
+}
+
+} // namespace adbc
diff --git a/drivers/util.h b/drivers/util.h
new file mode 100644
index 0000000..7552dfd
--- /dev/null
+++ b/drivers/util.h
@@ -0,0 +1,43 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <string>
+#include <unordered_map>
+
+#include "arrow/result.h"
+#include "arrow/util/string_view.h"
+
+// XXX: We want to export Adbc* functions, but inside
+// AdbcSqliteDriverInit, we want to always point to the local
+// function, not the global function (so we can cooperate with the
+// driver manager). "protected" visibility gives us this.
+// RTLD_DEEPBIND also works but is glibc-specific and does not work
+// with AddressSanitizer.
+// TODO: should this go in adbc.h instead?
+#ifdef __linux__
+#define ADBC_DRIVER_EXPORT __attribute__((visibility("protected")))
+#else
+#define ADBC_DRIVER_EXPORT
+#endif // ifdef __linux__
+
+namespace adbc {
+
+/// \brief Parse a connection string.
+arrow::Result<std::unordered_map<std::string, std::string>> ParseConnectionString(
+ arrow::util::string_view target);
+
+} // namespace adbc