You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by GitBox <gi...@apache.org> on 2022/12/16 20:13:36 UTC
[GitHub] [arrow-site] lidavidm commented on a diff in pull request #288: [Website] WIP: Add nanoarrow blog post
lidavidm commented on code in PR #288:
URL: https://github.com/apache/arrow-site/pull/288#discussion_r1051121802
##########
_posts/2022-12-14-nanoarrow.md:
##########
@@ -0,0 +1,186 @@
+---
+layout: post
+title: "Introducing nanoarrow"
+date: "2022-12-14 00:00:00"
+author: paleolimbot
+categories: [application]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+The adoption of the
+[Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html)
+and the [Arrow C Stream Interface](https://arrow.apache.org/docs/format/CStreamInterface.html)
+since their
+[introduction](https://arrow.apache.org/blog/2020/05/03/introducing-arrow-c-data-interface/)
+have been impressive and enthusiastic: not only have Arrow language bindings
+adopted the standard to pass data among themselves, a growing number of
+high-profile libraries like
+[GDAL](https://gdal.org/development/rfc/rfc86_column_oriented_api.html) and
+[DuckDB](https://duckdb.org/2021/12/03/duck-arrow.html) use the standard to
+improve performance and provide an ABI-stable interface to tabular input and output.
+
+GDAL and DuckDB are fortunate to have hard-working and forward-thinking maintainers
+that were motivated to provide support for the Arrow C Data and Stream interfaces
+even though the code to do so required an intimate knowledge of both the interface
+and the columnar specification on which it is based.
+
+The vision of the [nanoarrow](https://github.com/apache/arrow-nanoarrow) project
+is that it should be trivial for a library or application to implement an Arrow-based
+interface: if a library consumes or produces tabular data, Arrow should be the
+first place developers look. Developers shouldn't have to be familiar with the
+details of the columnar specification---nor should they have to take on any
+build-time dependencies---to get started.
+
+The [Arrow Database Connectivity (ADBC)](https://arrow.apache.org/docs/format/ADBC.html)
+specification is a good example of such a project, and provided a strong
+motivator for the development of nanoarrow: at the heart of ADBC is the
+idea of a core "driver manager" and database-specific drivers that are distributed
+as independent C/C++/Python/R/Java/Go projects. At least in R and Python, the
+embedding an existing Arrow implementation (e.g., Arrow C++) is challenging
+in the context of multiple packages intended to be loaded into the same process.
+As of this writing, ADBC includes nanoarrow-based SQLite and PostgreSQL drivers
+and a nanoarrow-based validation suite for drivers.
+
+## Using nanoarrow in C
+
+The nanoarrow C library is distributed as
+[two files (nanoarrow.h and nanoarrow.c)](https://github.com/apache/arrow-nanoarrow/tree/main/dist)
+that can be copied and vendored into an existing code base. This results in
+a static library of about 50 KB and builds in less than a second. In addition to
+creating an array directly from buffers (for those familiar with the columnar
+specification), nanoarrow includes an
+[API for building arrays element-wise](https://apache.github.io/arrow-nanoarrow/dev/c.html#creating-arrays):
+
+```c
+#include "nanoarrow.h"
+
+int make_simple_array(struct ArrowArray* array_out, struct ArrowSchema* schema_out) {
+ struct ArrowError error;
+ array_out->release = NULL;
+ schema_out->release = NULL;
+
+ NANOARROW_RETURN_NOT_OK(ArrowArrayInit(array_out, NANOARROW_TYPE_INT32));
+
+ NANOARROW_RETURN_NOT_OK(ArrowArrayStartAppending(array_out));
+ NANOARROW_RETURN_NOT_OK(ArrowArrayAppendInt(array_out, 1));
+ NANOARROW_RETURN_NOT_OK(ArrowArrayAppendInt(array_out, 2));
+ NANOARROW_RETURN_NOT_OK(ArrowArrayAppendInt(array_out, 3));
+ NANOARROW_RETURN_NOT_OK(ArrowArrayFinishBuilding(array_out, &error));
+
+ NANOARROW_RETURN_NOT_OK(ArrowSchemaInit(schema_out, NANOARROW_TYPE_INT32));
+
+ return NANOARROW_OK;
+}
+```
+
+Similarly, nanoarrow provides an
+[API to extract elements element-wise](https://apache.github.io/arrow-nanoarrow/dev/c.html#reading-arrays)
+from an existing array.
+
+```c
+#include <stdio.h>
+#include "nanoarrow.h"
+
+int print_simple_array(struct ArrowArray* array, struct ArrowSchema* schema) {
+ struct ArrowError error;
+ struct ArrowArrayView array_view;
+ NANOARROW_RETURN_NOT_OK(ArrowArrayViewInitFromSchema(&array_view, schema, &error));
+
+ if (array_view.storage_type != NANOARROW_TYPE_INT32) {
+ printf("Array has storage that is not int32\n");
+ }
+
+ int result = ArrowArrayViewSetArray(&array_view, array, &error);
+ if (result != NANOARROW_OK) {
+ ArrowArrayViewReset(&array_view);
+ return result;
+ }
+
+ for (int64_t i = 0; i < array->length; i++) {
+ printf("%d\n", (int)ArrowArrayViewGetIntUnsafe(&array_view, i));
+ }
+
+ ArrowArrayViewReset(&array_view);
+ return NANOARROW_OK;
+}
+```
+
+## Using nanoarrow in C++, R, and Python
+
+Recognizing that many projects for which nanoarrow may be useful will have
+access a higher-level runtime than C, there are experiments to provide
+these users with a minimal set of useful tools.
+
+For C++ projects, an experimental
+["nanoarrow.hpp"](https://apache.github.io/arrow-nanoarrow/dev/cpp.html)
+interface provides `unique_ptr`-like wrappers for nanoarrow C objects to
+reduce the verbosity of using the nanoarrow API. For example, the previous
+`print_simple_array()` implementation would collapse to:
+
+```cpp
+#include <stdio.h>
+#include "nanoarrow.hpp"
+
+int print_simple_array2(struct ArrowArray* array, struct ArrowSchema* schema) {
+ struct ArrowError error;
+ nanoarrow::UniqueArrayView array_view;
+ NANOARROW_RETURN_NOT_OK(ArrowArrayViewInitFromSchema(array_view.get(), schema, &error));
+ NANOARROW_RETURN_NOT_OK(ArrowArrayViewSetArray(array_view.get(), array, &error));
+ for (int64_t i = 0; i < array->length; i++) {
+ printf("%d\n", (int)ArrowArrayViewGetIntUnsafe(array_view.get(), i));
+ }
+ return NANOARROW_OK;
+}
+```
+
+For R packages, experimental
+[R bindings](https://apache.github.io/arrow-nanoarrow/dev/r/index.html) provide
+a limited set of conversions between R vectors and Arrow arrays such that
+R bindings for a library with an Arrow-based interface do not need to provide
+this behaviour themselves. Additional features include printing and validating
+the content of the C structures at the heart of the C Data and C Stream
+interfaces to facilitate the development of bindings to Arrow-based libraries.
+
+```r
+# install.packages("remotes")
+remotes::install_github("apache/arrow-nanoarrow/r", build = FALSE)
+library(nanoarrow)
+
+as_nanoarrow_array(1:5)
+#> <nanoarrow_array int32[5]>
+#> $ length : int 5
+#> $ null_count: int 0
+#> $ offset : int 0
+#> $ buffers :List of 2
+#> ..$ :<nanoarrow_buffer_validity[0 b] at 0x0>
+#> ..$ :<nanoarrow_buffer_data_int32[20 b] at 0x135d13c28>
+#> $ dictionary: NULL
+#> $ children : list()
+```
+
+A [Python package skeleton](https://github.com/apache/arrow-nanoarrow/tree/main/python)
+exists in the nanoarrow repository and further functionality may be added once
+the C library interface has stabilized.
+
+## Development roadmap
+
+The nanoarrow library is very new and everything about it is experimental
+and contingent on user feedback! An initial 0.1 release is planned for January 2023
+to facilitate this feedback.
Review Comment:
Maybe a bit stronger of a call to action? Repeat the download link and link to the issue tracker?
##########
_posts/2022-12-14-nanoarrow.md:
##########
@@ -0,0 +1,186 @@
+---
+layout: post
+title: "Introducing nanoarrow"
+date: "2022-12-14 00:00:00"
+author: paleolimbot
+categories: [application]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+The adoption of the
+[Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html)
+and the [Arrow C Stream Interface](https://arrow.apache.org/docs/format/CStreamInterface.html)
+since their
+[introduction](https://arrow.apache.org/blog/2020/05/03/introducing-arrow-c-data-interface/)
+have been impressive and enthusiastic: not only have Arrow language bindings
+adopted the standard to pass data among themselves, a growing number of
+high-profile libraries like
+[GDAL](https://gdal.org/development/rfc/rfc86_column_oriented_api.html) and
+[DuckDB](https://duckdb.org/2021/12/03/duck-arrow.html) use the standard to
+improve performance and provide an ABI-stable interface to tabular input and output.
+
+GDAL and DuckDB are fortunate to have hard-working and forward-thinking maintainers
+that were motivated to provide support for the Arrow C Data and Stream interfaces
+even though the code to do so required an intimate knowledge of both the interface
+and the columnar specification on which it is based.
+
+The vision of the [nanoarrow](https://github.com/apache/arrow-nanoarrow) project
+is that it should be trivial for a library or application to implement an Arrow-based
+interface: if a library consumes or produces tabular data, Arrow should be the
+first place developers look. Developers shouldn't have to be familiar with the
+details of the columnar specification---nor should they have to take on any
+build-time dependencies---to get started.
+
+The [Arrow Database Connectivity (ADBC)](https://arrow.apache.org/docs/format/ADBC.html)
+specification is a good example of such a project, and provided a strong
+motivator for the development of nanoarrow: at the heart of ADBC is the
+idea of a core "driver manager" and database-specific drivers that are distributed
+as independent C/C++/Python/R/Java/Go projects. At least in R and Python, the
+embedding an existing Arrow implementation (e.g., Arrow C++) is challenging
Review Comment:
```suggestion
as independent C/C++/Python/R/Java/Go projects. At least in R and Python,
embedding an existing Arrow implementation (e.g., Arrow C++) is challenging
```
##########
_posts/2022-12-14-nanoarrow.md:
##########
@@ -0,0 +1,186 @@
+---
+layout: post
+title: "Introducing nanoarrow"
+date: "2022-12-14 00:00:00"
+author: paleolimbot
+categories: [application]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+The adoption of the
+[Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html)
+and the [Arrow C Stream Interface](https://arrow.apache.org/docs/format/CStreamInterface.html)
+since their
+[introduction](https://arrow.apache.org/blog/2020/05/03/introducing-arrow-c-data-interface/)
+have been impressive and enthusiastic: not only have Arrow language bindings
+adopted the standard to pass data among themselves, a growing number of
+high-profile libraries like
+[GDAL](https://gdal.org/development/rfc/rfc86_column_oriented_api.html) and
+[DuckDB](https://duckdb.org/2021/12/03/duck-arrow.html) use the standard to
+improve performance and provide an ABI-stable interface to tabular input and output.
+
+GDAL and DuckDB are fortunate to have hard-working and forward-thinking maintainers
+that were motivated to provide support for the Arrow C Data and Stream interfaces
+even though the code to do so required an intimate knowledge of both the interface
+and the columnar specification on which it is based.
+
+The vision of the [nanoarrow](https://github.com/apache/arrow-nanoarrow) project
+is that it should be trivial for a library or application to implement an Arrow-based
+interface: if a library consumes or produces tabular data, Arrow should be the
+first place developers look. Developers shouldn't have to be familiar with the
+details of the columnar specification---nor should they have to take on any
+build-time dependencies---to get started.
Review Comment:
Maybe be a bit more concrete about what nanoarrow actually provides up front?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org