You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by al...@apache.org on 2023/05/04 10:36:35 UTC
[arrow-site] 01/01: Update date on cloud query post
This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch blog/cloudquery
in repository https://gitbox.apache.org/repos/asf/arrow-site.git
commit 9852b90a855436c706f59fb6c842ff327adcdd8e
Author: Andrew Lamb <an...@nerdnetworks.org>
AuthorDate: Thu May 4 06:33:17 2023 -0400
Update date on cloud query post
---
...udquery.md => 2023-05-04-adopting-apache-arrow-at-cloudquery.md} | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/_posts/2023-04-24-adopting-apache-arrow-at-cloudquery.md b/_posts/2023-05-04-adopting-apache-arrow-at-cloudquery.md
similarity index 98%
rename from _posts/2023-04-24-adopting-apache-arrow-at-cloudquery.md
rename to _posts/2023-05-04-adopting-apache-arrow-at-cloudquery.md
index cd326face08..94810618565 100644
--- a/_posts/2023-04-24-adopting-apache-arrow-at-cloudquery.md
+++ b/_posts/2023-05-04-adopting-apache-arrow-at-cloudquery.md
@@ -1,7 +1,7 @@
---
layout: post
title: "Adopting Apache Arrow at CloudQuery"
-date: "2023-04-30 00:00:00"
+date: "2023-05-04 00:00:00"
author: Yevgeny Pats
categories: [application]
---
@@ -55,11 +55,11 @@ Before Arrow, we used our own type system that supported more than 14 types. Thi
This is where Arrow comes in. Apache Arrow defines a language-independent columnar format for flat and hierarchical data, and brings the following advantages:
1. Cross-language with extensive libraries for different languages - The [format](https://arrow.apache.org/docs/format/Columnar.html) is defined via flatbuffers in such way that you can parse it in any language and already has extensive support in C/C++, C#, Go, Java, JavaScript, Julia, Matlab, Python, R, Ruby and Rust (at the time of writing). For CloudQuery this is important as it makes it much easier to develop source or destination plugins in different languages.
-2. Performance: Arrow adoption is rising especially in columnar based databases ([DuckDB](https://duckdb.org/2021/12/03/duck-arrow.html), [ClickHouse](https://clickhouse.com/docs/en/integrations/data-formats/arrow-avro-orc), [BigQuery](https://cloud.google.com/bigquery/docs/samples/bigquerystorage-arrow-quickstart)) and file formats ([Parquet](https://arrow.apache.org/docs/python/parquet.html)) which makes it easier to write CloudQuery destination or source plugins for databases that alr [...]
+2. Performance: Arrow adoption is rising especially in columnar based databases ([DuckDB](https://duckdb.org/2021/12/03/duck-arrow.html), [ClickHouse](https://clickhouse.com/docs/en/integrations/data-formats/arrow-avro-orc), [BigQuery](https://cloud.google.com/bigquery/docs/samples/bigquerystorage-arrow-quickstart)) and file formats ([Parquet](https://arrow.apache.org/docs/python/parquet.html)) which makes it easier to write CloudQuery destination or source plugins for databases that alr [...]
3. Rich Data Types: Arrow supports more than [35 types](https://arrow.apache.org/docs/python/api/datatypes.html) including composite types (i.e. lists, structs and maps of all the available types) and ability to extend the type system with custom types. Also, there is already built-in mapping from/to the arrow type system and the parquet type system (including nested types) which already supported in many of the arrow libraries as explained [here](https://arrow.apache.org/blog/2022/10/08 [...]
# Summary
Adopting Apache Arrow as the CloudQuery in-memory type system enables us to gain better performance, data interoperability and developer experience. Some plugins that are going to gain an immediate boost of rich type systems are our database-to-database replication plugins such as [PostgreSQL CDC](https://www.cloudquery.io/docs/plugins/sources/postgresql/overview) source plugin (and all [database destinations](https://www.cloudquery.io/docs/plugins/destinations/overview)) that are going [...]
-We are excited about this step and joining the growing Arrow community. We already contributed more than [30](https://github.com/search?q=is%3Apr+author%3Ayevgenypats+author%3Ahermanschaaf+author%3Acandiduslynx+author%3Adisq+label%3A%22Component%3A+Go%22++is%3Amerged+&ref=simplesearch) upstream pull requests that were quickly reviewed by the Arrow maintainers, thank you!
\ No newline at end of file
+We are excited about this step and joining the growing Arrow community. We already contributed more than [30](https://github.com/search?q=is%3Apr+author%3Ayevgenypats+author%3Ahermanschaaf+author%3Acandiduslynx+author%3Adisq+label%3A%22Component%3A+Go%22++is%3Amerged+&ref=simplesearch) upstream pull requests that were quickly reviewed by the Arrow maintainers, thank you!