You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by al...@apache.org on 2023/05/04 10:34:07 UTC
[arrow-site] branch blog/cloudquery created (now cf519556be4)
This is an automated email from the ASF dual-hosted git repository.
alamb pushed a change to branch blog/cloudquery
in repository https://gitbox.apache.org/repos/asf/arrow-site.git
at cf519556be4 Merge remote-tracking branch 'origin/main' into blog/cloudquery
This branch includes the following new commits:
new 569349fe5f4 Update date
new cf519556be4 Merge remote-tracking branch 'origin/main' into blog/cloudquery
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[arrow-site] 01/02: Update date
Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch blog/cloudquery
in repository https://gitbox.apache.org/repos/asf/arrow-site.git
commit 569349fe5f481aab2067b7237f75e45ba3e04cb5
Author: Andrew Lamb <an...@nerdnetworks.org>
AuthorDate: Thu May 4 06:33:17 2023 -0400
Update date
---
...udquery.md => 2023-05-04-adopting-apache-arrow-at-cloudquery.md} | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/_posts/2023-04-24-adopting-apache-arrow-at-cloudquery.md b/_posts/2023-05-04-adopting-apache-arrow-at-cloudquery.md
similarity index 98%
rename from _posts/2023-04-24-adopting-apache-arrow-at-cloudquery.md
rename to _posts/2023-05-04-adopting-apache-arrow-at-cloudquery.md
index cd326face08..94810618565 100644
--- a/_posts/2023-04-24-adopting-apache-arrow-at-cloudquery.md
+++ b/_posts/2023-05-04-adopting-apache-arrow-at-cloudquery.md
@@ -1,7 +1,7 @@
---
layout: post
title: "Adopting Apache Arrow at CloudQuery"
-date: "2023-04-30 00:00:00"
+date: "2023-05-04 00:00:00"
author: Yevgeny Pats
categories: [application]
---
@@ -55,11 +55,11 @@ Before Arrow, we used our own type system that supported more than 14 types. Thi
This is where Arrow comes in. Apache Arrow defines a language-independent columnar format for flat and hierarchical data, and brings the following advantages:
1. Cross-language with extensive libraries for different languages - The [format](https://arrow.apache.org/docs/format/Columnar.html) is defined via flatbuffers in such way that you can parse it in any language and already has extensive support in C/C++, C#, Go, Java, JavaScript, Julia, Matlab, Python, R, Ruby and Rust (at the time of writing). For CloudQuery this is important as it makes it much easier to develop source or destination plugins in different languages.
-2. Performance: Arrow adoption is rising especially in columnar based databases ([DuckDB](https://duckdb.org/2021/12/03/duck-arrow.html), [ClickHouse](https://clickhouse.com/docs/en/integrations/data-formats/arrow-avro-orc), [BigQuery](https://cloud.google.com/bigquery/docs/samples/bigquerystorage-arrow-quickstart)) and file formats ([Parquet](https://arrow.apache.org/docs/python/parquet.html)) which makes it easier to write CloudQuery destination or source plugins for databases that alr [...]
+2. Performance: Arrow adoption is rising especially in columnar based databases ([DuckDB](https://duckdb.org/2021/12/03/duck-arrow.html), [ClickHouse](https://clickhouse.com/docs/en/integrations/data-formats/arrow-avro-orc), [BigQuery](https://cloud.google.com/bigquery/docs/samples/bigquerystorage-arrow-quickstart)) and file formats ([Parquet](https://arrow.apache.org/docs/python/parquet.html)) which makes it easier to write CloudQuery destination or source plugins for databases that alr [...]
3. Rich Data Types: Arrow supports more than [35 types](https://arrow.apache.org/docs/python/api/datatypes.html) including composite types (i.e. lists, structs and maps of all the available types) and ability to extend the type system with custom types. Also, there is already built-in mapping from/to the arrow type system and the parquet type system (including nested types) which already supported in many of the arrow libraries as explained [here](https://arrow.apache.org/blog/2022/10/08 [...]
# Summary
Adopting Apache Arrow as the CloudQuery in-memory type system enables us to gain better performance, data interoperability and developer experience. Some plugins that are going to gain an immediate boost of rich type systems are our database-to-database replication plugins such as [PostgreSQL CDC](https://www.cloudquery.io/docs/plugins/sources/postgresql/overview) source plugin (and all [database destinations](https://www.cloudquery.io/docs/plugins/destinations/overview)) that are going [...]
-We are excited about this step and joining the growing Arrow community. We already contributed more than [30](https://github.com/search?q=is%3Apr+author%3Ayevgenypats+author%3Ahermanschaaf+author%3Acandiduslynx+author%3Adisq+label%3A%22Component%3A+Go%22++is%3Amerged+&ref=simplesearch) upstream pull requests that were quickly reviewed by the Arrow maintainers, thank you!
\ No newline at end of file
+We are excited about this step and joining the growing Arrow community. We already contributed more than [30](https://github.com/search?q=is%3Apr+author%3Ayevgenypats+author%3Ahermanschaaf+author%3Acandiduslynx+author%3Adisq+label%3A%22Component%3A+Go%22++is%3Amerged+&ref=simplesearch) upstream pull requests that were quickly reviewed by the Arrow maintainers, thank you!
[arrow-site] 02/02: Merge remote-tracking branch 'origin/main' into blog/cloudquery
Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch blog/cloudquery
in repository https://gitbox.apache.org/repos/asf/arrow-site.git
commit cf519556be4b8d1aeab0be72febf25743eca5ab9
Merge: 569349fe5f4 e2ca56d1c60
Author: Andrew Lamb <an...@nerdnetworks.org>
AuthorDate: Thu May 4 06:33:55 2023 -0400
Merge remote-tracking branch 'origin/main' into blog/cloudquery
_data/versions.yml | 28 +--
_release/12.0.0.md | 658 +++++++++++++++++++++++++++++++++++++++++++++++++++++
_release/index.md | 154 ++++++-------
powered_by.md | 3 +
4 files changed, 753 insertions(+), 90 deletions(-)