You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by al...@apache.org on 2023/05/04 10:36:34 UTC
[arrow-site] branch blog/cloudquery updated (cf519556be4 -> 9852b90a855)
This is an automated email from the ASF dual-hosted git repository.
alamb pushed a change to branch blog/cloudquery
in repository https://gitbox.apache.org/repos/asf/arrow-site.git
discard cf519556be4 Merge remote-tracking branch 'origin/main' into blog/cloudquery
discard 569349fe5f4 Update date
omit 8e7d3761062 Update _posts/2023-04-24-adopting-apache-arrow-at-cloudquery.md
omit 659cd4cebd8 update date for toda
omit 9fc07467dab add review
omit 9c3624d358f review fixes
omit 5fdadc751cf Update _posts/2023-04-24-adopting-apache-arrow-at-cloudquery.md
omit 2e3e8a307b1 Update _posts/2023-04-24-adopting-apache-arrow-at-cloudquery.md
omit 108784ea1ce Update _posts/2023-04-24-adopting-apache-arrow-at-cloudquery.md
omit ce50e846d7a cross post link
omit 62956031df6 Update _posts/2023-04-24-adopting-apache-arrow-at-cloudquery.md
omit ae53ab70b64 Update _posts/2023-04-24-adopting-apache-arrow-at-cloudquery.md
omit d9ff43a15a7 Update _posts/2023-04-24-adopting-apache-arrow-at-cloudquery.md
omit 1d045f264fc Update _posts/2023-04-24-adopting-apache-arrow-at-cloudquery.md
omit 758a2b2e57e Update _posts/2023-04-24-adopting-apache-arrow-at-cloudquery.md
omit d3962c39f3b [Website]: Adopting Apache Arrow at CloudQuery
add e4f66c4eacc [Website]: Adopting Apache Arrow at CloudQuery (#348)
new 9852b90a855 Update date on cloud query post
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (cf519556be4)
\
N -- N -- N refs/heads/blog/cloudquery (9852b90a855)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
[arrow-site] 01/01: Update date on cloud query post
Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch blog/cloudquery
in repository https://gitbox.apache.org/repos/asf/arrow-site.git
commit 9852b90a855436c706f59fb6c842ff327adcdd8e
Author: Andrew Lamb <an...@nerdnetworks.org>
AuthorDate: Thu May 4 06:33:17 2023 -0400
Update date on cloud query post
---
...udquery.md => 2023-05-04-adopting-apache-arrow-at-cloudquery.md} | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/_posts/2023-04-24-adopting-apache-arrow-at-cloudquery.md b/_posts/2023-05-04-adopting-apache-arrow-at-cloudquery.md
similarity index 98%
rename from _posts/2023-04-24-adopting-apache-arrow-at-cloudquery.md
rename to _posts/2023-05-04-adopting-apache-arrow-at-cloudquery.md
index cd326face08..94810618565 100644
--- a/_posts/2023-04-24-adopting-apache-arrow-at-cloudquery.md
+++ b/_posts/2023-05-04-adopting-apache-arrow-at-cloudquery.md
@@ -1,7 +1,7 @@
---
layout: post
title: "Adopting Apache Arrow at CloudQuery"
-date: "2023-04-30 00:00:00"
+date: "2023-05-04 00:00:00"
author: Yevgeny Pats
categories: [application]
---
@@ -55,11 +55,11 @@ Before Arrow, we used our own type system that supported more than 14 types. Thi
This is where Arrow comes in. Apache Arrow defines a language-independent columnar format for flat and hierarchical data, and brings the following advantages:
1. Cross-language with extensive libraries for different languages - The [format](https://arrow.apache.org/docs/format/Columnar.html) is defined via flatbuffers in such way that you can parse it in any language and already has extensive support in C/C++, C#, Go, Java, JavaScript, Julia, Matlab, Python, R, Ruby and Rust (at the time of writing). For CloudQuery this is important as it makes it much easier to develop source or destination plugins in different languages.
-2. Performance: Arrow adoption is rising especially in columnar based databases ([DuckDB](https://duckdb.org/2021/12/03/duck-arrow.html), [ClickHouse](https://clickhouse.com/docs/en/integrations/data-formats/arrow-avro-orc), [BigQuery](https://cloud.google.com/bigquery/docs/samples/bigquerystorage-arrow-quickstart)) and file formats ([Parquet](https://arrow.apache.org/docs/python/parquet.html)) which makes it easier to write CloudQuery destination or source plugins for databases that alr [...]
+2. Performance: Arrow adoption is rising especially in columnar based databases ([DuckDB](https://duckdb.org/2021/12/03/duck-arrow.html), [ClickHouse](https://clickhouse.com/docs/en/integrations/data-formats/arrow-avro-orc), [BigQuery](https://cloud.google.com/bigquery/docs/samples/bigquerystorage-arrow-quickstart)) and file formats ([Parquet](https://arrow.apache.org/docs/python/parquet.html)) which makes it easier to write CloudQuery destination or source plugins for databases that alr [...]
3. Rich Data Types: Arrow supports more than [35 types](https://arrow.apache.org/docs/python/api/datatypes.html) including composite types (i.e. lists, structs and maps of all the available types) and ability to extend the type system with custom types. Also, there is already built-in mapping from/to the arrow type system and the parquet type system (including nested types) which already supported in many of the arrow libraries as explained [here](https://arrow.apache.org/blog/2022/10/08 [...]
# Summary
Adopting Apache Arrow as the CloudQuery in-memory type system enables us to gain better performance, data interoperability and developer experience. Some plugins that are going to gain an immediate boost of rich type systems are our database-to-database replication plugins such as [PostgreSQL CDC](https://www.cloudquery.io/docs/plugins/sources/postgresql/overview) source plugin (and all [database destinations](https://www.cloudquery.io/docs/plugins/destinations/overview)) that are going [...]
-We are excited about this step and joining the growing Arrow community. We already contributed more than [30](https://github.com/search?q=is%3Apr+author%3Ayevgenypats+author%3Ahermanschaaf+author%3Acandiduslynx+author%3Adisq+label%3A%22Component%3A+Go%22++is%3Amerged+&ref=simplesearch) upstream pull requests that were quickly reviewed by the Arrow maintainers, thank you!
\ No newline at end of file
+We are excited about this step and joining the growing Arrow community. We already contributed more than [30](https://github.com/search?q=is%3Apr+author%3Ayevgenypats+author%3Ahermanschaaf+author%3Acandiduslynx+author%3Adisq+label%3A%22Component%3A+Go%22++is%3Amerged+&ref=simplesearch) upstream pull requests that were quickly reviewed by the Arrow maintainers, thank you!