You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@arrow.apache.org by GitBox <gi...@apache.org> on 2021/07/16 17:40:02 UTC

[GitHub] [arrow-site] nealrichardson opened a new pull request #127: Blog post for 5.0.0 release

nealrichardson opened a new pull request #127:
URL: https://github.com/apache/arrow-site/pull/127


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] ianmcook commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

ianmcook commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678783309



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,264 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+3 months of development work and includes **XX commits** from
+[**XX distinct contributors**][1] in 2 repositories. See the Install Page to

Review comment:
       ```suggestion
   3 months of development work and includes **684 commits** from
   [**99 distinct contributors**][1] in 2 repositories. See the Install Page to
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] nealrichardson commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

nealrichardson commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r677729911



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,96 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format) and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+## C++ notes
+
+## C# notes
+
+## Go notes
+
+## Java notes
+
+## JavaScript notes
+
+## Python notes

Review comment:
       ```suggestion
   ## Python notes
   
   * Datasets can now scan files asynchronously when the `use_async=True` option is provided to `Dataset.scanner`, `Dataset.to_table`, or `Dataset.to_batches` methods. This should provide better performance in environments where I/O can be slow, such as with remote sources.
   * Arrow now provides builtin support for writing CSV files through `pyarrow.csv.write_csv`
   * Wheels for Apple M1 Macs are now provided.
   * Many new `pyarrow.compute` functions are available, and introspection of the functions was improved so that they look more like standard Python functions.
   * It is now possible to access ORC file metadata from Python `ORCFile` objects
   * Building a `StructArray` now accepts a `mask` like other arrays
   * Many updates and fixes for the documentation
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] westonpace commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

westonpace commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678553219



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,229 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format)
+and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for
+the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has
+been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+`MakeArrayFromScalar` now works for fixed-size binary types (ARROW-13321).
+
+### Compute layer
+
+The following [compute functions]({{site.baseurl}}/docs/cpp/compute.html)
+were added:
+
+* aggregations: "index"
+
+* scalar arithmetic and math functions: "abs", "abs_checked", "acos",
+  "acos_checked", "asin", "asin_checked", "atan", "atan2", "ceil", "cos",
+  "cos_checked", "floor", "ln", "ln_checked", "log10", "log10_checked",
+  "log1p", "log1p_checked", "log2", "log2_checked", "negate", "negate_checked",
+  "sign", "sin", "sin_checked", "tan", "tan_checked", "trunc"
+
+* scalar bitwise functions: "bit_wise_and", "bit_wise_not", "bit_wise_or",
+  "bit_wise_xor", "shift_left", "shift_left_checked", "shift_right",
+  "shift_right_checked"
+
+* scalar string functions: "ascii_center", "ascii_lpad", "ascii_reverse",
+  "ascii_rpad", "binary_join", "binary_join_element_wise",
+  "binary_replace_slice", "count_substring", "count_substring_regex",
+  "ends_with", "find_substring", "find_substring_regex", "match_like",
+  "split_pattern_regex", "starts_with", "utf8_center", "utf8_lpad",
+  "utf8_replace_slice", "utf8_rpad", "utf8_reverse", "utf8_slice_codepoints"
+
+* scalar temporal functions: "day", "day_of_week", "day_of_year",
+  "iso_calendar", "iso_week", "iso_year", "hour", "microsecond", "millisecond",
+  "minute", "month", "nanosecond", "quarter", "second", "subsecond", "year"
+
+* other scalar functions: "case_when", "coalesce", "if_else", "is_finite",
+  "is_inf", "is_nan", "max_element_wise", "min_element_wise", "make_struct"
+
+* vector functions: "replace_with_mask"
+
+Duplicates are now allowed in `SetLookupOptions::value_set` (ARROW-12554).
+
+Decimal types are now supported by some basic arithmetic functions (ARROW-12074).
+
+The "take" function now supports dense unions (ARROW-13005).
+
+It is now possible to cast between dictionary types with different index
+types (ARROW-11673).
+
+Sorting is now implemented for boolean input (ARROW-12016).
+
+### CSV
+
+The streaming CSV reader can now read in parallel (ARROW-11889).
+
+The CSV reader tries to make its errors more informative by adding the
+row number when it is known, i.e. when parallel reading is disabled (ARROW-12675).
+
+A new option `ReaderOptions::skip_rows_after_names` allows skipping a number
+of rows _after_ reading the column names (as opposed to
+`ReaderOptions::skip_rows`).
+
+Quoted strings can now be treated as always non-null (ARROW-10115).
+
+### Dataset layer
+

Review comment:
       The asynchronous scanner was never really introduced (as, explained to the users) in 4.0.0.  It seems the only reference to it in the 4.0.0 docs was [this](https://arrow.apache.org/docs/cpp/api/dataset.html#_CPPv4N5arrow7dataset11ScanOptions9use_asyncE).  Maybe we can add a bit to this.
   
   ```
   The asynchronous scanner introduced in 4.0.0 has been improved, with truly asynchronous readers implemented for CSV, Parquet, and IPC file formats and file-level parallelism added.  This mode is controlled by a flag `use_async` that can be passed into methods which scan a dataset.  Setting this flag to True will have significant improvements on filesystems with high latency and generally some smaller improvements elsewhere.  This flag will default to True in some future version.
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] nealrichardson commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

nealrichardson commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r677731124



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,113 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format) and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+## C# notes
+
+## Go notes
+
+## Java notes
+
+## JavaScript notes
+
+* Tables do not extend DataFrames anymore. This enables smaller bundles. [#10277](https://github.com/apache/arrow/pull/10277)
+* Arrow uses closure compiler for all UMD bundles, making them smaller. [#10281](https://github.com/apache/arrow/pull/10281)
+* The npm package now comes with declaration maps for better navigation from types to source code. [#10673](https://github.com/apache/arrow/pull/10673)
+* Updated dependencies and improvements to the code.
+
+## Python notes
+
+* Datasets can now scan files asynchronously when the `use_async=True` option is provided to `Dataset.scanner`, `Dataset.to_table`, or `Dataset.to_batches` methods. This should provide better performance in environments where I/O can be slow, such as with remote sources.
+* Arrow now provides builtin support for writing CSV files through `pyarrow.csv.write_csv`
+* Wheels for Apple M1 Macs are now provided.
+* Many new `pyarrow.compute` functions are available, and introspection of the functions was improved so that they look more like standard Python functions.
+* It is now possible to access ORC file metadata from Python `ORCFile` objects
+* Building a `StructArray` now accepts a `mask` like other arrays
+* Many updates and fixes for the documentation
+
+## R notes
+
+In this release, we've more than doubled the number of functions you can call on Arrow Datasets inside `dplyr::filter()` and `mutate()`, including many more string, datetime, and math functions. You can also write Datasets to CSV files, in addition to Parquet and Feather. We've also deepened support for the Arrow C interface, which is used in the Python interface and allows integration with other projects, such as DuckDB.
+
+For more on what’s in the 5.0.0 R package, see the [R changelog][4].
+
+## Ruby and C GLib notes

Review comment:
       Anything you want to note here @kou? There isn't a whole lot on https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%205.0.0%20AND%20component%20in%20(GLib%2C%20Ruby)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] ianmcook commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

ianmcook commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678389316



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,229 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format)
+and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for
+the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has
+been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+`MakeArrayFromScalar` now works for fixed-size binary types (ARROW-13321).
+
+### Compute layer
+
+The following [compute functions]({{site.baseurl}}/docs/cpp/compute.html)
+were added:
+
+* aggregations: "index"
+
+* scalar arithmetic and math functions: "abs", "abs_checked", "acos",
+  "acos_checked", "asin", "asin_checked", "atan", "atan2", "ceil", "cos",
+  "cos_checked", "floor", "ln", "ln_checked", "log10", "log10_checked",
+  "log1p", "log1p_checked", "log2", "log2_checked", "negate", "negate_checked",
+  "sign", "sin", "sin_checked", "tan", "tan_checked", "trunc"
+
+* scalar bitwise functions: "bit_wise_and", "bit_wise_not", "bit_wise_or",
+  "bit_wise_xor", "shift_left", "shift_left_checked", "shift_right",
+  "shift_right_checked"
+
+* scalar string functions: "ascii_center", "ascii_lpad", "ascii_reverse",
+  "ascii_rpad", "binary_join", "binary_join_element_wise",
+  "binary_replace_slice", "count_substring", "count_substring_regex",
+  "ends_with", "find_substring", "find_substring_regex", "match_like",
+  "split_pattern_regex", "starts_with", "utf8_center", "utf8_lpad",
+  "utf8_replace_slice", "utf8_rpad", "utf8_reverse", "utf8_slice_codepoints"
+
+* scalar temporal functions: "day", "day_of_week", "day_of_year",
+  "iso_calendar", "iso_week", "iso_year", "hour", "microsecond", "millisecond",
+  "minute", "month", "nanosecond", "quarter", "second", "subsecond", "year"
+
+* other scalar functions: "case_when", "coalesce", "if_else", "is_finite",
+  "is_inf", "is_nan", "max_element_wise", "min_element_wise", "make_struct"
+
+* vector functions: "replace_with_mask"
+
+Duplicates are now allowed in `SetLookupOptions::value_set` (ARROW-12554).
+
+Decimal types are now supported by some basic arithmetic functions (ARROW-12074).
+
+The "take" function now supports dense unions (ARROW-13005).
+
+It is now possible to cast between dictionary types with different index
+types (ARROW-11673).
+
+Sorting is now implemented for boolean input (ARROW-12016).
+
+### CSV
+
+The streaming CSV reader can now read in parallel (ARROW-11889).
+
+The CSV reader tries to make its errors more informative by adding the
+row number when it is known, i.e. when parallel reading is disabled (ARROW-12675).
+
+A new option `ReaderOptions::skip_rows_after_names` allows skipping a number
+of rows _after_ reading the column names (as opposed to
+`ReaderOptions::skip_rows`).
+
+Quoted strings can now be treated as always non-null (ARROW-10115).
+
+### Dataset layer
+
+### IO and Filesystem layer
+
+The I/O thread pool size can now be adjuted at runtime (ARROW-12760).
+The default size remains 8 threads.
+
+Streams now can have auxiliary metadata, depending on the backend.  This
+has been implemented for the S3 filesystems, where a couple metadata
+keys are supported such as "Content-Type" and "ACL" (ARROW-11161, ARROW-12719).
+
+The HadoopFileSystem implementation now implements the FileSystem abstraction
+more faithfully (ARROW-12790).
+
+### Parquet
+
+The new LZ4_RAW compression scheme was implemented (PARQUET-1998).
+Unlike the legacy LZ4 compression scheme, it is defined unambiguously
+and should provide better portability once other Parquet implementations
+catch up.
+
+## Go notes
+
+## Java notes
+
+## JavaScript notes
+
+* Tables do not extend DataFrames anymore. This enables smaller bundles. [#10277](https://github.com/apache/arrow/pull/10277)
+* Arrow uses closure compiler for all UMD bundles, making them smaller. [#10281](https://github.com/apache/arrow/pull/10281)
+* The npm package now comes with declaration maps for better navigation from types to source code. [#10673](https://github.com/apache/arrow/pull/10673)
+* Updated dependencies and improvements to the code.
+
+## Python notes
+
+* Datasets can now scan files asynchronously when the `use_async=True` option is provided to `Dataset.scanner`, `Dataset.to_table`, or `Dataset.to_batches` methods. This should provide better performance in environments where I/O can be slow, such as with remote sources.
+* Arrow now provides builtin support for writing CSV files through `pyarrow.csv.write_csv`
+* Wheels for Apple M1 Macs are now provided.
+* Many new `pyarrow.compute` functions are available, and introspection of the functions was improved so that they look more like standard Python functions.
+* It is now possible to access ORC file metadata from Python `ORCFile` objects
+* Building a `StructArray` now accepts a `mask` like other arrays
+* Many updates and fixes for the documentation
+
+## R notes
+
+In this release, we've more than doubled the number of functions you can call on Arrow Datasets inside `dplyr::filter()` and `mutate()`, including many more string, datetime, and math functions. You can also write Datasets to CSV files, in addition to Parquet and Feather. We've also deepened support for the Arrow C interface, which is used in the Python interface and allows integration with other projects, such as DuckDB.
+
+For more on what’s in the 5.0.0 R package, see the [R changelog][4].
+
+## Ruby and C GLib notes
+
+Apache Arrow Flight support is started. But `ListFlights` is only supported for now. More features will be implemented in the next major release.
+
+### Ruby
+
+You need gobject-introspection gem 3.4.5 or later to implement your Apache Arrow Flight server. If you only use Apache Arrow Flight client, gobject-introspection gem 3.4.5 or later isn't required.
+
+Here are highlighted improvements:
+
+* Compute functions accept raw Ruby objects such as `true`, `Integer`, `Array` and `String`:
+
+  ```ruby
+  add_function = Arrow::Function.find("add")
+  # Not shortcut version
+  augend = Arrow::Int8Array.new([1, 2, 3])
+  addend = Arrow::Int8Scalar.new(5)
+  args = [
+    Arrow::ArrayDatum.new(augend),
+    Arrow::ScalarDatum.new(addend),
+  ]
+  add_function.execute(args).value.to_a # => [6, 7, 8]
+  # Shortcut version
+  add_function.execute([[1, 2, 3], 5]).value.to_a # => [6, 7, 8]
+  ```
+* `Arrow::PrimaryArray` and `Arrow::Buffer` can be used as MemoryView that is added in Ruby 3.0.
+
+There are some backward incompatible changes:
+
+* `Arrow::CountOptions` and `Arrow::CountMode` are removed. Use `Arrow::ScalarAggregateOptions` instead.
+
+### C GLib
+
+There are some backward incompatible changes:
+
+* `GArrowCountOptions` and `GArrowCountMode` are removed. Use `GArrowScalarAggregateOptions` instead.
+* `garrow_array_equal_range()` requires `GArrowEqualOptions`.
+* Prefix in arrow-dataset-glib is changed to `gadataset_`/`GADATASET_` from `gad_`/`GAD_`.
+* `GADScanOptions`, `GADScanTask`  and `GADInMemoryScanTask` are removed. Use `gadataset_begin_scan()` or `gadataset_to_table()` instead.
+* `GArrowCompareOptions`, `GArrowCompareOperator` and `garrow_*_array_compare()` are removed. Use `equal`, `not_equal`, `less_than`, `less_than_equal`, `greater_than` and `greater_than_equal` compute functions directly instead.
+
+## Rust notes
+
+
+[1]: https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%205.0.0
+[2]: {{ site.baseurl }}/release/5.0.0.html#contributors
+[3]: {{ site.baseurl }}/release/5.0.0.html

Review comment:
       ```suggestion
   [1]: {{ site.baseurl }}/release/5.0.0.html#contributors
   [2]: {{ site.baseurl }}/release/5.0.0.html
   [3]: https://github.com/apache/arrow-rs/blob/5.0.0/CHANGELOG.md
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] ianmcook commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

ianmcook commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678592244



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,259 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+3 months of development work and includes **XX commits** from

Review comment:
       @kszucs let's please get https://github.com/apache/arrow/pull/10774 reviewed and merged and use that to fill in the XXs




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] liyafan82 commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

liyafan82 commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678780589



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,264 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+3 months of development work and includes **XX commits** from
+[**XX distinct contributors**][1] in 2 repositories. See the Install Page to
+learn how to get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the complete changelogs for the [`apache/arrow`][2] and
+[`apache/arrow-rs`][3] repositories.
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format)
+and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend `.arrow` as the IPC file format file extension and `.arrows` for
+the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has
+been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+`MakeArrayFromScalar` now works for fixed-size binary types (ARROW-13321).
+
+### Compute layer
+
+The following [compute functions]({{site.baseurl}}/docs/cpp/compute.html)
+were added:
+
+* aggregations: `index`
+
+* scalar arithmetic and math functions: `abs`, `abs_checked`, `acos`,
+  `acos_checked`, `asin`, `asin_checked`, `atan`, `atan2`, `ceil`, `cos`,
+  `cos_checked`, `floor`, `ln`, `ln_checked`, `log10`, `log10_checked`,
+  `log1p`, `log1p_checked`, `log2`, `log2_checked`, `negate`, `negate_checked`,
+  `sign`, `sin`, `sin_checked`, `tan`, `tan_checked`, `trunc`
+
+* scalar bitwise functions: `bit_wise_and`, `bit_wise_not`, `bit_wise_or`,
+  `bit_wise_xor`, `shift_left`, `shift_left_checked`, `shift_right`,
+  `shift_right_checked`
+
+* scalar string functions: `ascii_center`, `ascii_lpad`, `ascii_reverse`,
+  `ascii_rpad`, `binary_join`, `binary_join_element_wise`,
+  `binary_replace_slice`, `count_substring`, `count_substring_regex`,
+  `ends_with`, `find_substring`, `find_substring_regex`, `match_like`,
+  `split_pattern_regex`, `starts_with`, `utf8_center`, `utf8_lpad`,
+  `utf8_replace_slice`, `utf8_rpad`, `utf8_reverse`, `utf8_slice_codepoints`
+
+* scalar temporal functions: `day`, `day_of_week`, `day_of_year`,
+  `iso_calendar`, `iso_week`, `iso_year`, `hour`, `microsecond`, `millisecond`,
+  `minute`, `month`, `nanosecond`, `quarter`, `second`, `subsecond`, `year`
+
+* other scalar functions: `case_when`, `coalesce`, `if_else`, `is_finite`,
+  `is_inf`, `is_nan`, `max_element_wise`, `min_element_wise`, `make_struct`
+
+* vector functions: `replace_with_mask`
+
+Duplicates are now allowed in `SetLookupOptions::value_set` (ARROW-12554).
+
+Decimal types are now supported by some basic arithmetic functions (ARROW-12074).
+
+The `take` function now supports dense unions (ARROW-13005).
+
+It is now possible to cast between dictionary types with different index
+types (ARROW-11673).
+
+Sorting is now implemented for boolean input (ARROW-12016).
+
+### CSV
+
+The streaming CSV reader can now take some advantage of multiple threads (ARROW-11889).
+
+The CSV reader tries to make its errors more informative by adding the
+row number when it is known, i.e. when parallel reading is disabled (ARROW-12675).
+
+A new option `ReaderOptions::skip_rows_after_names` allows skipping a number
+of rows _after_ reading the column names (as opposed to
+`ReaderOptions::skip_rows`).
+
+Quoted strings can now be treated as always non-null (ARROW-10115).
+
+### Dataset layer
+
+The asynchronous scanner introduced in 4.0.0 has been improved with truly 
+asynchronous readers implemented for CSV, Parquet, and IPC file formats and 
+file-level parallelism added.  This mode is controlled by a flag `use_async` that
+can be passed into methods which scan a dataset.  Setting this flag to True
+will have significant improvements on filesystems with high latency or parallel
+reads (e.g. S3).
+
+A CountRows method has been added to count rows matching a predicate; where
+possible, this will use metadata in files instead of reading the data itself.
+
+CSV datasets can now be written, and when reading a CSV dataset, explicit types can
+now be specified for a subset of columns while allowing the rest to still be inferred. 
+
+### IO and Filesystem layer
+
+The I/O thread pool size can now be adjuted at runtime (ARROW-12760).
+The default size remains 8 threads.
+
+Streams now can have auxiliary metadata, depending on the backend.  This
+has been implemented for the S3 filesystems, where a couple metadata
+keys are supported such as `Content-Type` and `ACL` (ARROW-11161, ARROW-12719).
+
+The HadoopFileSystem implementation now implements the FileSystem abstraction
+more faithfully (ARROW-12790).
+
+### Parquet
+
+The new LZ4_RAW compression scheme was implemented (PARQUET-1998).
+Unlike the legacy LZ4 compression scheme, it is defined unambiguously
+and should provide better portability once other Parquet implementations
+catch up.
+
+## Go notes
+
+
+### Flight 
+
+* Flight Client and Server now support Custom Metadata through the functions `flight.NewClientWithMiddleware` and `flight.NewServerWithMiddleware`. Functions
+`flight.NewFlightClient`, `flight.NewFlightServer`, `flight.CreateServerBearerTokenAuthInterceptors` have been deprecated in favor of using the new middleware. [#10633](https://github.com/apache/arrow/pull/10633)
+* Flight Client `AuthHandler` no longer overwrites outgoing metadata, correctly appending new metadata without overwriting existing metadata [#10297](https://github.com/apache/arrow/pull/10297)
+* Flight AppMetadata field is now exposed both for Reading and Writing via `flight.Reader#LatestAppMetadata()` and `flight.Writer#WriteWithAppMetadata` functions [#10142](https://github.com/apache/arrow/pull/10142)
+
+### Other enhancements 
+
+* [Map](https://github.com/apache/arrow/pull/10106) and [Extension](https://github.com/apache/arrow/pull/10203) Datatypes are now implemented for Arrow Arrays 
+* [Schema package](https://github.com/apache/arrow/10071) and first part of [Encoding package](https://github.com/apache/arrow/10379) added for Golang Parquet Implementation
+
+## Java notes
+
+Highlighted improvements and fixes:
+
+* Improved support for extension types using a complex storage type, e.g. struct, map or union. These can now extend
+the `ExtensionTypeVector` base class.
+* Union vectors now extend `AbstractContainerVector` to be consistent with other vectors.
+* Guava dependency updated to 30.1.1
+* Memory leak fixed if an exception occurs when reading IPC messages from a channel. [#10423](https://github.com/apache/arrow/pull/10423)
+* Flight error metadata is now propagated to the client. [#10370](https://github.com/apache/arrow/pull/10370)
+* JDBC adapter now preserves nullability. [#10285](https://github.com/apache/arrow/pull/10285)
+

Review comment:
       ```suggestion
   * The memory rounding policy is respected when allocating vector buffers. This helps saving memory space. [#13147] (https://github.com/apache/arrow/pull/10576)
   
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] ianmcook commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

ianmcook commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r682066764



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,267 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-29 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+3 months of development work and includes **684 commits** from
+[**99 distinct contributors**][1] in 2 repositories. See the Install Page to
+learn how to get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the complete changelogs for the [`apache/arrow`][2] and
+[`apache/arrow-rs`][3] repositories.
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format)
+and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend `.arrow` as the IPC file format file extension and `.arrows` for
+the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has
+been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+`MakeArrayFromScalar` now works for fixed-size binary types (ARROW-13321).
+
+### Compute layer
+
+The following [compute functions]({{site.baseurl}}/docs/cpp/compute.html)
+were added:
+
+* aggregations: `index`
+
+* scalar arithmetic and math functions: `abs`, `abs_checked`, `acos`,
+  `acos_checked`, `asin`, `asin_checked`, `atan`, `atan2`, `ceil`, `cos`,
+  `cos_checked`, `floor`, `ln`, `ln_checked`, `log10`, `log10_checked`,
+  `log1p`, `log1p_checked`, `log2`, `log2_checked`, `negate`, `negate_checked`,
+  `sign`, `sin`, `sin_checked`, `tan`, `tan_checked`, `trunc`
+
+* scalar bitwise functions: `bit_wise_and`, `bit_wise_not`, `bit_wise_or`,
+  `bit_wise_xor`, `shift_left`, `shift_left_checked`, `shift_right`,
+  `shift_right_checked`
+
+* scalar string functions: `ascii_center`, `ascii_lpad`, `ascii_reverse`,
+  `ascii_rpad`, `binary_join`, `binary_join_element_wise`,
+  `binary_replace_slice`, `count_substring`, `count_substring_regex`,
+  `ends_with`, `find_substring`, `find_substring_regex`, `match_like`,
+  `split_pattern_regex`, `starts_with`, `utf8_center`, `utf8_lpad`,
+  `utf8_replace_slice`, `utf8_rpad`, `utf8_reverse`, `utf8_slice_codepoints`
+
+* scalar temporal functions: `day`, `day_of_week`, `day_of_year`,
+  `iso_calendar`, `iso_week`, `iso_year`, `hour`, `microsecond`, `millisecond`,
+  `minute`, `month`, `nanosecond`, `quarter`, `second`, `subsecond`, `year`
+
+* other scalar functions: `case_when`, `coalesce`, `if_else`, `is_finite`,
+  `is_inf`, `is_nan`, `max_element_wise`, `min_element_wise`, `make_struct`
+
+* vector functions: `replace_with_mask`
+
+Duplicates are now allowed in `SetLookupOptions::value_set` (ARROW-12554).
+
+Decimal types are now supported by some basic arithmetic functions (ARROW-12074).
+
+The `take` function now supports dense unions (ARROW-13005).
+
+It is now possible to cast between dictionary types with different index
+types (ARROW-11673).
+
+Sorting is now implemented for boolean input (ARROW-12016).
+
+### CSV
+
+The streaming CSV reader can now take some advantage of multiple threads (ARROW-11889).
+
+The CSV reader tries to make its errors more informative by adding the
+row number when it is known, i.e. when parallel reading is disabled (ARROW-12675).
+
+A new option `ReaderOptions::skip_rows_after_names` allows skipping a number
+of rows _after_ reading the column names (as opposed to
+`ReaderOptions::skip_rows`).
+
+Quoted strings can now be treated as always non-null (ARROW-10115).
+
+### Dataset layer
+
+The asynchronous scanner introduced in 4.0.0 has been improved with truly 
+asynchronous readers implemented for CSV, Parquet, and IPC file formats and 
+file-level parallelism added.  This mode is controlled by a flag `use_async` that
+can be passed into methods which scan a dataset.  Setting this flag to True
+will have significant improvements on filesystems with high latency or parallel
+reads (e.g. S3).
+
+A CountRows method has been added to count rows matching a predicate; where
+possible, this will use metadata in files instead of reading the data itself.
+
+CSV datasets can now be written, and when reading a CSV dataset, explicit types can
+now be specified for a subset of columns while allowing the rest to still be inferred. 
+
+### IO and Filesystem layer
+
+The I/O thread pool size can now be adjusted at runtime (ARROW-12760).
+The default size remains 8 threads.
+
+Streams now can have auxiliary metadata, depending on the backend.  This
+has been implemented for the S3 filesystems, where a couple metadata
+keys are supported such as `Content-Type` and `ACL` (ARROW-11161, ARROW-12719).
+
+The HadoopFileSystem implementation now implements the FileSystem abstraction
+more faithfully (ARROW-12790).
+
+### Parquet
+
+The new LZ4_RAW compression scheme was implemented (PARQUET-1998).
+Unlike the legacy LZ4 compression scheme, it is defined unambiguously
+and should provide better portability once other Parquet implementations
+catch up.
+
+## Go notes
+
+
+### Flight 
+
+* Flight Client and Server now support Custom Metadata through the functions `flight.NewClientWithMiddleware` and `flight.NewServerWithMiddleware`. Functions
+`flight.NewFlightClient`, `flight.NewFlightServer`, `flight.CreateServerBearerTokenAuthInterceptors` have been deprecated in favor of using the new middleware. [#10633](https://github.com/apache/arrow/pull/10633)
+* Flight Client `AuthHandler` no longer overwrites outgoing metadata, correctly appending new metadata without overwriting existing metadata [#10297](https://github.com/apache/arrow/pull/10297)
+* Flight AppMetadata field is now exposed both for Reading and Writing via `flight.Reader#LatestAppMetadata()` and `flight.Writer#WriteWithAppMetadata` functions [#10142](https://github.com/apache/arrow/pull/10142)
+
+### Other enhancements 
+
+* [Map](https://github.com/apache/arrow/pull/10106) and [Extension](https://github.com/apache/arrow/pull/10203) Datatypes are now implemented for Arrow Arrays 
+* [Schema package](https://github.com/apache/arrow/10071) and first part of [Encoding package](https://github.com/apache/arrow/10379) added for Golang Parquet Implementation
+
+## Java notes
+
+Highlighted improvements and fixes:
+
+* Improved support for extension types using a complex storage type, e.g. struct, map or union. These can now extend
+the `ExtensionTypeVector` base class.
+* Union vectors now extend `AbstractContainerVector` to be consistent with other vectors.
+* Guava dependency updated to 30.1.1
+* Memory leak fixed if an exception occurs when reading IPC messages from a channel. [#10423](https://github.com/apache/arrow/pull/10423)
+* Flight error metadata is now propagated to the client. [#10370](https://github.com/apache/arrow/pull/10370)
+* JDBC adapter now preserves nullability. [#10285](https://github.com/apache/arrow/pull/10285)
+* The memory rounding policy is respected when allocating vector buffers. This helps saving memory space. [#10576](https://github.com/apache/arrow/pull/10576)
+
+API compatibility changes:
+
+* Complex vectors now return covariant types from `getObject(int)`. [#9964](https://github.com/apache/arrow/pull/9964)
+
+## JavaScript notes
+
+* Tables do not extend DataFrames anymore. This enables smaller bundles. [#10277](https://github.com/apache/arrow/pull/10277)
+* Arrow uses closure compiler for all UMD bundles, making them smaller. [#10281](https://github.com/apache/arrow/pull/10281)
+* The npm package now comes with declaration maps for better navigation from types to source code. [#10673](https://github.com/apache/arrow/pull/10673)
+* Updated dependencies and improvements to the code.
+
+## Python notes
+
+* Datasets can now scan files asynchronously when the `use_async=True` option is provided to `Dataset.scanner`, `Dataset.to_table`, or `Dataset.to_batches` methods. This should provide better performance in environments where I/O can be slow, such as with remote sources.
+* Arrow now provides builtin support for writing CSV files through `pyarrow.csv.write_csv`
+* Wheels for Apple M1 Macs are now provided.
+* Many new `pyarrow.compute` functions are available (see the C++ notes above
+  for more details), and introspection of the functions was improved so that
+  they look more like standard Python functions.
+* It is now possible to access ORC file metadata from Python `ORCFile` objects
+* Building a `StructArray` now accepts a `mask` like other arrays
+* Many updates and fixes for the documentation
+
+## R notes
+
+In this release, we've more than doubled the number of functions you can call on Arrow Datasets inside `dplyr::filter()` and `mutate()`, including many more string, datetime, and math functions. You can also write Datasets to CSV files, in addition to Parquet and Feather. We've also deepened support for the Arrow C interface, which is used in the Python interface and allows integration with other projects, such as DuckDB.

Review comment:
       ```suggestion
   In this release, we've more than doubled the number of functions you can call on Arrow Datasets inside `dplyr::filter()`, `mutate()`, and `arrange()`, including many more string, datetime, and math functions. You can also write Datasets to CSV files, in addition to Parquet and Feather. We've also deepened support for the Arrow C interface, which is used in the Python interface and allows integration with other projects, such as DuckDB.
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] nealrichardson commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

nealrichardson commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r677731693



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,113 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format) and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+## C# notes
+
+## Go notes

Review comment:
       @zeroshade anything you want to note here?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] alamb commented on pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

alamb commented on pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#issuecomment-881891451


   I have draft content for the 5.0.0 rust blog here: https://docs.google.com/document/d/1QVvngReUcAtBf7Tz_TAHZZeXT9W1mha3UwgF_uGys9E/edit?ts=60ee7f5d#
   
   I would personally prefer a separate blog post as the overall arrow release is pretty large and the rust details may get lost several pages down but I am curious what others think
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] nealrichardson commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

nealrichardson commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678584794



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,259 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+

Review comment:
       ```suggestion
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] westonpace commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

westonpace commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678594536



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,262 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+3 months of development work and includes **XX commits** from
+[**XX distinct contributors**][1] in 2 repositories. See the Install Page to
+learn how to get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the complete changelogs for the [`apache/arrow`][2] and
+[`apache/arrow-rs`][3] repositories.
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format)
+and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend `.arrow` as the IPC file format file extension and `.arrows` for
+the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has
+been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+`MakeArrayFromScalar` now works for fixed-size binary types (ARROW-13321).
+
+### Compute layer
+
+The following [compute functions]({{site.baseurl}}/docs/cpp/compute.html)
+were added:
+
+* aggregations: `index`
+
+* scalar arithmetic and math functions: `abs`, `abs_checked`, `acos`,
+  `acos_checked`, `asin`, `asin_checked`, `atan`, `atan2`, `ceil`, `cos`,
+  `cos_checked`, `floor`, `ln`, `ln_checked`, `log10`, `log10_checked`,
+  `log1p`, `log1p_checked`, `log2`, `log2_checked`, `negate`, `negate_checked`,
+  `sign`, `sin`, `sin_checked`, `tan`, `tan_checked`, `trunc`
+
+* scalar bitwise functions: `bit_wise_and`, `bit_wise_not`, `bit_wise_or`,
+  `bit_wise_xor`, `shift_left`, `shift_left_checked`, `shift_right`,
+  `shift_right_checked`
+
+* scalar string functions: `ascii_center`, `ascii_lpad`, `ascii_reverse`,
+  `ascii_rpad`, `binary_join`, `binary_join_element_wise`,
+  `binary_replace_slice`, `count_substring`, `count_substring_regex`,
+  `ends_with`, `find_substring`, `find_substring_regex`, `match_like`,
+  `split_pattern_regex`, `starts_with`, `utf8_center`, `utf8_lpad`,
+  `utf8_replace_slice`, `utf8_rpad`, `utf8_reverse`, `utf8_slice_codepoints`
+
+* scalar temporal functions: `day`, `day_of_week`, `day_of_year`,
+  `iso_calendar`, `iso_week`, `iso_year`, `hour`, `microsecond`, `millisecond`,
+  `minute`, `month`, `nanosecond`, `quarter`, `second`, `subsecond`, `year`
+
+* other scalar functions: `case_when`, `coalesce`, `if_else`, `is_finite`,
+  `is_inf`, `is_nan`, `max_element_wise`, `min_element_wise`, `make_struct`
+
+* vector functions: `replace_with_mask`
+
+Duplicates are now allowed in `SetLookupOptions::value_set` (ARROW-12554).
+
+Decimal types are now supported by some basic arithmetic functions (ARROW-12074).
+
+The `take` function now supports dense unions (ARROW-13005).
+
+It is now possible to cast between dictionary types with different index
+types (ARROW-11673).
+
+Sorting is now implemented for boolean input (ARROW-12016).
+
+### CSV
+
+The streaming CSV reader can now take some advantage of multiple threads (ARROW-11889).
+
+The CSV reader tries to make its errors more informative by adding the
+row number when it is known, i.e. when parallel reading is disabled (ARROW-12675).
+
+A new option `ReaderOptions::skip_rows_after_names` allows skipping a number
+of rows _after_ reading the column names (as opposed to
+`ReaderOptions::skip_rows`).
+
+Quoted strings can now be treated as always non-null (ARROW-10115).
+
+### Dataset layer
+The asynchronous scanner introduced in 4.0.0 has been improved with truly 
+asynchronous readers implemented for CSV, Parquet, and IPC file formats and 
+file-level parallelism added.  This mode is controlled by a flag `use_async` that
+can be passed into methods which scan a dataset.  Setting this flag to True
+will have significant improvements on filesystems with high latency or parallel
+reads (e.g. S3).
+
+A CountRows method has been added to count rows matching a predicate; where
+possible, this will use metadata in files instead of reading the data itself.
+
+CSV datasets can now be written, and when reading a CSV dataset, explicit types can
+now be specified for a subset of columns while allowing the rest to still be inferred. 
+### IO and Filesystem layer

Review comment:
       ```suggestion
   ### Dataset layer
   
   The asynchronous scanner introduced in 4.0.0 has been improved with truly 
   asynchronous readers implemented for CSV, Parquet, and IPC file formats and 
   file-level parallelism added.  This mode is controlled by a flag `use_async` that
   can be passed into methods which scan a dataset.  Setting this flag to True
   will have significant improvements on filesystems with high latency or parallel
   reads (e.g. S3).
   
   A CountRows method has been added to count rows matching a predicate; where
   possible, this will use metadata in files instead of reading the data itself.
   
   CSV datasets can now be written, and when reading a CSV dataset, explicit types can
   now be specified for a subset of columns while allowing the rest to still be inferred. 
   
   ### IO and Filesystem layer
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] thisisnic commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

thisisnic commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r677516896



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,100 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].

Review comment:
       Agreed!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] ianmcook commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

ianmcook commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678388328



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,229 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].

Review comment:
       ```suggestion
   The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
   3 months of development work and includes [**XX commits**] from
   [**XX distinct contributors**][1] in 2 repositories. See the Install Page to
   learn how to get the libraries for your platform.
   
   The release notes below are not exhaustive and only expose selected highlights
   of the release. Many other bugfixes and improvements have been made: we refer
   you to the complete changelogs for the [`apache/arrow`][2] and
   [`apache/arrow-rs`][3] repositories.
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] BryanCutler commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

BryanCutler commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678477639



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,113 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format) and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+## C# notes
+
+## Go notes
+
+## Java notes

Review comment:
       I think there are some things, I'll add them




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] ianmcook commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

ianmcook commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678540966



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,244 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+3 months of development work and includes **XX commits** from
+[**XX distinct contributors**][1] in 2 repositories. See the Install Page to
+learn how to get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the complete changelogs for the [`apache/arrow`][2] and
+[`apache/arrow-rs`][3] repositories.
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format)
+and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for
+the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has
+been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+`MakeArrayFromScalar` now works for fixed-size binary types (ARROW-13321).
+
+### Compute layer
+
+The following [compute functions]({{site.baseurl}}/docs/cpp/compute.html)
+were added:
+
+* aggregations: "index"
+
+* scalar arithmetic and math functions: "abs", "abs_checked", "acos",
+  "acos_checked", "asin", "asin_checked", "atan", "atan2", "ceil", "cos",
+  "cos_checked", "floor", "ln", "ln_checked", "log10", "log10_checked",
+  "log1p", "log1p_checked", "log2", "log2_checked", "negate", "negate_checked",
+  "sign", "sin", "sin_checked", "tan", "tan_checked", "trunc"
+
+* scalar bitwise functions: "bit_wise_and", "bit_wise_not", "bit_wise_or",
+  "bit_wise_xor", "shift_left", "shift_left_checked", "shift_right",
+  "shift_right_checked"
+
+* scalar string functions: "ascii_center", "ascii_lpad", "ascii_reverse",
+  "ascii_rpad", "binary_join", "binary_join_element_wise",
+  "binary_replace_slice", "count_substring", "count_substring_regex",
+  "ends_with", "find_substring", "find_substring_regex", "match_like",
+  "split_pattern_regex", "starts_with", "utf8_center", "utf8_lpad",
+  "utf8_replace_slice", "utf8_rpad", "utf8_reverse", "utf8_slice_codepoints"
+
+* scalar temporal functions: "day", "day_of_week", "day_of_year",
+  "iso_calendar", "iso_week", "iso_year", "hour", "microsecond", "millisecond",
+  "minute", "month", "nanosecond", "quarter", "second", "subsecond", "year"
+
+* other scalar functions: "case_when", "coalesce", "if_else", "is_finite",
+  "is_inf", "is_nan", "max_element_wise", "min_element_wise", "make_struct"
+
+* vector functions: "replace_with_mask"
+
+Duplicates are now allowed in `SetLookupOptions::value_set` (ARROW-12554).
+
+Decimal types are now supported by some basic arithmetic functions (ARROW-12074).
+
+The "take" function now supports dense unions (ARROW-13005).
+
+It is now possible to cast between dictionary types with different index
+types (ARROW-11673).
+
+Sorting is now implemented for boolean input (ARROW-12016).
+
+### CSV
+
+The streaming CSV reader can now read in parallel (ARROW-11889).
+
+The CSV reader tries to make its errors more informative by adding the
+row number when it is known, i.e. when parallel reading is disabled (ARROW-12675).
+
+A new option `ReaderOptions::skip_rows_after_names` allows skipping a number
+of rows _after_ reading the column names (as opposed to
+`ReaderOptions::skip_rows`).
+
+Quoted strings can now be treated as always non-null (ARROW-10115).
+
+### Dataset layer
+
+### IO and Filesystem layer
+
+The I/O thread pool size can now be adjuted at runtime (ARROW-12760).
+The default size remains 8 threads.
+
+Streams now can have auxiliary metadata, depending on the backend.  This
+has been implemented for the S3 filesystems, where a couple metadata
+keys are supported such as "Content-Type" and "ACL" (ARROW-11161, ARROW-12719).
+
+The HadoopFileSystem implementation now implements the FileSystem abstraction
+more faithfully (ARROW-12790).
+
+### Parquet
+
+The new LZ4_RAW compression scheme was implemented (PARQUET-1998).
+Unlike the legacy LZ4 compression scheme, it is defined unambiguously
+and should provide better portability once other Parquet implementations
+catch up.
+
+## Go notes
+
+## Java notes
+
+Highlighted improvements and fixes:
+
+* Improved support for extension types using a complex storage type, e.g. struct, map or union. These can now extend
+the `ExtensionTypeVector` base class.
+* Union vectors now extend `AbstractContainerVector` to be consistent with other vectors.
+* Guava dependency updated to 30.1.1
+* Memory leak fixed if an exception occurs when reading IPC messages from a channel. [#10423](https://github.com/apache/arrow/pull/10423)
+* Flight error metadata is now propagated to the client. [#10370](https://github.com/apache/arrow/pull/10370)
+* JDBC adapter now preserves nullability. [#10285](https://github.com/apache/arrow/pull/10285)
+
+API compatibility changes:
+
+* Complex vectors now return covariant types from `getObject(int)`. [#9964](https://github.com/apache/arrow/pull/9964)
+
+## JavaScript notes
+
+* Tables do not extend DataFrames anymore. This enables smaller bundles. [#10277](https://github.com/apache/arrow/pull/10277)
+* Arrow uses closure compiler for all UMD bundles, making them smaller. [#10281](https://github.com/apache/arrow/pull/10281)
+* The npm package now comes with declaration maps for better navigation from types to source code. [#10673](https://github.com/apache/arrow/pull/10673)
+* Updated dependencies and improvements to the code.
+
+## Python notes
+
+* Datasets can now scan files asynchronously when the `use_async=True` option is provided to `Dataset.scanner`, `Dataset.to_table`, or `Dataset.to_batches` methods. This should provide better performance in environments where I/O can be slow, such as with remote sources.
+* Arrow now provides builtin support for writing CSV files through `pyarrow.csv.write_csv`
+* Wheels for Apple M1 Macs are now provided.
+* Many new `pyarrow.compute` functions are available, and introspection of the functions was improved so that they look more like standard Python functions.
+* It is now possible to access ORC file metadata from Python `ORCFile` objects
+* Building a `StructArray` now accepts a `mask` like other arrays
+* Many updates and fixes for the documentation
+
+## R notes
+
+In this release, we've more than doubled the number of functions you can call on Arrow Datasets inside `dplyr::filter()` and `mutate()`, including many more string, datetime, and math functions. You can also write Datasets to CSV files, in addition to Parquet and Feather. We've also deepened support for the Arrow C interface, which is used in the Python interface and allows integration with other projects, such as DuckDB.
+
+For more on what’s in the 5.0.0 R package, see the [R changelog][4].
+
+## Ruby and C GLib notes
+
+Apache Arrow Flight support is started. But `ListFlights` is only supported for now. More features will be implemented in the next major release.
+
+### Ruby
+
+You need gobject-introspection gem 3.4.5 or later to implement your Apache Arrow Flight server. If you only use Apache Arrow Flight client, gobject-introspection gem 3.4.5 or later isn't required.
+
+Here are highlighted improvements:
+
+* Compute functions accept raw Ruby objects such as `true`, `Integer`, `Array` and `String`:
+
+  ```ruby
+  add_function = Arrow::Function.find("add")
+  # Not shortcut version
+  augend = Arrow::Int8Array.new([1, 2, 3])
+  addend = Arrow::Int8Scalar.new(5)
+  args = [
+    Arrow::ArrayDatum.new(augend),
+    Arrow::ScalarDatum.new(addend),
+  ]
+  add_function.execute(args).value.to_a # => [6, 7, 8]
+  # Shortcut version
+  add_function.execute([[1, 2, 3], 5]).value.to_a # => [6, 7, 8]
+  ```
+* `Arrow::PrimaryArray` and `Arrow::Buffer` can be used as MemoryView that is added in Ruby 3.0.
+
+There are some backward incompatible changes:
+
+* `Arrow::CountOptions` and `Arrow::CountMode` are removed. Use `Arrow::ScalarAggregateOptions` instead.
+
+### C GLib
+
+There are some backward incompatible changes:
+
+* `GArrowCountOptions` and `GArrowCountMode` are removed. Use `GArrowScalarAggregateOptions` instead.
+* `garrow_array_equal_range()` requires `GArrowEqualOptions`.
+* Prefix in arrow-dataset-glib is changed to `gadataset_`/`GADATASET_` from `gad_`/`GAD_`.
+* `GADScanOptions`, `GADScanTask`  and `GADInMemoryScanTask` are removed. Use `gadataset_begin_scan()` or `gadataset_to_table()` instead.
+* `GArrowCompareOptions`, `GArrowCompareOperator` and `garrow_*_array_compare()` are removed. Use `equal`, `not_equal`, `less_than`, `less_than_equal`, `greater_than` and `greater_than_equal` compute functions directly instead.
+
+## Rust notes
+

Review comment:
       ```suggestion
   
   The Rust projects have moved to separate repositories outside the
   main Arrow monorepo. For notes on the 5.0.0 release of the Rust
   implementation, see the [Arrow Rust changelog][3] and the
   "Apache Arrow Rust 5.0.0 Release" blog post.
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] ianmcook commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

ianmcook commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r677464606



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,100 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].

Review comment:
       @thisisnic we should tweak this to be consistent with the changes to the release notes in https://github.com/apache/arrow/pull/10774.
   
   Currently this counts resolved issues, with a link to Jira. The release notes does not count resolved Jiras; it counts commits. Rust and DataFusion use GitHub Issues, not Jira.
   
   I think the easiest thing to do would be to simply change `[**XX resolved issues**][1]` to `[**XX commits**]` and remove the link to Jira. The number of commits can just be copied from the release notes. What do you think?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] amol- commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

amol- commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r673961443



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,96 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format) and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+## C++ notes
+
+## C# notes
+
+## Go notes
+
+## Java notes
+
+## JavaScript notes
+
+## Python notes

Review comment:
       Here I tried to provide a short version of the major changes released for Python bindings -> https://docs.google.com/document/d/1O7TKle5PZxaFzXWZ_-siiy8RHmqPexTTOjjapgS4Yfk/edit?usp=sharing




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] liyafan82 commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

liyafan82 commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678781003



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,113 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format) and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+## C# notes
+
+## Go notes
+
+## Java notes

Review comment:
       Thanks for the good summary @BryanCutler 
   I want to add one more item @nealrichardson 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] ianmcook commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

ianmcook commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r682062655



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,267 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-29 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+3 months of development work and includes **684 commits** from
+[**99 distinct contributors**][1] in 2 repositories. See the Install Page to
+learn how to get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the complete changelogs for the [`apache/arrow`][2] and
+[`apache/arrow-rs`][3] repositories.
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format)
+and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend `.arrow` as the IPC file format file extension and `.arrows` for
+the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has
+been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+`MakeArrayFromScalar` now works for fixed-size binary types (ARROW-13321).
+
+### Compute layer
+
+The following [compute functions]({{site.baseurl}}/docs/cpp/compute.html)
+were added:
+
+* aggregations: `index`
+
+* scalar arithmetic and math functions: `abs`, `abs_checked`, `acos`,
+  `acos_checked`, `asin`, `asin_checked`, `atan`, `atan2`, `ceil`, `cos`,
+  `cos_checked`, `floor`, `ln`, `ln_checked`, `log10`, `log10_checked`,
+  `log1p`, `log1p_checked`, `log2`, `log2_checked`, `negate`, `negate_checked`,
+  `sign`, `sin`, `sin_checked`, `tan`, `tan_checked`, `trunc`
+
+* scalar bitwise functions: `bit_wise_and`, `bit_wise_not`, `bit_wise_or`,
+  `bit_wise_xor`, `shift_left`, `shift_left_checked`, `shift_right`,
+  `shift_right_checked`
+
+* scalar string functions: `ascii_center`, `ascii_lpad`, `ascii_reverse`,
+  `ascii_rpad`, `binary_join`, `binary_join_element_wise`,
+  `binary_replace_slice`, `count_substring`, `count_substring_regex`,
+  `ends_with`, `find_substring`, `find_substring_regex`, `match_like`,
+  `split_pattern_regex`, `starts_with`, `utf8_center`, `utf8_lpad`,
+  `utf8_replace_slice`, `utf8_rpad`, `utf8_reverse`, `utf8_slice_codepoints`
+
+* scalar temporal functions: `day`, `day_of_week`, `day_of_year`,
+  `iso_calendar`, `iso_week`, `iso_year`, `hour`, `microsecond`, `millisecond`,
+  `minute`, `month`, `nanosecond`, `quarter`, `second`, `subsecond`, `year`
+
+* other scalar functions: `case_when`, `coalesce`, `if_else`, `is_finite`,
+  `is_inf`, `is_nan`, `max_element_wise`, `min_element_wise`, `make_struct`
+
+* vector functions: `replace_with_mask`
+
+Duplicates are now allowed in `SetLookupOptions::value_set` (ARROW-12554).
+
+Decimal types are now supported by some basic arithmetic functions (ARROW-12074).
+
+The `take` function now supports dense unions (ARROW-13005).
+
+It is now possible to cast between dictionary types with different index
+types (ARROW-11673).
+
+Sorting is now implemented for boolean input (ARROW-12016).
+
+### CSV
+
+The streaming CSV reader can now take some advantage of multiple threads (ARROW-11889).
+
+The CSV reader tries to make its errors more informative by adding the
+row number when it is known, i.e. when parallel reading is disabled (ARROW-12675).
+
+A new option `ReaderOptions::skip_rows_after_names` allows skipping a number
+of rows _after_ reading the column names (as opposed to
+`ReaderOptions::skip_rows`).
+
+Quoted strings can now be treated as always non-null (ARROW-10115).
+
+### Dataset layer
+
+The asynchronous scanner introduced in 4.0.0 has been improved with truly 
+asynchronous readers implemented for CSV, Parquet, and IPC file formats and 
+file-level parallelism added.  This mode is controlled by a flag `use_async` that
+can be passed into methods which scan a dataset.  Setting this flag to True
+will have significant improvements on filesystems with high latency or parallel
+reads (e.g. S3).
+
+A CountRows method has been added to count rows matching a predicate; where
+possible, this will use metadata in files instead of reading the data itself.
+
+CSV datasets can now be written, and when reading a CSV dataset, explicit types can
+now be specified for a subset of columns while allowing the rest to still be inferred. 
+
+### IO and Filesystem layer
+
+The I/O thread pool size can now be adjusted at runtime (ARROW-12760).
+The default size remains 8 threads.
+
+Streams now can have auxiliary metadata, depending on the backend.  This
+has been implemented for the S3 filesystems, where a couple metadata
+keys are supported such as `Content-Type` and `ACL` (ARROW-11161, ARROW-12719).
+
+The HadoopFileSystem implementation now implements the FileSystem abstraction
+more faithfully (ARROW-12790).
+
+### Parquet
+
+The new LZ4_RAW compression scheme was implemented (PARQUET-1998).
+Unlike the legacy LZ4 compression scheme, it is defined unambiguously
+and should provide better portability once other Parquet implementations
+catch up.
+
+## Go notes
+
+
+### Flight 
+
+* Flight Client and Server now support Custom Metadata through the functions `flight.NewClientWithMiddleware` and `flight.NewServerWithMiddleware`. Functions
+`flight.NewFlightClient`, `flight.NewFlightServer`, `flight.CreateServerBearerTokenAuthInterceptors` have been deprecated in favor of using the new middleware. [#10633](https://github.com/apache/arrow/pull/10633)
+* Flight Client `AuthHandler` no longer overwrites outgoing metadata, correctly appending new metadata without overwriting existing metadata [#10297](https://github.com/apache/arrow/pull/10297)
+* Flight AppMetadata field is now exposed both for Reading and Writing via `flight.Reader#LatestAppMetadata()` and `flight.Writer#WriteWithAppMetadata` functions [#10142](https://github.com/apache/arrow/pull/10142)
+
+### Other enhancements 
+
+* [Map](https://github.com/apache/arrow/pull/10106) and [Extension](https://github.com/apache/arrow/pull/10203) Datatypes are now implemented for Arrow Arrays 
+* [Schema package](https://github.com/apache/arrow/10071) and first part of [Encoding package](https://github.com/apache/arrow/10379) added for Golang Parquet Implementation
+
+## Java notes
+
+Highlighted improvements and fixes:
+
+* Improved support for extension types using a complex storage type, e.g. struct, map or union. These can now extend
+the `ExtensionTypeVector` base class.
+* Union vectors now extend `AbstractContainerVector` to be consistent with other vectors.
+* Guava dependency updated to 30.1.1
+* Memory leak fixed if an exception occurs when reading IPC messages from a channel. [#10423](https://github.com/apache/arrow/pull/10423)
+* Flight error metadata is now propagated to the client. [#10370](https://github.com/apache/arrow/pull/10370)
+* JDBC adapter now preserves nullability. [#10285](https://github.com/apache/arrow/pull/10285)
+* The memory rounding policy is respected when allocating vector buffers. This helps saving memory space. [#13147] (https://github.com/apache/arrow/pull/10576)

Review comment:
       ```suggestion
   * The memory rounding policy is respected when allocating vector buffers. This helps saving memory space. [#13147](https://github.com/apache/arrow/pull/10576)
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] westonpace commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

westonpace commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678553219



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,229 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format)
+and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for
+the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has
+been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+`MakeArrayFromScalar` now works for fixed-size binary types (ARROW-13321).
+
+### Compute layer
+
+The following [compute functions]({{site.baseurl}}/docs/cpp/compute.html)
+were added:
+
+* aggregations: "index"
+
+* scalar arithmetic and math functions: "abs", "abs_checked", "acos",
+  "acos_checked", "asin", "asin_checked", "atan", "atan2", "ceil", "cos",
+  "cos_checked", "floor", "ln", "ln_checked", "log10", "log10_checked",
+  "log1p", "log1p_checked", "log2", "log2_checked", "negate", "negate_checked",
+  "sign", "sin", "sin_checked", "tan", "tan_checked", "trunc"
+
+* scalar bitwise functions: "bit_wise_and", "bit_wise_not", "bit_wise_or",
+  "bit_wise_xor", "shift_left", "shift_left_checked", "shift_right",
+  "shift_right_checked"
+
+* scalar string functions: "ascii_center", "ascii_lpad", "ascii_reverse",
+  "ascii_rpad", "binary_join", "binary_join_element_wise",
+  "binary_replace_slice", "count_substring", "count_substring_regex",
+  "ends_with", "find_substring", "find_substring_regex", "match_like",
+  "split_pattern_regex", "starts_with", "utf8_center", "utf8_lpad",
+  "utf8_replace_slice", "utf8_rpad", "utf8_reverse", "utf8_slice_codepoints"
+
+* scalar temporal functions: "day", "day_of_week", "day_of_year",
+  "iso_calendar", "iso_week", "iso_year", "hour", "microsecond", "millisecond",
+  "minute", "month", "nanosecond", "quarter", "second", "subsecond", "year"
+
+* other scalar functions: "case_when", "coalesce", "if_else", "is_finite",
+  "is_inf", "is_nan", "max_element_wise", "min_element_wise", "make_struct"
+
+* vector functions: "replace_with_mask"
+
+Duplicates are now allowed in `SetLookupOptions::value_set` (ARROW-12554).
+
+Decimal types are now supported by some basic arithmetic functions (ARROW-12074).
+
+The "take" function now supports dense unions (ARROW-13005).
+
+It is now possible to cast between dictionary types with different index
+types (ARROW-11673).
+
+Sorting is now implemented for boolean input (ARROW-12016).
+
+### CSV
+
+The streaming CSV reader can now read in parallel (ARROW-11889).
+
+The CSV reader tries to make its errors more informative by adding the
+row number when it is known, i.e. when parallel reading is disabled (ARROW-12675).
+
+A new option `ReaderOptions::skip_rows_after_names` allows skipping a number
+of rows _after_ reading the column names (as opposed to
+`ReaderOptions::skip_rows`).
+
+Quoted strings can now be treated as always non-null (ARROW-10115).
+
+### Dataset layer
+

Review comment:
       The asynchronous scanner was never really introduced (as, explained to the users) in 4.0.0.  It seems the only reference to it in the 4.0.0 docs was [this](https://arrow.apache.org/docs/cpp/api/dataset.html#_CPPv4N5arrow7dataset11ScanOptions9use_asyncE).  Maybe we can add a bit to this.
   
   ```
   The asynchronous scanner introduced in 4.0.0 has been improved, with truly asynchronous readers
   implemented for CSV, Parquet, and IPC file formats and file-level parallelism added.  This mode is controlled by
   a flag `use_async` that can be passed into methods which scan a dataset.  Setting this flag to True will have
   significant improvements on filesystems with high latency and generally some smaller improvements elsewhere. 
   This flag will default to True in some future version.
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] nealrichardson commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

nealrichardson commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678584346



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,259 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+3 months of development work and includes **XX commits** from
+[**XX distinct contributors**][1] in 2 repositories. See the Install Page to
+learn how to get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the complete changelogs for the [`apache/arrow`][2] and
+[`apache/arrow-rs`][3] repositories.
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format)
+and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend `.arrow` as the IPC file format file extension and `.arrows` for
+the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has
+been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+`MakeArrayFromScalar` now works for fixed-size binary types (ARROW-13321).
+
+### Compute layer
+
+The following [compute functions]({{site.baseurl}}/docs/cpp/compute.html)
+were added:
+
+* aggregations: `index`
+
+* scalar arithmetic and math functions: `abs`, `abs_checked`, `acos`,
+  `acos_checked`, `asin`, `asin_checked`, `atan`, `atan2`, `ceil`, `cos`,
+  `cos_checked`, `floor`, `ln`, `ln_checked`, `log10`, `log10_checked`,
+  `log1p`, `log1p_checked`, `log2`, `log2_checked`, `negate`, `negate_checked`,
+  `sign`, `sin`, `sin_checked`, `tan`, `tan_checked`, `trunc`
+
+* scalar bitwise functions: `bit_wise_and`, `bit_wise_not`, `bit_wise_or`,
+  `bit_wise_xor`, `shift_left`, `shift_left_checked`, `shift_right`,
+  `shift_right_checked`
+
+* scalar string functions: `ascii_center`, `ascii_lpad`, `ascii_reverse`,
+  `ascii_rpad`, `binary_join`, `binary_join_element_wise`,
+  `binary_replace_slice`, `count_substring`, `count_substring_regex`,
+  `ends_with`, `find_substring`, `find_substring_regex`, `match_like`,
+  `split_pattern_regex`, `starts_with`, `utf8_center`, `utf8_lpad`,
+  `utf8_replace_slice`, `utf8_rpad`, `utf8_reverse`, `utf8_slice_codepoints`
+
+* scalar temporal functions: `day`, `day_of_week`, `day_of_year`,
+  `iso_calendar`, `iso_week`, `iso_year`, `hour`, `microsecond`, `millisecond`,
+  `minute`, `month`, `nanosecond`, `quarter`, `second`, `subsecond`, `year`
+
+* other scalar functions: `case_when`, `coalesce`, `if_else`, `is_finite`,
+  `is_inf`, `is_nan`, `max_element_wise`, `min_element_wise`, `make_struct`
+
+* vector functions: `replace_with_mask`
+
+Duplicates are now allowed in `SetLookupOptions::value_set` (ARROW-12554).
+
+Decimal types are now supported by some basic arithmetic functions (ARROW-12074).
+
+The `take` function now supports dense unions (ARROW-13005).
+
+It is now possible to cast between dictionary types with different index
+types (ARROW-11673).
+
+Sorting is now implemented for boolean input (ARROW-12016).
+
+### CSV
+
+The streaming CSV reader can now take some advantage of multiple threads (ARROW-11889).
+
+The CSV reader tries to make its errors more informative by adding the
+row number when it is known, i.e. when parallel reading is disabled (ARROW-12675).
+
+A new option `ReaderOptions::skip_rows_after_names` allows skipping a number
+of rows _after_ reading the column names (as opposed to
+`ReaderOptions::skip_rows`).
+
+Quoted strings can now be treated as always non-null (ARROW-10115).
+
+### Dataset layer
+
+### IO and Filesystem layer
+
+The I/O thread pool size can now be adjuted at runtime (ARROW-12760).
+The default size remains 8 threads.
+
+Streams now can have auxiliary metadata, depending on the backend.  This
+has been implemented for the S3 filesystems, where a couple metadata
+keys are supported such as `Content-Type` and `ACL` (ARROW-11161, ARROW-12719).
+
+The HadoopFileSystem implementation now implements the FileSystem abstraction
+more faithfully (ARROW-12790).
+
+### Parquet
+
+The new LZ4_RAW compression scheme was implemented (PARQUET-1998).
+Unlike the legacy LZ4 compression scheme, it is defined unambiguously
+and should provide better portability once other Parquet implementations
+catch up.
+
+## Go notes
+
+* Flight Client and Server now support Custom Metadata through the functions `flight.NewClientWithMiddleware` and `flight.NewServerWithMiddleware`. Functions
+`flight.NewFlightClient`, `flight.NewFlightServer`, `flight.CreateServerBearerTokenAuthInterceptors` have been deprecated in favor of using the new middleware. [#10633](https://github.com/apache/arrow/pull/10633)
+* Fixed build failure for Parquet implementation on s390x [#10475](https://github.com/apache/arrow/pull/10475)
+* Flight Client `AuthHandler` no longer overwrites outgoing metadata, correctly appending new metadata without overwriting existing metadata [#10297](https://github.com/apache/arrow/pull/10297)
+* Flight AppMetadata field is now exposed both for Reading and Writing via `flight.Reader#LatestAppMetadata()` and `flight.Writer#WriteWithAppMetadata` functions [#10142](https://github.com/apache/arrow/pull/10142)
+* Extension Datatype is now implemented for Arrow Arrays [#10203](https://github.com/apache/arrow/pull/10203)
+* Map Datatype is now implemented for Arrow Arrays [#10106](https://github.com/apache/arrow/pull/10106)
+* Decimal integration tests with Go as both producer or consumer have been fixed [#10116](https://github.com/apache/arrow/10116)
+* Schema Package added for Golang Parquet Implementation [#10071](https://github.com/apache/arrow/10071)
+* First part of Encoding package for Golang Parquet Implementation added [#10379](https://github.com/apache/arrow/10379)
+
+## Java notes
+
+Highlighted improvements and fixes:
+
+* Improved support for extension types using a complex storage type, e.g. struct, map or union. These can now extend
+the `ExtensionTypeVector` base class.
+* Union vectors now extend `AbstractContainerVector` to be consistent with other vectors.
+* Guava dependency updated to 30.1.1
+* Memory leak fixed if an exception occurs when reading IPC messages from a channel. [#10423](https://github.com/apache/arrow/pull/10423)
+* Flight error metadata is now propagated to the client. [#10370](https://github.com/apache/arrow/pull/10370)
+* JDBC adapter now preserves nullability. [#10285](https://github.com/apache/arrow/pull/10285)
+
+API compatibility changes:
+
+* Complex vectors now return covariant types from `getObject(int)`. [#9964](https://github.com/apache/arrow/pull/9964)
+
+## JavaScript notes
+
+* Tables do not extend DataFrames anymore. This enables smaller bundles. [#10277](https://github.com/apache/arrow/pull/10277)
+* Arrow uses closure compiler for all UMD bundles, making them smaller. [#10281](https://github.com/apache/arrow/pull/10281)
+* The npm package now comes with declaration maps for better navigation from types to source code. [#10673](https://github.com/apache/arrow/pull/10673)
+* Updated dependencies and improvements to the code.
+
+## Python notes
+
+* Datasets can now scan files asynchronously when the `use_async=True` option is provided to `Dataset.scanner`, `Dataset.to_table`, or `Dataset.to_batches` methods. This should provide better performance in environments where I/O can be slow, such as with remote sources.
+* Arrow now provides builtin support for writing CSV files through `pyarrow.csv.write_csv`
+* Wheels for Apple M1 Macs are now provided.
+* Many new `pyarrow.compute` functions are available, and introspection of the functions was improved so that they look more like standard Python functions.
+* It is now possible to access ORC file metadata from Python `ORCFile` objects
+* Building a `StructArray` now accepts a `mask` like other arrays
+* Many updates and fixes for the documentation
+
+## R notes
+
+In this release, we've more than doubled the number of functions you can call on Arrow Datasets inside `dplyr::filter()` and `mutate()`, including many more string, datetime, and math functions. You can also write Datasets to CSV files, in addition to Parquet and Feather. We've also deepened support for the Arrow C interface, which is used in the Python interface and allows integration with other projects, such as DuckDB.
+
+For more on what’s in the 5.0.0 R package, see the [R changelog][4].
+
+## Ruby and C GLib notes
+
+Apache Arrow Flight support is started. But `ListFlights` is only supported for now. More features will be implemented in the next major release.
+
+### Ruby
+
+You need gobject-introspection gem 3.4.5 or later to implement your Apache Arrow Flight server. If you only use Apache Arrow Flight client, gobject-introspection gem 3.4.5 or later isn't required.
+
+Here are highlighted improvements:
+
+* Compute functions accept raw Ruby objects such as `true`, `Integer`, `Array` and `String`:
+
+  ```ruby
+  add_function = Arrow::Function.find("add")
+  # Not shortcut version
+  augend = Arrow::Int8Array.new([1, 2, 3])
+  addend = Arrow::Int8Scalar.new(5)
+  args = [
+    Arrow::ArrayDatum.new(augend),
+    Arrow::ScalarDatum.new(addend),
+  ]
+  add_function.execute(args).value.to_a # => [6, 7, 8]
+  # Shortcut version
+  add_function.execute([[1, 2, 3], 5]).value.to_a # => [6, 7, 8]
+  ```
+* `Arrow::PrimaryArray` and `Arrow::Buffer` can be used as MemoryView that is added in Ruby 3.0.
+
+There are some backward incompatible changes:
+
+* `Arrow::CountOptions` and `Arrow::CountMode` are removed. Use `Arrow::ScalarAggregateOptions` instead.
+
+### C GLib
+
+There are some backward incompatible changes:
+
+* `GArrowCountOptions` and `GArrowCountMode` are removed. Use `GArrowScalarAggregateOptions` instead.
+* `garrow_array_equal_range()` requires `GArrowEqualOptions`.
+* Prefix in arrow-dataset-glib is changed to `gadataset_`/`GADATASET_` from `gad_`/`GAD_`.
+* `GADScanOptions`, `GADScanTask`  and `GADInMemoryScanTask` are removed. Use `gadataset_begin_scan()` or `gadataset_to_table()` instead.
+* `GArrowCompareOptions`, `GArrowCompareOperator` and `garrow_*_array_compare()` are removed. Use `equal`, `not_equal`, `less_than`, `less_than_equal`, `greater_than` and `greater_than_equal` compute functions directly instead.
+
+## Rust notes
+
+The Rust projects have moved to separate repositories outside the
+main Arrow monorepo. For notes on the 5.0.0 release of the Rust
+implementation, see the [Arrow Rust changelog][3] and the
+"Apache Arrow Rust 5.0.0 Release" blog post.

Review comment:
       TODO: link to that, when published




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] lidavidm commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

lidavidm commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678497249



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,229 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format)
+and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for
+the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has
+been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+`MakeArrayFromScalar` now works for fixed-size binary types (ARROW-13321).
+
+### Compute layer
+
+The following [compute functions]({{site.baseurl}}/docs/cpp/compute.html)
+were added:
+
+* aggregations: "index"
+
+* scalar arithmetic and math functions: "abs", "abs_checked", "acos",
+  "acos_checked", "asin", "asin_checked", "atan", "atan2", "ceil", "cos",
+  "cos_checked", "floor", "ln", "ln_checked", "log10", "log10_checked",
+  "log1p", "log1p_checked", "log2", "log2_checked", "negate", "negate_checked",
+  "sign", "sin", "sin_checked", "tan", "tan_checked", "trunc"
+
+* scalar bitwise functions: "bit_wise_and", "bit_wise_not", "bit_wise_or",
+  "bit_wise_xor", "shift_left", "shift_left_checked", "shift_right",
+  "shift_right_checked"
+
+* scalar string functions: "ascii_center", "ascii_lpad", "ascii_reverse",
+  "ascii_rpad", "binary_join", "binary_join_element_wise",
+  "binary_replace_slice", "count_substring", "count_substring_regex",
+  "ends_with", "find_substring", "find_substring_regex", "match_like",
+  "split_pattern_regex", "starts_with", "utf8_center", "utf8_lpad",
+  "utf8_replace_slice", "utf8_rpad", "utf8_reverse", "utf8_slice_codepoints"
+
+* scalar temporal functions: "day", "day_of_week", "day_of_year",
+  "iso_calendar", "iso_week", "iso_year", "hour", "microsecond", "millisecond",
+  "minute", "month", "nanosecond", "quarter", "second", "subsecond", "year"
+
+* other scalar functions: "case_when", "coalesce", "if_else", "is_finite",
+  "is_inf", "is_nan", "max_element_wise", "min_element_wise", "make_struct"
+
+* vector functions: "replace_with_mask"
+
+Duplicates are now allowed in `SetLookupOptions::value_set` (ARROW-12554).
+
+Decimal types are now supported by some basic arithmetic functions (ARROW-12074).
+
+The "take" function now supports dense unions (ARROW-13005).
+
+It is now possible to cast between dictionary types with different index
+types (ARROW-11673).
+
+Sorting is now implemented for boolean input (ARROW-12016).
+
+### CSV
+
+The streaming CSV reader can now read in parallel (ARROW-11889).
+
+The CSV reader tries to make its errors more informative by adding the
+row number when it is known, i.e. when parallel reading is disabled (ARROW-12675).
+
+A new option `ReaderOptions::skip_rows_after_names` allows skipping a number
+of rows _after_ reading the column names (as opposed to
+`ReaderOptions::skip_rows`).
+
+Quoted strings can now be treated as always non-null (ARROW-10115).
+
+### Dataset layer
+

Review comment:
       ```suggestion
   The asynchronous scanner introduced in 4.0.0 has been improved, with truly asynchronous readers implemented for CSV, Parquet, and IPC file formats and file-level parallelism added.
   
   A CountRows method has been added to count rows matching a predicate; where possible, this will use metadata in files instead of reading the data itself.
   
   CSV datasets can now be written, and when reading a CSV dataset, explicit types can now be specified for a subset of columns while allowing the rest to still be inferred.
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] github-actions[bot] commented on pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

github-actions[bot] commented on pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#issuecomment-881611095


   <!--
     Licensed to the Apache Software Foundation (ASF) under one
     or more contributor license agreements.  See the NOTICE file
     distributed with this work for additional information
     regarding copyright ownership.  The ASF licenses this file
     to you under the Apache License, Version 2.0 (the
     "License"); you may not use this file except in compliance
     with the License.  You may obtain a copy of the License at
   
       http://www.apache.org/licenses/LICENSE-2.0
   
     Unless required by applicable law or agreed to in writing,
     software distributed under the License is distributed on an
     "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
     KIND, either express or implied.  See the License for the
     specific language governing permissions and limitations
     under the License.
   -->
   
   Thanks for opening a pull request!
   
   Could you open an issue for this pull request on JIRA?
   https://issues.apache.org/jira/browse/ARROW
   
   Then could you also rename pull request title in the following format?
   
       ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}
   
   See also:
   
     * [Other pull requests](https://github.com/apache/arrow-site/pulls/)
     * [Contribution Guidelines - How to contribute patches](https://arrow.apache.org/docs/developers/contributing.html#how-to-contribute-patches)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] westonpace commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

westonpace commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678556014



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,248 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+3 months of development work and includes **XX commits** from
+[**XX distinct contributors**][1] in 2 repositories. See the Install Page to
+learn how to get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the complete changelogs for the [`apache/arrow`][2] and
+[`apache/arrow-rs`][3] repositories.
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format)
+and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend `.arrow` as the IPC file format file extension and `.arrows` for
+the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has
+been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+`MakeArrayFromScalar` now works for fixed-size binary types (ARROW-13321).
+
+### Compute layer
+
+The following [compute functions]({{site.baseurl}}/docs/cpp/compute.html)
+were added:
+
+* aggregations: `index`
+
+* scalar arithmetic and math functions: `abs`, `abs_checked`, `acos`,
+  `acos_checked`, `asin`, `asin_checked`, `atan`, `atan2`, `ceil`, `cos`,
+  `cos_checked`, `floor`, `ln`, `ln_checked`, `log10`, `log10_checked`,
+  `log1p`, `log1p_checked`, `log2`, `log2_checked`, `negate`, `negate_checked`,
+  `sign`, `sin`, `sin_checked`, `tan`, `tan_checked`, `trunc`
+
+* scalar bitwise functions: `bit_wise_and`, `bit_wise_not`, `bit_wise_or`,
+  `bit_wise_xor`, `shift_left`, `shift_left_checked`, `shift_right`,
+  `shift_right_checked`
+
+* scalar string functions: `ascii_center`, `ascii_lpad`, `ascii_reverse`,
+  `ascii_rpad`, `binary_join`, `binary_join_element_wise`,
+  `binary_replace_slice`, `count_substring`, `count_substring_regex`,
+  `ends_with`, `find_substring`, `find_substring_regex`, `match_like`,
+  `split_pattern_regex`, `starts_with`, `utf8_center`, `utf8_lpad`,
+  `utf8_replace_slice`, `utf8_rpad`, `utf8_reverse`, `utf8_slice_codepoints`
+
+* scalar temporal functions: `day`, `day_of_week`, `day_of_year`,
+  `iso_calendar`, `iso_week`, `iso_year`, `hour`, `microsecond`, `millisecond`,
+  `minute`, `month`, `nanosecond`, `quarter`, `second`, `subsecond`, `year`
+
+* other scalar functions: `case_when`, `coalesce`, `if_else`, `is_finite`,
+  `is_inf`, `is_nan`, `max_element_wise`, `min_element_wise`, `make_struct`
+
+* vector functions: `replace_with_mask`
+
+Duplicates are now allowed in `SetLookupOptions::value_set` (ARROW-12554).
+
+Decimal types are now supported by some basic arithmetic functions (ARROW-12074).
+
+The `take` function now supports dense unions (ARROW-13005).
+
+It is now possible to cast between dictionary types with different index
+types (ARROW-11673).
+
+Sorting is now implemented for boolean input (ARROW-12016).
+
+### CSV
+
+The streaming CSV reader can now read in parallel (ARROW-11889).

Review comment:
       ```suggestion
   The streaming CSV reader can now take some advantage of multiple threads (ARROW-11889).
   ```
   Probably not safe to say it can read in parallel until ARROW-13155 is released.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] pitrou commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

pitrou commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678263745



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,113 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format) and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes

Review comment:
       Done.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] ianmcook commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

ianmcook commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r681879683



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,267 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-29 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+3 months of development work and includes **684 commits** from
+[**99 distinct contributors**][1] in 2 repositories. See the Install Page to
+learn how to get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the complete changelogs for the [`apache/arrow`][2] and
+[`apache/arrow-rs`][3] repositories.
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format)
+and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend `.arrow` as the IPC file format file extension and `.arrows` for
+the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has
+been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+`MakeArrayFromScalar` now works for fixed-size binary types (ARROW-13321).
+
+### Compute layer
+
+The following [compute functions]({{site.baseurl}}/docs/cpp/compute.html)
+were added:
+
+* aggregations: `index`
+
+* scalar arithmetic and math functions: `abs`, `abs_checked`, `acos`,
+  `acos_checked`, `asin`, `asin_checked`, `atan`, `atan2`, `ceil`, `cos`,
+  `cos_checked`, `floor`, `ln`, `ln_checked`, `log10`, `log10_checked`,
+  `log1p`, `log1p_checked`, `log2`, `log2_checked`, `negate`, `negate_checked`,
+  `sign`, `sin`, `sin_checked`, `tan`, `tan_checked`, `trunc`
+
+* scalar bitwise functions: `bit_wise_and`, `bit_wise_not`, `bit_wise_or`,
+  `bit_wise_xor`, `shift_left`, `shift_left_checked`, `shift_right`,
+  `shift_right_checked`
+
+* scalar string functions: `ascii_center`, `ascii_lpad`, `ascii_reverse`,
+  `ascii_rpad`, `binary_join`, `binary_join_element_wise`,
+  `binary_replace_slice`, `count_substring`, `count_substring_regex`,
+  `ends_with`, `find_substring`, `find_substring_regex`, `match_like`,
+  `split_pattern_regex`, `starts_with`, `utf8_center`, `utf8_lpad`,
+  `utf8_replace_slice`, `utf8_rpad`, `utf8_reverse`, `utf8_slice_codepoints`
+
+* scalar temporal functions: `day`, `day_of_week`, `day_of_year`,
+  `iso_calendar`, `iso_week`, `iso_year`, `hour`, `microsecond`, `millisecond`,
+  `minute`, `month`, `nanosecond`, `quarter`, `second`, `subsecond`, `year`
+
+* other scalar functions: `case_when`, `coalesce`, `if_else`, `is_finite`,
+  `is_inf`, `is_nan`, `max_element_wise`, `min_element_wise`, `make_struct`
+
+* vector functions: `replace_with_mask`
+
+Duplicates are now allowed in `SetLookupOptions::value_set` (ARROW-12554).
+
+Decimal types are now supported by some basic arithmetic functions (ARROW-12074).
+
+The `take` function now supports dense unions (ARROW-13005).
+
+It is now possible to cast between dictionary types with different index
+types (ARROW-11673).
+
+Sorting is now implemented for boolean input (ARROW-12016).
+
+### CSV
+
+The streaming CSV reader can now take some advantage of multiple threads (ARROW-11889).
+
+The CSV reader tries to make its errors more informative by adding the
+row number when it is known, i.e. when parallel reading is disabled (ARROW-12675).
+
+A new option `ReaderOptions::skip_rows_after_names` allows skipping a number
+of rows _after_ reading the column names (as opposed to
+`ReaderOptions::skip_rows`).
+
+Quoted strings can now be treated as always non-null (ARROW-10115).
+
+### Dataset layer
+
+The asynchronous scanner introduced in 4.0.0 has been improved with truly 
+asynchronous readers implemented for CSV, Parquet, and IPC file formats and 
+file-level parallelism added.  This mode is controlled by a flag `use_async` that
+can be passed into methods which scan a dataset.  Setting this flag to True
+will have significant improvements on filesystems with high latency or parallel
+reads (e.g. S3).
+
+A CountRows method has been added to count rows matching a predicate; where
+possible, this will use metadata in files instead of reading the data itself.
+
+CSV datasets can now be written, and when reading a CSV dataset, explicit types can
+now be specified for a subset of columns while allowing the rest to still be inferred. 
+
+### IO and Filesystem layer
+
+The I/O thread pool size can now be adjusted at runtime (ARROW-12760).
+The default size remains 8 threads.
+
+Streams now can have auxiliary metadata, depending on the backend.  This
+has been implemented for the S3 filesystems, where a couple metadata
+keys are supported such as `Content-Type` and `ACL` (ARROW-11161, ARROW-12719).
+
+The HadoopFileSystem implementation now implements the FileSystem abstraction
+more faithfully (ARROW-12790).
+
+### Parquet
+
+The new LZ4_RAW compression scheme was implemented (PARQUET-1998).
+Unlike the legacy LZ4 compression scheme, it is defined unambiguously
+and should provide better portability once other Parquet implementations
+catch up.
+
+## Go notes
+
+
+### Flight 
+
+* Flight Client and Server now support Custom Metadata through the functions `flight.NewClientWithMiddleware` and `flight.NewServerWithMiddleware`. Functions
+`flight.NewFlightClient`, `flight.NewFlightServer`, `flight.CreateServerBearerTokenAuthInterceptors` have been deprecated in favor of using the new middleware. [#10633](https://github.com/apache/arrow/pull/10633)
+* Flight Client `AuthHandler` no longer overwrites outgoing metadata, correctly appending new metadata without overwriting existing metadata [#10297](https://github.com/apache/arrow/pull/10297)
+* Flight AppMetadata field is now exposed both for Reading and Writing via `flight.Reader#LatestAppMetadata()` and `flight.Writer#WriteWithAppMetadata` functions [#10142](https://github.com/apache/arrow/pull/10142)
+
+### Other enhancements 
+
+* [Map](https://github.com/apache/arrow/pull/10106) and [Extension](https://github.com/apache/arrow/pull/10203) Datatypes are now implemented for Arrow Arrays 
+* [Schema package](https://github.com/apache/arrow/10071) and first part of [Encoding package](https://github.com/apache/arrow/10379) added for Golang Parquet Implementation
+
+## Java notes
+
+Highlighted improvements and fixes:
+
+* Improved support for extension types using a complex storage type, e.g. struct, map or union. These can now extend
+the `ExtensionTypeVector` base class.
+* Union vectors now extend `AbstractContainerVector` to be consistent with other vectors.
+* Guava dependency updated to 30.1.1
+* Memory leak fixed if an exception occurs when reading IPC messages from a channel. [#10423](https://github.com/apache/arrow/pull/10423)
+* Flight error metadata is now propagated to the client. [#10370](https://github.com/apache/arrow/pull/10370)
+* JDBC adapter now preserves nullability. [#10285](https://github.com/apache/arrow/pull/10285)
+* The memory rounding policy is respected when allocating vector buffers. This helps saving memory space. [#13147] (https://github.com/apache/arrow/pull/10576)
+
+API compatibility changes:
+
+* Complex vectors now return covariant types from `getObject(int)`. [#9964](https://github.com/apache/arrow/pull/9964)
+
+## JavaScript notes
+
+* Tables do not extend DataFrames anymore. This enables smaller bundles. [#10277](https://github.com/apache/arrow/pull/10277)
+* Arrow uses closure compiler for all UMD bundles, making them smaller. [#10281](https://github.com/apache/arrow/pull/10281)
+* The npm package now comes with declaration maps for better navigation from types to source code. [#10673](https://github.com/apache/arrow/pull/10673)
+* Updated dependencies and improvements to the code.
+
+## Python notes
+
+* Datasets can now scan files asynchronously when the `use_async=True` option is provided to `Dataset.scanner`, `Dataset.to_table`, or `Dataset.to_batches` methods. This should provide better performance in environments where I/O can be slow, such as with remote sources.
+* Arrow now provides builtin support for writing CSV files through `pyarrow.csv.write_csv`
+* Wheels for Apple M1 Macs are now provided.
+* Many new `pyarrow.compute` functions are available (see the C++ notes above
+  for more details), and introspection of the functions was improved so that
+  they look more like standard Python functions.
+* It is now possible to access ORC file metadata from Python `ORCFile` objects
+* Building a `StructArray` now accepts a `mask` like other arrays
+* Many updates and fixes for the documentation
+
+## R notes
+
+In this release, we've more than doubled the number of functions you can call on Arrow Datasets inside `dplyr::filter()` and `mutate()`, including many more string, datetime, and math functions. You can also write Datasets to CSV files, in addition to Parquet and Feather. We've also deepened support for the Arrow C interface, which is used in the Python interface and allows integration with other projects, such as DuckDB.
+
+For more on what’s in the 5.0.0 R package, see the [R changelog][4].
+
+## Ruby and C GLib notes
+
+Apache Arrow Flight support is started. But `ListFlights` is only supported for now. More features will be implemented in the next major release.
+
+### Ruby
+
+You need gobject-introspection gem 3.4.5 or later to implement your Apache Arrow Flight server. If you only use Apache Arrow Flight client, gobject-introspection gem 3.4.5 or later isn't required.
+
+Here are highlighted improvements:
+
+* Compute functions accept raw Ruby objects such as `true`, `Integer`, `Array` and `String`:
+
+  ```ruby
+  add_function = Arrow::Function.find("add")
+  # Not shortcut version
+  augend = Arrow::Int8Array.new([1, 2, 3])
+  addend = Arrow::Int8Scalar.new(5)
+  args = [
+    Arrow::ArrayDatum.new(augend),
+    Arrow::ScalarDatum.new(addend),
+  ]
+  add_function.execute(args).value.to_a # => [6, 7, 8]
+  # Shortcut version
+  add_function.execute([[1, 2, 3], 5]).value.to_a # => [6, 7, 8]
+  ```
+* `Arrow::PrimaryArray` and `Arrow::Buffer` can be used as MemoryView that is added in Ruby 3.0.
+
+There are some backward incompatible changes:
+
+* `Arrow::CountOptions` and `Arrow::CountMode` are removed. Use `Arrow::ScalarAggregateOptions` instead.
+
+### C GLib
+
+There are some backward incompatible changes:
+
+* `GArrowCountOptions` and `GArrowCountMode` are removed. Use `GArrowScalarAggregateOptions` instead.
+* `garrow_array_equal_range()` requires `GArrowEqualOptions`.
+* Prefix in arrow-dataset-glib is changed to `gadataset_`/`GADATASET_` from `gad_`/`GAD_`.
+* `GADScanOptions`, `GADScanTask`  and `GADInMemoryScanTask` are removed. Use `gadataset_begin_scan()` or `gadataset_to_table()` instead.
+* `GArrowCompareOptions`, `GArrowCompareOperator` and `garrow_*_array_compare()` are removed. Use `equal`, `not_equal`, `less_than`, `less_than_equal`, `greater_than` and `greater_than_equal` compute functions directly instead.
+
+## Rust notes
+
+The Rust projects have moved to separate repositories outside the
+main Arrow monorepo. For notes on the 5.0.0 release of the Rust
+implementation, see the [Arrow Rust changelog][3] and the
+["Apache Arrow Rust 5.0.0 Release" blog post]({% post_url 2021-07-20-5.0.0-rs-release %}).

Review comment:
       Remove unneeded quotes
   ```suggestion
   [Apache Arrow Rust 5.0.0 Release blog post]({% post_url 2021-07-20-5.0.0-rs-release %}).
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] nealrichardson commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

nealrichardson commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r679202244



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,264 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+3 months of development work and includes **XX commits** from
+[**XX distinct contributors**][1] in 2 repositories. See the Install Page to

Review comment:
       @ianmcook would you mind also updating the template (at the top level of the repo) with these changes?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] lidavidm commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

lidavidm commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678497249



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,229 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format)
+and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for
+the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has
+been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+`MakeArrayFromScalar` now works for fixed-size binary types (ARROW-13321).
+
+### Compute layer
+
+The following [compute functions]({{site.baseurl}}/docs/cpp/compute.html)
+were added:
+
+* aggregations: "index"
+
+* scalar arithmetic and math functions: "abs", "abs_checked", "acos",
+  "acos_checked", "asin", "asin_checked", "atan", "atan2", "ceil", "cos",
+  "cos_checked", "floor", "ln", "ln_checked", "log10", "log10_checked",
+  "log1p", "log1p_checked", "log2", "log2_checked", "negate", "negate_checked",
+  "sign", "sin", "sin_checked", "tan", "tan_checked", "trunc"
+
+* scalar bitwise functions: "bit_wise_and", "bit_wise_not", "bit_wise_or",
+  "bit_wise_xor", "shift_left", "shift_left_checked", "shift_right",
+  "shift_right_checked"
+
+* scalar string functions: "ascii_center", "ascii_lpad", "ascii_reverse",
+  "ascii_rpad", "binary_join", "binary_join_element_wise",
+  "binary_replace_slice", "count_substring", "count_substring_regex",
+  "ends_with", "find_substring", "find_substring_regex", "match_like",
+  "split_pattern_regex", "starts_with", "utf8_center", "utf8_lpad",
+  "utf8_replace_slice", "utf8_rpad", "utf8_reverse", "utf8_slice_codepoints"
+
+* scalar temporal functions: "day", "day_of_week", "day_of_year",
+  "iso_calendar", "iso_week", "iso_year", "hour", "microsecond", "millisecond",
+  "minute", "month", "nanosecond", "quarter", "second", "subsecond", "year"
+
+* other scalar functions: "case_when", "coalesce", "if_else", "is_finite",
+  "is_inf", "is_nan", "max_element_wise", "min_element_wise", "make_struct"
+
+* vector functions: "replace_with_mask"
+
+Duplicates are now allowed in `SetLookupOptions::value_set` (ARROW-12554).
+
+Decimal types are now supported by some basic arithmetic functions (ARROW-12074).
+
+The "take" function now supports dense unions (ARROW-13005).
+
+It is now possible to cast between dictionary types with different index
+types (ARROW-11673).
+
+Sorting is now implemented for boolean input (ARROW-12016).
+
+### CSV
+
+The streaming CSV reader can now read in parallel (ARROW-11889).
+
+The CSV reader tries to make its errors more informative by adding the
+row number when it is known, i.e. when parallel reading is disabled (ARROW-12675).
+
+A new option `ReaderOptions::skip_rows_after_names` allows skipping a number
+of rows _after_ reading the column names (as opposed to
+`ReaderOptions::skip_rows`).
+
+Quoted strings can now be treated as always non-null (ARROW-10115).
+
+### Dataset layer
+

Review comment:
       ```suggestion
   The asynchronous scanner introduced in 4.0.0 has been improved with truly 
   asynchronous readers implemented for CSV, Parquet, and IPC file formats and 
   file-level parallelism added.  This mode is controlled by a flag `use_async` that
   can be passed into methods which scan a dataset.  Setting this flag to True
   will have significant improvements on filesystems with high latency or parallel
   reads (e.g. S3).
   
   A CountRows method has been added to count rows matching a predicate; where
   possible, this will use metadata in files instead of reading the data itself.
   
   CSV datasets can now be written, and when reading a CSV dataset, explicit types can
   now be specified for a subset of columns while allowing the rest to still be inferred. 
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] alamb commented on pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

alamb commented on pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#issuecomment-882851381


   Here is proposed 5.0 content for Rust: https://github.com/apache/arrow-site/pull/128
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] ianmcook commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

ianmcook commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678388328



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,229 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].

Review comment:
       ```suggestion
   The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
   3 months of development work and includes **XX commits** from
   [**XX distinct contributors**][1] in 2 repositories. See the Install Page to
   learn how to get the libraries for your platform.
   
   The release notes below are not exhaustive and only expose selected highlights
   of the release. Many other bugfixes and improvements have been made: we refer
   you to the complete changelogs for the [`apache/arrow`][2] and
   [`apache/arrow-rs`][3] repositories.
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] ianmcook merged pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

ianmcook merged pull request #127:
URL: https://github.com/apache/arrow-site/pull/127


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] nealrichardson commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

nealrichardson commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r679206385



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,259 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+3 months of development work and includes **XX commits** from
+[**XX distinct contributors**][1] in 2 repositories. See the Install Page to
+learn how to get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the complete changelogs for the [`apache/arrow`][2] and
+[`apache/arrow-rs`][3] repositories.
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format)
+and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend `.arrow` as the IPC file format file extension and `.arrows` for
+the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has
+been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+`MakeArrayFromScalar` now works for fixed-size binary types (ARROW-13321).
+
+### Compute layer
+
+The following [compute functions]({{site.baseurl}}/docs/cpp/compute.html)
+were added:
+
+* aggregations: `index`
+
+* scalar arithmetic and math functions: `abs`, `abs_checked`, `acos`,
+  `acos_checked`, `asin`, `asin_checked`, `atan`, `atan2`, `ceil`, `cos`,
+  `cos_checked`, `floor`, `ln`, `ln_checked`, `log10`, `log10_checked`,
+  `log1p`, `log1p_checked`, `log2`, `log2_checked`, `negate`, `negate_checked`,
+  `sign`, `sin`, `sin_checked`, `tan`, `tan_checked`, `trunc`
+
+* scalar bitwise functions: `bit_wise_and`, `bit_wise_not`, `bit_wise_or`,
+  `bit_wise_xor`, `shift_left`, `shift_left_checked`, `shift_right`,
+  `shift_right_checked`
+
+* scalar string functions: `ascii_center`, `ascii_lpad`, `ascii_reverse`,
+  `ascii_rpad`, `binary_join`, `binary_join_element_wise`,
+  `binary_replace_slice`, `count_substring`, `count_substring_regex`,
+  `ends_with`, `find_substring`, `find_substring_regex`, `match_like`,
+  `split_pattern_regex`, `starts_with`, `utf8_center`, `utf8_lpad`,
+  `utf8_replace_slice`, `utf8_rpad`, `utf8_reverse`, `utf8_slice_codepoints`
+
+* scalar temporal functions: `day`, `day_of_week`, `day_of_year`,
+  `iso_calendar`, `iso_week`, `iso_year`, `hour`, `microsecond`, `millisecond`,
+  `minute`, `month`, `nanosecond`, `quarter`, `second`, `subsecond`, `year`
+
+* other scalar functions: `case_when`, `coalesce`, `if_else`, `is_finite`,
+  `is_inf`, `is_nan`, `max_element_wise`, `min_element_wise`, `make_struct`
+
+* vector functions: `replace_with_mask`
+
+Duplicates are now allowed in `SetLookupOptions::value_set` (ARROW-12554).
+
+Decimal types are now supported by some basic arithmetic functions (ARROW-12074).
+
+The `take` function now supports dense unions (ARROW-13005).
+
+It is now possible to cast between dictionary types with different index
+types (ARROW-11673).
+
+Sorting is now implemented for boolean input (ARROW-12016).
+
+### CSV
+
+The streaming CSV reader can now take some advantage of multiple threads (ARROW-11889).
+
+The CSV reader tries to make its errors more informative by adding the
+row number when it is known, i.e. when parallel reading is disabled (ARROW-12675).
+
+A new option `ReaderOptions::skip_rows_after_names` allows skipping a number
+of rows _after_ reading the column names (as opposed to
+`ReaderOptions::skip_rows`).
+
+Quoted strings can now be treated as always non-null (ARROW-10115).
+
+### Dataset layer
+
+### IO and Filesystem layer
+
+The I/O thread pool size can now be adjuted at runtime (ARROW-12760).
+The default size remains 8 threads.
+
+Streams now can have auxiliary metadata, depending on the backend.  This
+has been implemented for the S3 filesystems, where a couple metadata
+keys are supported such as `Content-Type` and `ACL` (ARROW-11161, ARROW-12719).
+
+The HadoopFileSystem implementation now implements the FileSystem abstraction
+more faithfully (ARROW-12790).
+
+### Parquet
+
+The new LZ4_RAW compression scheme was implemented (PARQUET-1998).
+Unlike the legacy LZ4 compression scheme, it is defined unambiguously
+and should provide better portability once other Parquet implementations
+catch up.
+
+## Go notes
+
+* Flight Client and Server now support Custom Metadata through the functions `flight.NewClientWithMiddleware` and `flight.NewServerWithMiddleware`. Functions
+`flight.NewFlightClient`, `flight.NewFlightServer`, `flight.CreateServerBearerTokenAuthInterceptors` have been deprecated in favor of using the new middleware. [#10633](https://github.com/apache/arrow/pull/10633)
+* Fixed build failure for Parquet implementation on s390x [#10475](https://github.com/apache/arrow/pull/10475)
+* Flight Client `AuthHandler` no longer overwrites outgoing metadata, correctly appending new metadata without overwriting existing metadata [#10297](https://github.com/apache/arrow/pull/10297)
+* Flight AppMetadata field is now exposed both for Reading and Writing via `flight.Reader#LatestAppMetadata()` and `flight.Writer#WriteWithAppMetadata` functions [#10142](https://github.com/apache/arrow/pull/10142)
+* Extension Datatype is now implemented for Arrow Arrays [#10203](https://github.com/apache/arrow/pull/10203)
+* Map Datatype is now implemented for Arrow Arrays [#10106](https://github.com/apache/arrow/pull/10106)
+* Decimal integration tests with Go as both producer or consumer have been fixed [#10116](https://github.com/apache/arrow/10116)
+* Schema Package added for Golang Parquet Implementation [#10071](https://github.com/apache/arrow/10071)
+* First part of Encoding package for Golang Parquet Implementation added [#10379](https://github.com/apache/arrow/10379)
+
+## Java notes
+
+Highlighted improvements and fixes:
+
+* Improved support for extension types using a complex storage type, e.g. struct, map or union. These can now extend
+the `ExtensionTypeVector` base class.
+* Union vectors now extend `AbstractContainerVector` to be consistent with other vectors.
+* Guava dependency updated to 30.1.1
+* Memory leak fixed if an exception occurs when reading IPC messages from a channel. [#10423](https://github.com/apache/arrow/pull/10423)
+* Flight error metadata is now propagated to the client. [#10370](https://github.com/apache/arrow/pull/10370)
+* JDBC adapter now preserves nullability. [#10285](https://github.com/apache/arrow/pull/10285)
+
+API compatibility changes:
+
+* Complex vectors now return covariant types from `getObject(int)`. [#9964](https://github.com/apache/arrow/pull/9964)
+
+## JavaScript notes
+
+* Tables do not extend DataFrames anymore. This enables smaller bundles. [#10277](https://github.com/apache/arrow/pull/10277)
+* Arrow uses closure compiler for all UMD bundles, making them smaller. [#10281](https://github.com/apache/arrow/pull/10281)
+* The npm package now comes with declaration maps for better navigation from types to source code. [#10673](https://github.com/apache/arrow/pull/10673)
+* Updated dependencies and improvements to the code.
+
+## Python notes
+
+* Datasets can now scan files asynchronously when the `use_async=True` option is provided to `Dataset.scanner`, `Dataset.to_table`, or `Dataset.to_batches` methods. This should provide better performance in environments where I/O can be slow, such as with remote sources.
+* Arrow now provides builtin support for writing CSV files through `pyarrow.csv.write_csv`
+* Wheels for Apple M1 Macs are now provided.
+* Many new `pyarrow.compute` functions are available, and introspection of the functions was improved so that they look more like standard Python functions.
+* It is now possible to access ORC file metadata from Python `ORCFile` objects
+* Building a `StructArray` now accepts a `mask` like other arrays
+* Many updates and fixes for the documentation
+
+## R notes
+
+In this release, we've more than doubled the number of functions you can call on Arrow Datasets inside `dplyr::filter()` and `mutate()`, including many more string, datetime, and math functions. You can also write Datasets to CSV files, in addition to Parquet and Feather. We've also deepened support for the Arrow C interface, which is used in the Python interface and allows integration with other projects, such as DuckDB.
+
+For more on what’s in the 5.0.0 R package, see the [R changelog][4].
+
+## Ruby and C GLib notes
+
+Apache Arrow Flight support is started. But `ListFlights` is only supported for now. More features will be implemented in the next major release.
+
+### Ruby
+
+You need gobject-introspection gem 3.4.5 or later to implement your Apache Arrow Flight server. If you only use Apache Arrow Flight client, gobject-introspection gem 3.4.5 or later isn't required.
+
+Here are highlighted improvements:
+
+* Compute functions accept raw Ruby objects such as `true`, `Integer`, `Array` and `String`:
+
+  ```ruby
+  add_function = Arrow::Function.find("add")
+  # Not shortcut version
+  augend = Arrow::Int8Array.new([1, 2, 3])
+  addend = Arrow::Int8Scalar.new(5)
+  args = [
+    Arrow::ArrayDatum.new(augend),
+    Arrow::ScalarDatum.new(addend),
+  ]
+  add_function.execute(args).value.to_a # => [6, 7, 8]
+  # Shortcut version
+  add_function.execute([[1, 2, 3], 5]).value.to_a # => [6, 7, 8]
+  ```
+* `Arrow::PrimaryArray` and `Arrow::Buffer` can be used as MemoryView that is added in Ruby 3.0.
+
+There are some backward incompatible changes:
+
+* `Arrow::CountOptions` and `Arrow::CountMode` are removed. Use `Arrow::ScalarAggregateOptions` instead.
+
+### C GLib
+
+There are some backward incompatible changes:
+
+* `GArrowCountOptions` and `GArrowCountMode` are removed. Use `GArrowScalarAggregateOptions` instead.
+* `garrow_array_equal_range()` requires `GArrowEqualOptions`.
+* Prefix in arrow-dataset-glib is changed to `gadataset_`/`GADATASET_` from `gad_`/`GAD_`.
+* `GADScanOptions`, `GADScanTask`  and `GADInMemoryScanTask` are removed. Use `gadataset_begin_scan()` or `gadataset_to_table()` instead.
+* `GArrowCompareOptions`, `GArrowCompareOperator` and `garrow_*_array_compare()` are removed. Use `equal`, `not_equal`, `less_than`, `less_than_equal`, `greater_than` and `greater_than_equal` compute functions directly instead.
+
+## Rust notes
+
+The Rust projects have moved to separate repositories outside the
+main Arrow monorepo. For notes on the 5.0.0 release of the Rust
+implementation, see the [Arrow Rust changelog][3] and the
+"Apache Arrow Rust 5.0.0 Release" blog post.

Review comment:
       ```suggestion
   ["Apache Arrow Rust 5.0.0 Release" blog post]({% post_url 2021-07-20-5.0.0-rs-release %}).
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] ianmcook merged pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

ianmcook merged pull request #127:
URL: https://github.com/apache/arrow-site/pull/127


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] nealrichardson commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

nealrichardson commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678584649



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,259 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+3 months of development work and includes **XX commits** from

Review comment:
       TODO: fill in these XXs




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] nealrichardson commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

nealrichardson commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678581870



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,229 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format)
+and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for
+the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has
+been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+`MakeArrayFromScalar` now works for fixed-size binary types (ARROW-13321).
+
+### Compute layer
+
+The following [compute functions]({{site.baseurl}}/docs/cpp/compute.html)
+were added:
+
+* aggregations: "index"
+
+* scalar arithmetic and math functions: "abs", "abs_checked", "acos",
+  "acos_checked", "asin", "asin_checked", "atan", "atan2", "ceil", "cos",
+  "cos_checked", "floor", "ln", "ln_checked", "log10", "log10_checked",
+  "log1p", "log1p_checked", "log2", "log2_checked", "negate", "negate_checked",
+  "sign", "sin", "sin_checked", "tan", "tan_checked", "trunc"
+
+* scalar bitwise functions: "bit_wise_and", "bit_wise_not", "bit_wise_or",
+  "bit_wise_xor", "shift_left", "shift_left_checked", "shift_right",
+  "shift_right_checked"
+
+* scalar string functions: "ascii_center", "ascii_lpad", "ascii_reverse",
+  "ascii_rpad", "binary_join", "binary_join_element_wise",
+  "binary_replace_slice", "count_substring", "count_substring_regex",
+  "ends_with", "find_substring", "find_substring_regex", "match_like",
+  "split_pattern_regex", "starts_with", "utf8_center", "utf8_lpad",
+  "utf8_replace_slice", "utf8_rpad", "utf8_reverse", "utf8_slice_codepoints"
+
+* scalar temporal functions: "day", "day_of_week", "day_of_year",
+  "iso_calendar", "iso_week", "iso_year", "hour", "microsecond", "millisecond",
+  "minute", "month", "nanosecond", "quarter", "second", "subsecond", "year"
+
+* other scalar functions: "case_when", "coalesce", "if_else", "is_finite",
+  "is_inf", "is_nan", "max_element_wise", "min_element_wise", "make_struct"
+
+* vector functions: "replace_with_mask"
+
+Duplicates are now allowed in `SetLookupOptions::value_set` (ARROW-12554).
+
+Decimal types are now supported by some basic arithmetic functions (ARROW-12074).
+
+The "take" function now supports dense unions (ARROW-13005).
+
+It is now possible to cast between dictionary types with different index
+types (ARROW-11673).
+
+Sorting is now implemented for boolean input (ARROW-12016).
+
+### CSV
+
+The streaming CSV reader can now read in parallel (ARROW-11889).
+
+The CSV reader tries to make its errors more informative by adding the
+row number when it is known, i.e. when parallel reading is disabled (ARROW-12675).
+
+A new option `ReaderOptions::skip_rows_after_names` allows skipping a number
+of rows _after_ reading the column names (as opposed to
+`ReaderOptions::skip_rows`).
+
+Quoted strings can now be treated as always non-null (ARROW-10115).
+
+### Dataset layer
+

Review comment:
       @westonpace please edit the suggestion to your satisfaction, then commit it




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] ianmcook commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

ianmcook commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678536922



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,100 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format) and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+## C# notes
+
+## Go notes
+
+## Java notes
+
+## JavaScript notes
+
+## Python notes
+
+## R notes
+
+In this release, we've more than doubled the number of functions you can call on Arrow Datasets inside `dplyr::filter()` and `mutate()`, including many more string, datetime, and math functions. You can also write Datasets to CSV files, in addition to Parquet and Feather. We've also deepened support for the Arrow C interface, which is used in the Python interface and allows integration with other projects, such as DuckDB.
+
+For more on what’s in the 5.0.0 R package, see the [R changelog][4].
+
+## Ruby and C GLib notes
+
+### Ruby
+
+### C GLib
+
+## Rust notes
+

Review comment:
       I don't think there is any way to know for sure what the URL of the Rust release blog post will be, and it might be posted after this one, so I'll just refer to it by name without a link for now.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] ianmcook commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

ianmcook commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r681888343



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,264 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+3 months of development work and includes **XX commits** from
+[**XX distinct contributors**][1] in 2 repositories. See the Install Page to

Review comment:
       Done in #136




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] ianmcook commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

ianmcook commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678389316



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,229 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format)
+and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for
+the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has
+been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+`MakeArrayFromScalar` now works for fixed-size binary types (ARROW-13321).
+
+### Compute layer
+
+The following [compute functions]({{site.baseurl}}/docs/cpp/compute.html)
+were added:
+
+* aggregations: "index"
+
+* scalar arithmetic and math functions: "abs", "abs_checked", "acos",
+  "acos_checked", "asin", "asin_checked", "atan", "atan2", "ceil", "cos",
+  "cos_checked", "floor", "ln", "ln_checked", "log10", "log10_checked",
+  "log1p", "log1p_checked", "log2", "log2_checked", "negate", "negate_checked",
+  "sign", "sin", "sin_checked", "tan", "tan_checked", "trunc"
+
+* scalar bitwise functions: "bit_wise_and", "bit_wise_not", "bit_wise_or",
+  "bit_wise_xor", "shift_left", "shift_left_checked", "shift_right",
+  "shift_right_checked"
+
+* scalar string functions: "ascii_center", "ascii_lpad", "ascii_reverse",
+  "ascii_rpad", "binary_join", "binary_join_element_wise",
+  "binary_replace_slice", "count_substring", "count_substring_regex",
+  "ends_with", "find_substring", "find_substring_regex", "match_like",
+  "split_pattern_regex", "starts_with", "utf8_center", "utf8_lpad",
+  "utf8_replace_slice", "utf8_rpad", "utf8_reverse", "utf8_slice_codepoints"
+
+* scalar temporal functions: "day", "day_of_week", "day_of_year",
+  "iso_calendar", "iso_week", "iso_year", "hour", "microsecond", "millisecond",
+  "minute", "month", "nanosecond", "quarter", "second", "subsecond", "year"
+
+* other scalar functions: "case_when", "coalesce", "if_else", "is_finite",
+  "is_inf", "is_nan", "max_element_wise", "min_element_wise", "make_struct"
+
+* vector functions: "replace_with_mask"
+
+Duplicates are now allowed in `SetLookupOptions::value_set` (ARROW-12554).
+
+Decimal types are now supported by some basic arithmetic functions (ARROW-12074).
+
+The "take" function now supports dense unions (ARROW-13005).
+
+It is now possible to cast between dictionary types with different index
+types (ARROW-11673).
+
+Sorting is now implemented for boolean input (ARROW-12016).
+
+### CSV
+
+The streaming CSV reader can now read in parallel (ARROW-11889).
+
+The CSV reader tries to make its errors more informative by adding the
+row number when it is known, i.e. when parallel reading is disabled (ARROW-12675).
+
+A new option `ReaderOptions::skip_rows_after_names` allows skipping a number
+of rows _after_ reading the column names (as opposed to
+`ReaderOptions::skip_rows`).
+
+Quoted strings can now be treated as always non-null (ARROW-10115).
+
+### Dataset layer
+
+### IO and Filesystem layer
+
+The I/O thread pool size can now be adjuted at runtime (ARROW-12760).
+The default size remains 8 threads.
+
+Streams now can have auxiliary metadata, depending on the backend.  This
+has been implemented for the S3 filesystems, where a couple metadata
+keys are supported such as "Content-Type" and "ACL" (ARROW-11161, ARROW-12719).
+
+The HadoopFileSystem implementation now implements the FileSystem abstraction
+more faithfully (ARROW-12790).
+
+### Parquet
+
+The new LZ4_RAW compression scheme was implemented (PARQUET-1998).
+Unlike the legacy LZ4 compression scheme, it is defined unambiguously
+and should provide better portability once other Parquet implementations
+catch up.
+
+## Go notes
+
+## Java notes
+
+## JavaScript notes
+
+* Tables do not extend DataFrames anymore. This enables smaller bundles. [#10277](https://github.com/apache/arrow/pull/10277)
+* Arrow uses closure compiler for all UMD bundles, making them smaller. [#10281](https://github.com/apache/arrow/pull/10281)
+* The npm package now comes with declaration maps for better navigation from types to source code. [#10673](https://github.com/apache/arrow/pull/10673)
+* Updated dependencies and improvements to the code.
+
+## Python notes
+
+* Datasets can now scan files asynchronously when the `use_async=True` option is provided to `Dataset.scanner`, `Dataset.to_table`, or `Dataset.to_batches` methods. This should provide better performance in environments where I/O can be slow, such as with remote sources.
+* Arrow now provides builtin support for writing CSV files through `pyarrow.csv.write_csv`
+* Wheels for Apple M1 Macs are now provided.
+* Many new `pyarrow.compute` functions are available, and introspection of the functions was improved so that they look more like standard Python functions.
+* It is now possible to access ORC file metadata from Python `ORCFile` objects
+* Building a `StructArray` now accepts a `mask` like other arrays
+* Many updates and fixes for the documentation
+
+## R notes
+
+In this release, we've more than doubled the number of functions you can call on Arrow Datasets inside `dplyr::filter()` and `mutate()`, including many more string, datetime, and math functions. You can also write Datasets to CSV files, in addition to Parquet and Feather. We've also deepened support for the Arrow C interface, which is used in the Python interface and allows integration with other projects, such as DuckDB.
+
+For more on what’s in the 5.0.0 R package, see the [R changelog][4].
+
+## Ruby and C GLib notes
+
+Apache Arrow Flight support is started. But `ListFlights` is only supported for now. More features will be implemented in the next major release.
+
+### Ruby
+
+You need gobject-introspection gem 3.4.5 or later to implement your Apache Arrow Flight server. If you only use Apache Arrow Flight client, gobject-introspection gem 3.4.5 or later isn't required.
+
+Here are highlighted improvements:
+
+* Compute functions accept raw Ruby objects such as `true`, `Integer`, `Array` and `String`:
+
+  ```ruby
+  add_function = Arrow::Function.find("add")
+  # Not shortcut version
+  augend = Arrow::Int8Array.new([1, 2, 3])
+  addend = Arrow::Int8Scalar.new(5)
+  args = [
+    Arrow::ArrayDatum.new(augend),
+    Arrow::ScalarDatum.new(addend),
+  ]
+  add_function.execute(args).value.to_a # => [6, 7, 8]
+  # Shortcut version
+  add_function.execute([[1, 2, 3], 5]).value.to_a # => [6, 7, 8]
+  ```
+* `Arrow::PrimaryArray` and `Arrow::Buffer` can be used as MemoryView that is added in Ruby 3.0.
+
+There are some backward incompatible changes:
+
+* `Arrow::CountOptions` and `Arrow::CountMode` are removed. Use `Arrow::ScalarAggregateOptions` instead.
+
+### C GLib
+
+There are some backward incompatible changes:
+
+* `GArrowCountOptions` and `GArrowCountMode` are removed. Use `GArrowScalarAggregateOptions` instead.
+* `garrow_array_equal_range()` requires `GArrowEqualOptions`.
+* Prefix in arrow-dataset-glib is changed to `gadataset_`/`GADATASET_` from `gad_`/`GAD_`.
+* `GADScanOptions`, `GADScanTask`  and `GADInMemoryScanTask` are removed. Use `gadataset_begin_scan()` or `gadataset_to_table()` instead.
+* `GArrowCompareOptions`, `GArrowCompareOperator` and `garrow_*_array_compare()` are removed. Use `equal`, `not_equal`, `less_than`, `less_than_equal`, `greater_than` and `greater_than_equal` compute functions directly instead.
+
+## Rust notes
+
+
+[1]: https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%205.0.0
+[2]: {{ site.baseurl }}/release/5.0.0.html#contributors
+[3]: {{ site.baseurl }}/release/5.0.0.html

Review comment:
       ```suggestion
   [1]: {{ site.baseurl }}/release/5.0.0.html#contributors
   [2]: {{ site.baseurl }}/release/5.0.0.html#changelog
   [3]: https://github.com/apache/arrow-rs/blob/5.0.0/CHANGELOG.md
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] zeroshade commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

zeroshade commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678359411



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,113 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format) and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+## C# notes
+
+## Go notes

Review comment:
       I don't have permission to add the commit to this PR so I'll just put my notes here, let me know if there's any conventions that I missed or need to edit/modify for:
   
   ```
   * Flight Client and Server now support Custom Metadata through the functions `flight.NewClientWithMiddleware` and `flight.NewServerWithMiddleware`. Functions
   `flight.NewFlightClient`, `flight.NewFlightServer`, `flight.CreateServerBearerTokenAuthInterceptors` have been deprecated in favor of using the new middleware. [#10633](https://github.com/apache/arrow/pull/10633)
   * Fixed build failure for Parquet implementation on s390x [#10475](https://github.com/apache/arrow/pull/10475)
   * Flight Client `AuthHandler` no longer overwrites outgoing metadata, correctly appending new metadata without overwriting existing metadata [#10297](https://github.com/apache/arrow/pull/10297)
   * Flight AppMetadata field is now exposed both for Reading and Writing via `flight.Reader#LatestAppMetadata()` and `flight.Writer#WriteWithAppMetadata` functions [#10142](https://github.com/apache/arrow/pull/10142)
   * Extension Datatype is now implemented for Arrow Arrays [#10203](https://github.com/apache/arrow/pull/10203)
   * Map Datatype is now implemented for Arrow Arrays [#10106](https://github.com/apache/arrow/pull/10106)
   * Decimal integration tests with Go as both producer or consumer have been fixed [#10116](https://github.com/apache/arrow/10116)
   * Schema Package added for Golang Parquet Implementation [#10071](https://github.com/apache/arrow/10071)
   * First part of Encoding package for Golang Parquet Implementation added [#10379](https://github.com/apache/arrow/10379)
   ```
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] BryanCutler commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

BryanCutler commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678522591



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,229 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format)
+and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for
+the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has
+been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+`MakeArrayFromScalar` now works for fixed-size binary types (ARROW-13321).
+
+### Compute layer
+
+The following [compute functions]({{site.baseurl}}/docs/cpp/compute.html)
+were added:
+
+* aggregations: "index"
+
+* scalar arithmetic and math functions: "abs", "abs_checked", "acos",
+  "acos_checked", "asin", "asin_checked", "atan", "atan2", "ceil", "cos",
+  "cos_checked", "floor", "ln", "ln_checked", "log10", "log10_checked",
+  "log1p", "log1p_checked", "log2", "log2_checked", "negate", "negate_checked",
+  "sign", "sin", "sin_checked", "tan", "tan_checked", "trunc"
+
+* scalar bitwise functions: "bit_wise_and", "bit_wise_not", "bit_wise_or",
+  "bit_wise_xor", "shift_left", "shift_left_checked", "shift_right",
+  "shift_right_checked"
+
+* scalar string functions: "ascii_center", "ascii_lpad", "ascii_reverse",
+  "ascii_rpad", "binary_join", "binary_join_element_wise",
+  "binary_replace_slice", "count_substring", "count_substring_regex",
+  "ends_with", "find_substring", "find_substring_regex", "match_like",
+  "split_pattern_regex", "starts_with", "utf8_center", "utf8_lpad",
+  "utf8_replace_slice", "utf8_rpad", "utf8_reverse", "utf8_slice_codepoints"
+
+* scalar temporal functions: "day", "day_of_week", "day_of_year",
+  "iso_calendar", "iso_week", "iso_year", "hour", "microsecond", "millisecond",
+  "minute", "month", "nanosecond", "quarter", "second", "subsecond", "year"
+
+* other scalar functions: "case_when", "coalesce", "if_else", "is_finite",
+  "is_inf", "is_nan", "max_element_wise", "min_element_wise", "make_struct"
+
+* vector functions: "replace_with_mask"
+
+Duplicates are now allowed in `SetLookupOptions::value_set` (ARROW-12554).
+
+Decimal types are now supported by some basic arithmetic functions (ARROW-12074).
+
+The "take" function now supports dense unions (ARROW-13005).
+
+It is now possible to cast between dictionary types with different index
+types (ARROW-11673).
+
+Sorting is now implemented for boolean input (ARROW-12016).
+
+### CSV
+
+The streaming CSV reader can now read in parallel (ARROW-11889).
+
+The CSV reader tries to make its errors more informative by adding the
+row number when it is known, i.e. when parallel reading is disabled (ARROW-12675).
+
+A new option `ReaderOptions::skip_rows_after_names` allows skipping a number
+of rows _after_ reading the column names (as opposed to
+`ReaderOptions::skip_rows`).
+
+Quoted strings can now be treated as always non-null (ARROW-10115).
+
+### Dataset layer
+
+### IO and Filesystem layer
+
+The I/O thread pool size can now be adjuted at runtime (ARROW-12760).
+The default size remains 8 threads.
+
+Streams now can have auxiliary metadata, depending on the backend.  This
+has been implemented for the S3 filesystems, where a couple metadata
+keys are supported such as "Content-Type" and "ACL" (ARROW-11161, ARROW-12719).
+
+The HadoopFileSystem implementation now implements the FileSystem abstraction
+more faithfully (ARROW-12790).
+
+### Parquet
+
+The new LZ4_RAW compression scheme was implemented (PARQUET-1998).
+Unlike the legacy LZ4 compression scheme, it is defined unambiguously
+and should provide better portability once other Parquet implementations
+catch up.
+
+## Go notes
+
+## Java notes

Review comment:
       ```suggestion
   ## Java notes
   
   Highlighted improvements and fixes:
   
   * Improved support for extension types using a complex storage type, e.g. struct, map or union. These can now extend
   the `ExtensionTypeVector` base class.
   * Union vectors now extend `AbstractContainerVector` to be consistent with other vectors.
   * Guava dependency updated to 30.1.1
   * Memory leak fixed if an exception occurs when reading IPC messages from a channel. [#10423](https://github.com/apache/arrow/pull/10423)
   * Flight error metadata is now propagated to the client. [#10370](https://github.com/apache/arrow/pull/10370)
   * JDBC adapter now preserves nullability. [#10285](https://github.com/apache/arrow/pull/10285)
   
   API compatibility changes:
   
   * Complex vectors now return covariant types from `getObject(int)`. [#9964](https://github.com/apache/arrow/pull/9964)
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] liyafan82 commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

liyafan82 commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678780589



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,264 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+3 months of development work and includes **XX commits** from
+[**XX distinct contributors**][1] in 2 repositories. See the Install Page to
+learn how to get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the complete changelogs for the [`apache/arrow`][2] and
+[`apache/arrow-rs`][3] repositories.
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format)
+and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend `.arrow` as the IPC file format file extension and `.arrows` for
+the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has
+been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+`MakeArrayFromScalar` now works for fixed-size binary types (ARROW-13321).
+
+### Compute layer
+
+The following [compute functions]({{site.baseurl}}/docs/cpp/compute.html)
+were added:
+
+* aggregations: `index`
+
+* scalar arithmetic and math functions: `abs`, `abs_checked`, `acos`,
+  `acos_checked`, `asin`, `asin_checked`, `atan`, `atan2`, `ceil`, `cos`,
+  `cos_checked`, `floor`, `ln`, `ln_checked`, `log10`, `log10_checked`,
+  `log1p`, `log1p_checked`, `log2`, `log2_checked`, `negate`, `negate_checked`,
+  `sign`, `sin`, `sin_checked`, `tan`, `tan_checked`, `trunc`
+
+* scalar bitwise functions: `bit_wise_and`, `bit_wise_not`, `bit_wise_or`,
+  `bit_wise_xor`, `shift_left`, `shift_left_checked`, `shift_right`,
+  `shift_right_checked`
+
+* scalar string functions: `ascii_center`, `ascii_lpad`, `ascii_reverse`,
+  `ascii_rpad`, `binary_join`, `binary_join_element_wise`,
+  `binary_replace_slice`, `count_substring`, `count_substring_regex`,
+  `ends_with`, `find_substring`, `find_substring_regex`, `match_like`,
+  `split_pattern_regex`, `starts_with`, `utf8_center`, `utf8_lpad`,
+  `utf8_replace_slice`, `utf8_rpad`, `utf8_reverse`, `utf8_slice_codepoints`
+
+* scalar temporal functions: `day`, `day_of_week`, `day_of_year`,
+  `iso_calendar`, `iso_week`, `iso_year`, `hour`, `microsecond`, `millisecond`,
+  `minute`, `month`, `nanosecond`, `quarter`, `second`, `subsecond`, `year`
+
+* other scalar functions: `case_when`, `coalesce`, `if_else`, `is_finite`,
+  `is_inf`, `is_nan`, `max_element_wise`, `min_element_wise`, `make_struct`
+
+* vector functions: `replace_with_mask`
+
+Duplicates are now allowed in `SetLookupOptions::value_set` (ARROW-12554).
+
+Decimal types are now supported by some basic arithmetic functions (ARROW-12074).
+
+The `take` function now supports dense unions (ARROW-13005).
+
+It is now possible to cast between dictionary types with different index
+types (ARROW-11673).
+
+Sorting is now implemented for boolean input (ARROW-12016).
+
+### CSV
+
+The streaming CSV reader can now take some advantage of multiple threads (ARROW-11889).
+
+The CSV reader tries to make its errors more informative by adding the
+row number when it is known, i.e. when parallel reading is disabled (ARROW-12675).
+
+A new option `ReaderOptions::skip_rows_after_names` allows skipping a number
+of rows _after_ reading the column names (as opposed to
+`ReaderOptions::skip_rows`).
+
+Quoted strings can now be treated as always non-null (ARROW-10115).
+
+### Dataset layer
+
+The asynchronous scanner introduced in 4.0.0 has been improved with truly 
+asynchronous readers implemented for CSV, Parquet, and IPC file formats and 
+file-level parallelism added.  This mode is controlled by a flag `use_async` that
+can be passed into methods which scan a dataset.  Setting this flag to True
+will have significant improvements on filesystems with high latency or parallel
+reads (e.g. S3).
+
+A CountRows method has been added to count rows matching a predicate; where
+possible, this will use metadata in files instead of reading the data itself.
+
+CSV datasets can now be written, and when reading a CSV dataset, explicit types can
+now be specified for a subset of columns while allowing the rest to still be inferred. 
+
+### IO and Filesystem layer
+
+The I/O thread pool size can now be adjuted at runtime (ARROW-12760).
+The default size remains 8 threads.
+
+Streams now can have auxiliary metadata, depending on the backend.  This
+has been implemented for the S3 filesystems, where a couple metadata
+keys are supported such as `Content-Type` and `ACL` (ARROW-11161, ARROW-12719).
+
+The HadoopFileSystem implementation now implements the FileSystem abstraction
+more faithfully (ARROW-12790).
+
+### Parquet
+
+The new LZ4_RAW compression scheme was implemented (PARQUET-1998).
+Unlike the legacy LZ4 compression scheme, it is defined unambiguously
+and should provide better portability once other Parquet implementations
+catch up.
+
+## Go notes
+
+
+### Flight 
+
+* Flight Client and Server now support Custom Metadata through the functions `flight.NewClientWithMiddleware` and `flight.NewServerWithMiddleware`. Functions
+`flight.NewFlightClient`, `flight.NewFlightServer`, `flight.CreateServerBearerTokenAuthInterceptors` have been deprecated in favor of using the new middleware. [#10633](https://github.com/apache/arrow/pull/10633)
+* Flight Client `AuthHandler` no longer overwrites outgoing metadata, correctly appending new metadata without overwriting existing metadata [#10297](https://github.com/apache/arrow/pull/10297)
+* Flight AppMetadata field is now exposed both for Reading and Writing via `flight.Reader#LatestAppMetadata()` and `flight.Writer#WriteWithAppMetadata` functions [#10142](https://github.com/apache/arrow/pull/10142)
+
+### Other enhancements 
+
+* [Map](https://github.com/apache/arrow/pull/10106) and [Extension](https://github.com/apache/arrow/pull/10203) Datatypes are now implemented for Arrow Arrays 
+* [Schema package](https://github.com/apache/arrow/10071) and first part of [Encoding package](https://github.com/apache/arrow/10379) added for Golang Parquet Implementation
+
+## Java notes
+
+Highlighted improvements and fixes:
+
+* Improved support for extension types using a complex storage type, e.g. struct, map or union. These can now extend
+the `ExtensionTypeVector` base class.
+* Union vectors now extend `AbstractContainerVector` to be consistent with other vectors.
+* Guava dependency updated to 30.1.1
+* Memory leak fixed if an exception occurs when reading IPC messages from a channel. [#10423](https://github.com/apache/arrow/pull/10423)
+* Flight error metadata is now propagated to the client. [#10370](https://github.com/apache/arrow/pull/10370)
+* JDBC adapter now preserves nullability. [#10285](https://github.com/apache/arrow/pull/10285)
+

Review comment:
       ```suggestion
   * The memory rounding policy is respected when allocating vector buffers. This helps saving memory space. [#13147] (https://github.com/apache/arrow/pull/10576)
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] zeroshade commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

zeroshade commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678359411



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,113 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format) and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+## C# notes
+
+## Go notes

Review comment:
       I don't have permission to add the commit to this PR so I'll just put my notes here, let me know if there's any conventions that I missed or need to edit/modify for:
   
   
   ```suggestion
   ## Go notes
   
   * Flight Client and Server now support Custom Metadata through the functions `flight.NewClientWithMiddleware` and `flight.NewServerWithMiddleware`. Functions
   `flight.NewFlightClient`, `flight.NewFlightServer`, `flight.CreateServerBearerTokenAuthInterceptors` have been deprecated in favor of using the new middleware. [#10633](https://github.com/apache/arrow/pull/10633)
   * Fixed build failure for Parquet implementation on s390x [#10475](https://github.com/apache/arrow/pull/10475)
   * Flight Client `AuthHandler` no longer overwrites outgoing metadata, correctly appending new metadata without overwriting existing metadata [#10297](https://github.com/apache/arrow/pull/10297)
   * Flight AppMetadata field is now exposed both for Reading and Writing via `flight.Reader#LatestAppMetadata()` and `flight.Writer#WriteWithAppMetadata` functions [#10142](https://github.com/apache/arrow/pull/10142)
   * Extension Datatype is now implemented for Arrow Arrays [#10203](https://github.com/apache/arrow/pull/10203)
   * Map Datatype is now implemented for Arrow Arrays [#10106](https://github.com/apache/arrow/pull/10106)
   * Decimal integration tests with Go as both producer or consumer have been fixed [#10116](https://github.com/apache/arrow/10116)
   * Schema Package added for Golang Parquet Implementation [#10071](https://github.com/apache/arrow/10071)
   * First part of Encoding package for Golang Parquet Implementation added [#10379](https://github.com/apache/arrow/10379)
   ```
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] ianmcook commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

ianmcook commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r677466710



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,100 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format) and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+## C# notes
+
+## Go notes
+
+## Java notes
+
+## JavaScript notes
+
+## Python notes
+
+## R notes
+
+In this release, we've more than doubled the number of functions you can call on Arrow Datasets inside `dplyr::filter()` and `mutate()`, including many more string, datetime, and math functions. You can also write Datasets to CSV files, in addition to Parquet and Feather. We've also deepened support for the Arrow C interface, which is used in the Python interface and allows integration with other projects, such as DuckDB.
+
+For more on what’s in the 5.0.0 R package, see the [R changelog][4].
+
+## Ruby and C GLib notes
+
+### Ruby
+
+### C GLib
+
+## Rust notes
+

Review comment:
       Link here to the Rust release blog post (in the works at https://github.com/apache/arrow-site/pull/128) and changelog (https://github.com/apache/arrow-rs/blob/5.0.0/CHANGELOG.md)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] kou commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

kou commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r677812946



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,113 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format) and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+## C# notes
+
+## Go notes
+
+## Java notes
+
+## JavaScript notes
+
+* Tables do not extend DataFrames anymore. This enables smaller bundles. [#10277](https://github.com/apache/arrow/pull/10277)
+* Arrow uses closure compiler for all UMD bundles, making them smaller. [#10281](https://github.com/apache/arrow/pull/10281)
+* The npm package now comes with declaration maps for better navigation from types to source code. [#10673](https://github.com/apache/arrow/pull/10673)
+* Updated dependencies and improvements to the code.
+
+## Python notes
+
+* Datasets can now scan files asynchronously when the `use_async=True` option is provided to `Dataset.scanner`, `Dataset.to_table`, or `Dataset.to_batches` methods. This should provide better performance in environments where I/O can be slow, such as with remote sources.
+* Arrow now provides builtin support for writing CSV files through `pyarrow.csv.write_csv`
+* Wheels for Apple M1 Macs are now provided.
+* Many new `pyarrow.compute` functions are available, and introspection of the functions was improved so that they look more like standard Python functions.
+* It is now possible to access ORC file metadata from Python `ORCFile` objects
+* Building a `StructArray` now accepts a `mask` like other arrays
+* Many updates and fixes for the documentation
+
+## R notes
+
+In this release, we've more than doubled the number of functions you can call on Arrow Datasets inside `dplyr::filter()` and `mutate()`, including many more string, datetime, and math functions. You can also write Datasets to CSV files, in addition to Parquet and Feather. We've also deepened support for the Arrow C interface, which is used in the Python interface and allows integration with other projects, such as DuckDB.
+
+For more on what’s in the 5.0.0 R package, see the [R changelog][4].
+
+## Ruby and C GLib notes

Review comment:
       Sorry for not working on this...
   I've added notes for Ruby and C GLib.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] alamb commented on pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

alamb commented on pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#issuecomment-882851381


   Here is proposed 5.0 content for Rust: https://github.com/apache/arrow-site/pull/128
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] ianmcook commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

ianmcook commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r682064224



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,267 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-29 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+3 months of development work and includes **684 commits** from
+[**99 distinct contributors**][1] in 2 repositories. See the Install Page to
+learn how to get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the complete changelogs for the [`apache/arrow`][2] and
+[`apache/arrow-rs`][3] repositories.
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format)
+and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend `.arrow` as the IPC file format file extension and `.arrows` for
+the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has
+been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+`MakeArrayFromScalar` now works for fixed-size binary types (ARROW-13321).
+
+### Compute layer
+
+The following [compute functions]({{site.baseurl}}/docs/cpp/compute.html)
+were added:
+
+* aggregations: `index`
+
+* scalar arithmetic and math functions: `abs`, `abs_checked`, `acos`,
+  `acos_checked`, `asin`, `asin_checked`, `atan`, `atan2`, `ceil`, `cos`,
+  `cos_checked`, `floor`, `ln`, `ln_checked`, `log10`, `log10_checked`,
+  `log1p`, `log1p_checked`, `log2`, `log2_checked`, `negate`, `negate_checked`,
+  `sign`, `sin`, `sin_checked`, `tan`, `tan_checked`, `trunc`
+
+* scalar bitwise functions: `bit_wise_and`, `bit_wise_not`, `bit_wise_or`,
+  `bit_wise_xor`, `shift_left`, `shift_left_checked`, `shift_right`,
+  `shift_right_checked`
+
+* scalar string functions: `ascii_center`, `ascii_lpad`, `ascii_reverse`,
+  `ascii_rpad`, `binary_join`, `binary_join_element_wise`,
+  `binary_replace_slice`, `count_substring`, `count_substring_regex`,
+  `ends_with`, `find_substring`, `find_substring_regex`, `match_like`,
+  `split_pattern_regex`, `starts_with`, `utf8_center`, `utf8_lpad`,
+  `utf8_replace_slice`, `utf8_rpad`, `utf8_reverse`, `utf8_slice_codepoints`
+
+* scalar temporal functions: `day`, `day_of_week`, `day_of_year`,
+  `iso_calendar`, `iso_week`, `iso_year`, `hour`, `microsecond`, `millisecond`,
+  `minute`, `month`, `nanosecond`, `quarter`, `second`, `subsecond`, `year`
+
+* other scalar functions: `case_when`, `coalesce`, `if_else`, `is_finite`,
+  `is_inf`, `is_nan`, `max_element_wise`, `min_element_wise`, `make_struct`
+
+* vector functions: `replace_with_mask`
+
+Duplicates are now allowed in `SetLookupOptions::value_set` (ARROW-12554).
+
+Decimal types are now supported by some basic arithmetic functions (ARROW-12074).
+
+The `take` function now supports dense unions (ARROW-13005).
+
+It is now possible to cast between dictionary types with different index
+types (ARROW-11673).
+
+Sorting is now implemented for boolean input (ARROW-12016).
+
+### CSV
+
+The streaming CSV reader can now take some advantage of multiple threads (ARROW-11889).
+
+The CSV reader tries to make its errors more informative by adding the
+row number when it is known, i.e. when parallel reading is disabled (ARROW-12675).
+
+A new option `ReaderOptions::skip_rows_after_names` allows skipping a number
+of rows _after_ reading the column names (as opposed to
+`ReaderOptions::skip_rows`).
+
+Quoted strings can now be treated as always non-null (ARROW-10115).
+
+### Dataset layer
+
+The asynchronous scanner introduced in 4.0.0 has been improved with truly 
+asynchronous readers implemented for CSV, Parquet, and IPC file formats and 
+file-level parallelism added.  This mode is controlled by a flag `use_async` that
+can be passed into methods which scan a dataset.  Setting this flag to True
+will have significant improvements on filesystems with high latency or parallel
+reads (e.g. S3).
+
+A CountRows method has been added to count rows matching a predicate; where
+possible, this will use metadata in files instead of reading the data itself.
+
+CSV datasets can now be written, and when reading a CSV dataset, explicit types can
+now be specified for a subset of columns while allowing the rest to still be inferred. 
+
+### IO and Filesystem layer
+
+The I/O thread pool size can now be adjusted at runtime (ARROW-12760).
+The default size remains 8 threads.
+
+Streams now can have auxiliary metadata, depending on the backend.  This
+has been implemented for the S3 filesystems, where a couple metadata
+keys are supported such as `Content-Type` and `ACL` (ARROW-11161, ARROW-12719).
+
+The HadoopFileSystem implementation now implements the FileSystem abstraction
+more faithfully (ARROW-12790).
+
+### Parquet
+
+The new LZ4_RAW compression scheme was implemented (PARQUET-1998).
+Unlike the legacy LZ4 compression scheme, it is defined unambiguously
+and should provide better portability once other Parquet implementations
+catch up.
+
+## Go notes
+
+
+### Flight 
+
+* Flight Client and Server now support Custom Metadata through the functions `flight.NewClientWithMiddleware` and `flight.NewServerWithMiddleware`. Functions
+`flight.NewFlightClient`, `flight.NewFlightServer`, `flight.CreateServerBearerTokenAuthInterceptors` have been deprecated in favor of using the new middleware. [#10633](https://github.com/apache/arrow/pull/10633)
+* Flight Client `AuthHandler` no longer overwrites outgoing metadata, correctly appending new metadata without overwriting existing metadata [#10297](https://github.com/apache/arrow/pull/10297)
+* Flight AppMetadata field is now exposed both for Reading and Writing via `flight.Reader#LatestAppMetadata()` and `flight.Writer#WriteWithAppMetadata` functions [#10142](https://github.com/apache/arrow/pull/10142)
+
+### Other enhancements 
+
+* [Map](https://github.com/apache/arrow/pull/10106) and [Extension](https://github.com/apache/arrow/pull/10203) Datatypes are now implemented for Arrow Arrays 
+* [Schema package](https://github.com/apache/arrow/10071) and first part of [Encoding package](https://github.com/apache/arrow/10379) added for Golang Parquet Implementation
+
+## Java notes
+
+Highlighted improvements and fixes:
+
+* Improved support for extension types using a complex storage type, e.g. struct, map or union. These can now extend
+the `ExtensionTypeVector` base class.
+* Union vectors now extend `AbstractContainerVector` to be consistent with other vectors.
+* Guava dependency updated to 30.1.1
+* Memory leak fixed if an exception occurs when reading IPC messages from a channel. [#10423](https://github.com/apache/arrow/pull/10423)
+* Flight error metadata is now propagated to the client. [#10370](https://github.com/apache/arrow/pull/10370)
+* JDBC adapter now preserves nullability. [#10285](https://github.com/apache/arrow/pull/10285)
+* The memory rounding policy is respected when allocating vector buffers. This helps saving memory space. [#13147](https://github.com/apache/arrow/pull/10576)

Review comment:
       ```suggestion
   * The memory rounding policy is respected when allocating vector buffers. This helps saving memory space. [#10576](https://github.com/apache/arrow/pull/10576)
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] domoritz commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

domoritz commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r677056470



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,100 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format) and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+## C# notes
+
+## Go notes
+
+## Java notes
+
+## JavaScript notes

Review comment:
       ```suggestion
   ## JavaScript notes
   
   * Tables do not extend DataFrames anymore. This enables smaller bundles. [#10277](https://github.com/apache/arrow/pull/10277)
   * Arrow uses closure compiler for all UMD bundles, making them smaller. [#10281](https://github.com/apache/arrow/pull/10281)
   * The npm package now comes with declaration maps for better navigation from types to source code. [#10673](https://github.com/apache/arrow/pull/10673)
   * Updated dependencies and improvements to the code.
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] nealrichardson commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

nealrichardson commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r677731998



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,113 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format) and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes

Review comment:
       @pitrou would you mind helping to fill the C++ section out?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] nealrichardson commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

nealrichardson commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r677732705



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,113 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format) and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+## C# notes

Review comment:
       ```suggestion
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] ianmcook commented on pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

ianmcook commented on pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#issuecomment-891967704


   If anyone has additional changes or additions, please suggest them as soon as possible so we can get this posted.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] nealrichardson commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

nealrichardson commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r679202485



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,264 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"

Review comment:
       ```suggestion
   date: "2020-07-29 00:00:00 -0600"
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] lidavidm commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

lidavidm commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r673979899



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,96 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format) and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+

Review comment:
       ```suggestion
   
   The Go implementation now supports custom metadata and middleware, and has been added to integration testing.
   
   In Python, some operations can now be interrupted via Control-C.
   
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] nealrichardson commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

nealrichardson commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r679206385



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,259 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+3 months of development work and includes **XX commits** from
+[**XX distinct contributors**][1] in 2 repositories. See the Install Page to
+learn how to get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the complete changelogs for the [`apache/arrow`][2] and
+[`apache/arrow-rs`][3] repositories.
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format)
+and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend `.arrow` as the IPC file format file extension and `.arrows` for
+the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has
+been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+`MakeArrayFromScalar` now works for fixed-size binary types (ARROW-13321).
+
+### Compute layer
+
+The following [compute functions]({{site.baseurl}}/docs/cpp/compute.html)
+were added:
+
+* aggregations: `index`
+
+* scalar arithmetic and math functions: `abs`, `abs_checked`, `acos`,
+  `acos_checked`, `asin`, `asin_checked`, `atan`, `atan2`, `ceil`, `cos`,
+  `cos_checked`, `floor`, `ln`, `ln_checked`, `log10`, `log10_checked`,
+  `log1p`, `log1p_checked`, `log2`, `log2_checked`, `negate`, `negate_checked`,
+  `sign`, `sin`, `sin_checked`, `tan`, `tan_checked`, `trunc`
+
+* scalar bitwise functions: `bit_wise_and`, `bit_wise_not`, `bit_wise_or`,
+  `bit_wise_xor`, `shift_left`, `shift_left_checked`, `shift_right`,
+  `shift_right_checked`
+
+* scalar string functions: `ascii_center`, `ascii_lpad`, `ascii_reverse`,
+  `ascii_rpad`, `binary_join`, `binary_join_element_wise`,
+  `binary_replace_slice`, `count_substring`, `count_substring_regex`,
+  `ends_with`, `find_substring`, `find_substring_regex`, `match_like`,
+  `split_pattern_regex`, `starts_with`, `utf8_center`, `utf8_lpad`,
+  `utf8_replace_slice`, `utf8_rpad`, `utf8_reverse`, `utf8_slice_codepoints`
+
+* scalar temporal functions: `day`, `day_of_week`, `day_of_year`,
+  `iso_calendar`, `iso_week`, `iso_year`, `hour`, `microsecond`, `millisecond`,
+  `minute`, `month`, `nanosecond`, `quarter`, `second`, `subsecond`, `year`
+
+* other scalar functions: `case_when`, `coalesce`, `if_else`, `is_finite`,
+  `is_inf`, `is_nan`, `max_element_wise`, `min_element_wise`, `make_struct`
+
+* vector functions: `replace_with_mask`
+
+Duplicates are now allowed in `SetLookupOptions::value_set` (ARROW-12554).
+
+Decimal types are now supported by some basic arithmetic functions (ARROW-12074).
+
+The `take` function now supports dense unions (ARROW-13005).
+
+It is now possible to cast between dictionary types with different index
+types (ARROW-11673).
+
+Sorting is now implemented for boolean input (ARROW-12016).
+
+### CSV
+
+The streaming CSV reader can now take some advantage of multiple threads (ARROW-11889).
+
+The CSV reader tries to make its errors more informative by adding the
+row number when it is known, i.e. when parallel reading is disabled (ARROW-12675).
+
+A new option `ReaderOptions::skip_rows_after_names` allows skipping a number
+of rows _after_ reading the column names (as opposed to
+`ReaderOptions::skip_rows`).
+
+Quoted strings can now be treated as always non-null (ARROW-10115).
+
+### Dataset layer
+
+### IO and Filesystem layer
+
+The I/O thread pool size can now be adjuted at runtime (ARROW-12760).
+The default size remains 8 threads.
+
+Streams now can have auxiliary metadata, depending on the backend.  This
+has been implemented for the S3 filesystems, where a couple metadata
+keys are supported such as `Content-Type` and `ACL` (ARROW-11161, ARROW-12719).
+
+The HadoopFileSystem implementation now implements the FileSystem abstraction
+more faithfully (ARROW-12790).
+
+### Parquet
+
+The new LZ4_RAW compression scheme was implemented (PARQUET-1998).
+Unlike the legacy LZ4 compression scheme, it is defined unambiguously
+and should provide better portability once other Parquet implementations
+catch up.
+
+## Go notes
+
+* Flight Client and Server now support Custom Metadata through the functions `flight.NewClientWithMiddleware` and `flight.NewServerWithMiddleware`. Functions
+`flight.NewFlightClient`, `flight.NewFlightServer`, `flight.CreateServerBearerTokenAuthInterceptors` have been deprecated in favor of using the new middleware. [#10633](https://github.com/apache/arrow/pull/10633)
+* Fixed build failure for Parquet implementation on s390x [#10475](https://github.com/apache/arrow/pull/10475)
+* Flight Client `AuthHandler` no longer overwrites outgoing metadata, correctly appending new metadata without overwriting existing metadata [#10297](https://github.com/apache/arrow/pull/10297)
+* Flight AppMetadata field is now exposed both for Reading and Writing via `flight.Reader#LatestAppMetadata()` and `flight.Writer#WriteWithAppMetadata` functions [#10142](https://github.com/apache/arrow/pull/10142)
+* Extension Datatype is now implemented for Arrow Arrays [#10203](https://github.com/apache/arrow/pull/10203)
+* Map Datatype is now implemented for Arrow Arrays [#10106](https://github.com/apache/arrow/pull/10106)
+* Decimal integration tests with Go as both producer or consumer have been fixed [#10116](https://github.com/apache/arrow/10116)
+* Schema Package added for Golang Parquet Implementation [#10071](https://github.com/apache/arrow/10071)
+* First part of Encoding package for Golang Parquet Implementation added [#10379](https://github.com/apache/arrow/10379)
+
+## Java notes
+
+Highlighted improvements and fixes:
+
+* Improved support for extension types using a complex storage type, e.g. struct, map or union. These can now extend
+the `ExtensionTypeVector` base class.
+* Union vectors now extend `AbstractContainerVector` to be consistent with other vectors.
+* Guava dependency updated to 30.1.1
+* Memory leak fixed if an exception occurs when reading IPC messages from a channel. [#10423](https://github.com/apache/arrow/pull/10423)
+* Flight error metadata is now propagated to the client. [#10370](https://github.com/apache/arrow/pull/10370)
+* JDBC adapter now preserves nullability. [#10285](https://github.com/apache/arrow/pull/10285)
+
+API compatibility changes:
+
+* Complex vectors now return covariant types from `getObject(int)`. [#9964](https://github.com/apache/arrow/pull/9964)
+
+## JavaScript notes
+
+* Tables do not extend DataFrames anymore. This enables smaller bundles. [#10277](https://github.com/apache/arrow/pull/10277)
+* Arrow uses closure compiler for all UMD bundles, making them smaller. [#10281](https://github.com/apache/arrow/pull/10281)
+* The npm package now comes with declaration maps for better navigation from types to source code. [#10673](https://github.com/apache/arrow/pull/10673)
+* Updated dependencies and improvements to the code.
+
+## Python notes
+
+* Datasets can now scan files asynchronously when the `use_async=True` option is provided to `Dataset.scanner`, `Dataset.to_table`, or `Dataset.to_batches` methods. This should provide better performance in environments where I/O can be slow, such as with remote sources.
+* Arrow now provides builtin support for writing CSV files through `pyarrow.csv.write_csv`
+* Wheels for Apple M1 Macs are now provided.
+* Many new `pyarrow.compute` functions are available, and introspection of the functions was improved so that they look more like standard Python functions.
+* It is now possible to access ORC file metadata from Python `ORCFile` objects
+* Building a `StructArray` now accepts a `mask` like other arrays
+* Many updates and fixes for the documentation
+
+## R notes
+
+In this release, we've more than doubled the number of functions you can call on Arrow Datasets inside `dplyr::filter()` and `mutate()`, including many more string, datetime, and math functions. You can also write Datasets to CSV files, in addition to Parquet and Feather. We've also deepened support for the Arrow C interface, which is used in the Python interface and allows integration with other projects, such as DuckDB.
+
+For more on what’s in the 5.0.0 R package, see the [R changelog][4].
+
+## Ruby and C GLib notes
+
+Apache Arrow Flight support is started. But `ListFlights` is only supported for now. More features will be implemented in the next major release.
+
+### Ruby
+
+You need gobject-introspection gem 3.4.5 or later to implement your Apache Arrow Flight server. If you only use Apache Arrow Flight client, gobject-introspection gem 3.4.5 or later isn't required.
+
+Here are highlighted improvements:
+
+* Compute functions accept raw Ruby objects such as `true`, `Integer`, `Array` and `String`:
+
+  ```ruby
+  add_function = Arrow::Function.find("add")
+  # Not shortcut version
+  augend = Arrow::Int8Array.new([1, 2, 3])
+  addend = Arrow::Int8Scalar.new(5)
+  args = [
+    Arrow::ArrayDatum.new(augend),
+    Arrow::ScalarDatum.new(addend),
+  ]
+  add_function.execute(args).value.to_a # => [6, 7, 8]
+  # Shortcut version
+  add_function.execute([[1, 2, 3], 5]).value.to_a # => [6, 7, 8]
+  ```
+* `Arrow::PrimaryArray` and `Arrow::Buffer` can be used as MemoryView that is added in Ruby 3.0.
+
+There are some backward incompatible changes:
+
+* `Arrow::CountOptions` and `Arrow::CountMode` are removed. Use `Arrow::ScalarAggregateOptions` instead.
+
+### C GLib
+
+There are some backward incompatible changes:
+
+* `GArrowCountOptions` and `GArrowCountMode` are removed. Use `GArrowScalarAggregateOptions` instead.
+* `garrow_array_equal_range()` requires `GArrowEqualOptions`.
+* Prefix in arrow-dataset-glib is changed to `gadataset_`/`GADATASET_` from `gad_`/`GAD_`.
+* `GADScanOptions`, `GADScanTask`  and `GADInMemoryScanTask` are removed. Use `gadataset_begin_scan()` or `gadataset_to_table()` instead.
+* `GArrowCompareOptions`, `GArrowCompareOperator` and `garrow_*_array_compare()` are removed. Use `equal`, `not_equal`, `less_than`, `less_than_equal`, `greater_than` and `greater_than_equal` compute functions directly instead.
+
+## Rust notes
+
+The Rust projects have moved to separate repositories outside the
+main Arrow monorepo. For notes on the 5.0.0 release of the Rust
+implementation, see the [Arrow Rust changelog][3] and the
+"Apache Arrow Rust 5.0.0 Release" blog post.

Review comment:
       ```suggestion
   ["Apache Arrow Rust 5.0.0 Release" blog post](({% post_url 2021-07-20-5.0.0-rs-release %})).
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] pitrou commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

pitrou commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678264514



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,229 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format)
+and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for
+the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has
+been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+`MakeArrayFromScalar` now works for fixed-size binary types (ARROW-13321).
+
+### Compute layer
+
+The following [compute functions]({{site.baseurl}}/docs/cpp/compute.html)
+were added:
+
+* aggregations: "index"
+
+* scalar arithmetic and math functions: "abs", "abs_checked", "acos",
+  "acos_checked", "asin", "asin_checked", "atan", "atan2", "ceil", "cos",
+  "cos_checked", "floor", "ln", "ln_checked", "log10", "log10_checked",
+  "log1p", "log1p_checked", "log2", "log2_checked", "negate", "negate_checked",
+  "sign", "sin", "sin_checked", "tan", "tan_checked", "trunc"
+
+* scalar bitwise functions: "bit_wise_and", "bit_wise_not", "bit_wise_or",
+  "bit_wise_xor", "shift_left", "shift_left_checked", "shift_right",
+  "shift_right_checked"
+
+* scalar string functions: "ascii_center", "ascii_lpad", "ascii_reverse",
+  "ascii_rpad", "binary_join", "binary_join_element_wise",
+  "binary_replace_slice", "count_substring", "count_substring_regex",
+  "ends_with", "find_substring", "find_substring_regex", "match_like",
+  "split_pattern_regex", "starts_with", "utf8_center", "utf8_lpad",
+  "utf8_replace_slice", "utf8_rpad", "utf8_reverse", "utf8_slice_codepoints"
+
+* scalar temporal functions: "day", "day_of_week", "day_of_year",
+  "iso_calendar", "iso_week", "iso_year", "hour", "microsecond", "millisecond",
+  "minute", "month", "nanosecond", "quarter", "second", "subsecond", "year"
+
+* other scalar functions: "case_when", "coalesce", "if_else", "is_finite",
+  "is_inf", "is_nan", "max_element_wise", "min_element_wise", "make_struct"
+
+* vector functions: "replace_with_mask"
+
+Duplicates are now allowed in `SetLookupOptions::value_set` (ARROW-12554).
+
+Decimal types are now supported by some basic arithmetic functions (ARROW-12074).
+
+The "take" function now supports dense unions (ARROW-13005).
+
+It is now possible to cast between dictionary types with different index
+types (ARROW-11673).
+
+Sorting is now implemented for boolean input (ARROW-12016).
+
+### CSV
+
+The streaming CSV reader can now read in parallel (ARROW-11889).
+
+The CSV reader tries to make its errors more informative by adding the
+row number when it is known, i.e. when parallel reading is disabled (ARROW-12675).
+
+A new option `ReaderOptions::skip_rows_after_names` allows skipping a number
+of rows _after_ reading the column names (as opposed to
+`ReaderOptions::skip_rows`).
+
+Quoted strings can now be treated as always non-null (ARROW-10115).
+
+### Dataset layer
+

Review comment:
       Someone needs to fill this. @bkietz @lidavidm 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] nealrichardson commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

nealrichardson commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r678583374



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,259 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+3 months of development work and includes **XX commits** from
+[**XX distinct contributors**][1] in 2 repositories. See the Install Page to
+learn how to get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the complete changelogs for the [`apache/arrow`][2] and
+[`apache/arrow-rs`][3] repositories.
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format)
+and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend `.arrow` as the IPC file format file extension and `.arrows` for
+the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has
+been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+`MakeArrayFromScalar` now works for fixed-size binary types (ARROW-13321).
+
+### Compute layer
+
+The following [compute functions]({{site.baseurl}}/docs/cpp/compute.html)
+were added:
+
+* aggregations: `index`
+
+* scalar arithmetic and math functions: `abs`, `abs_checked`, `acos`,
+  `acos_checked`, `asin`, `asin_checked`, `atan`, `atan2`, `ceil`, `cos`,
+  `cos_checked`, `floor`, `ln`, `ln_checked`, `log10`, `log10_checked`,
+  `log1p`, `log1p_checked`, `log2`, `log2_checked`, `negate`, `negate_checked`,
+  `sign`, `sin`, `sin_checked`, `tan`, `tan_checked`, `trunc`
+
+* scalar bitwise functions: `bit_wise_and`, `bit_wise_not`, `bit_wise_or`,
+  `bit_wise_xor`, `shift_left`, `shift_left_checked`, `shift_right`,
+  `shift_right_checked`
+
+* scalar string functions: `ascii_center`, `ascii_lpad`, `ascii_reverse`,
+  `ascii_rpad`, `binary_join`, `binary_join_element_wise`,
+  `binary_replace_slice`, `count_substring`, `count_substring_regex`,
+  `ends_with`, `find_substring`, `find_substring_regex`, `match_like`,
+  `split_pattern_regex`, `starts_with`, `utf8_center`, `utf8_lpad`,
+  `utf8_replace_slice`, `utf8_rpad`, `utf8_reverse`, `utf8_slice_codepoints`
+
+* scalar temporal functions: `day`, `day_of_week`, `day_of_year`,
+  `iso_calendar`, `iso_week`, `iso_year`, `hour`, `microsecond`, `millisecond`,
+  `minute`, `month`, `nanosecond`, `quarter`, `second`, `subsecond`, `year`
+
+* other scalar functions: `case_when`, `coalesce`, `if_else`, `is_finite`,
+  `is_inf`, `is_nan`, `max_element_wise`, `min_element_wise`, `make_struct`
+
+* vector functions: `replace_with_mask`
+
+Duplicates are now allowed in `SetLookupOptions::value_set` (ARROW-12554).
+
+Decimal types are now supported by some basic arithmetic functions (ARROW-12074).
+
+The `take` function now supports dense unions (ARROW-13005).
+
+It is now possible to cast between dictionary types with different index
+types (ARROW-11673).
+
+Sorting is now implemented for boolean input (ARROW-12016).
+
+### CSV
+
+The streaming CSV reader can now take some advantage of multiple threads (ARROW-11889).
+
+The CSV reader tries to make its errors more informative by adding the
+row number when it is known, i.e. when parallel reading is disabled (ARROW-12675).
+
+A new option `ReaderOptions::skip_rows_after_names` allows skipping a number
+of rows _after_ reading the column names (as opposed to
+`ReaderOptions::skip_rows`).
+
+Quoted strings can now be treated as always non-null (ARROW-10115).
+
+### Dataset layer
+
+### IO and Filesystem layer
+
+The I/O thread pool size can now be adjuted at runtime (ARROW-12760).
+The default size remains 8 threads.
+
+Streams now can have auxiliary metadata, depending on the backend.  This
+has been implemented for the S3 filesystems, where a couple metadata
+keys are supported such as `Content-Type` and `ACL` (ARROW-11161, ARROW-12719).
+
+The HadoopFileSystem implementation now implements the FileSystem abstraction
+more faithfully (ARROW-12790).
+
+### Parquet
+
+The new LZ4_RAW compression scheme was implemented (PARQUET-1998).
+Unlike the legacy LZ4 compression scheme, it is defined unambiguously
+and should provide better portability once other Parquet implementations
+catch up.
+
+## Go notes
+
+* Flight Client and Server now support Custom Metadata through the functions `flight.NewClientWithMiddleware` and `flight.NewServerWithMiddleware`. Functions
+`flight.NewFlightClient`, `flight.NewFlightServer`, `flight.CreateServerBearerTokenAuthInterceptors` have been deprecated in favor of using the new middleware. [#10633](https://github.com/apache/arrow/pull/10633)
+* Fixed build failure for Parquet implementation on s390x [#10475](https://github.com/apache/arrow/pull/10475)
+* Flight Client `AuthHandler` no longer overwrites outgoing metadata, correctly appending new metadata without overwriting existing metadata [#10297](https://github.com/apache/arrow/pull/10297)
+* Flight AppMetadata field is now exposed both for Reading and Writing via `flight.Reader#LatestAppMetadata()` and `flight.Writer#WriteWithAppMetadata` functions [#10142](https://github.com/apache/arrow/pull/10142)
+* Extension Datatype is now implemented for Arrow Arrays [#10203](https://github.com/apache/arrow/pull/10203)
+* Map Datatype is now implemented for Arrow Arrays [#10106](https://github.com/apache/arrow/pull/10106)
+* Decimal integration tests with Go as both producer or consumer have been fixed [#10116](https://github.com/apache/arrow/10116)
+* Schema Package added for Golang Parquet Implementation [#10071](https://github.com/apache/arrow/10071)
+* First part of Encoding package for Golang Parquet Implementation added [#10379](https://github.com/apache/arrow/10379)

Review comment:
       ```suggestion
   
   ### Flight 
   
   * Flight Client and Server now support Custom Metadata through the functions `flight.NewClientWithMiddleware` and `flight.NewServerWithMiddleware`. Functions
   `flight.NewFlightClient`, `flight.NewFlightServer`, `flight.CreateServerBearerTokenAuthInterceptors` have been deprecated in favor of using the new middleware. [#10633](https://github.com/apache/arrow/pull/10633)
   * Flight Client `AuthHandler` no longer overwrites outgoing metadata, correctly appending new metadata without overwriting existing metadata [#10297](https://github.com/apache/arrow/pull/10297)
   * Flight AppMetadata field is now exposed both for Reading and Writing via `flight.Reader#LatestAppMetadata()` and `flight.Writer#WriteWithAppMetadata` functions [#10142](https://github.com/apache/arrow/pull/10142)
   
   ### Other enhancements 
   
   * [Map](https://github.com/apache/arrow/pull/10106) and [Extension](https://github.com/apache/arrow/pull/10203) Datatypes are now implemented for Arrow Arrays 
   * [Schema package](https://github.com/apache/arrow/10071) and first part of [Encoding package](https://github.com/apache/arrow/10379) added for Golang Parquet Implementation
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] alamb commented on pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

alamb commented on pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#issuecomment-882851381


   Here is proposed 5.0 content for Rust: https://github.com/apache/arrow-site/pull/128
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-site] nealrichardson commented on a change in pull request #127: Blog post for 5.0.0 release

Posted by GitBox <gi...@apache.org>.

nealrichardson commented on a change in pull request #127:
URL: https://github.com/apache/arrow-site/pull/127#discussion_r677732531



##########
File path: _posts/2021-07-20-5.0.0-release.md
##########
@@ -0,0 +1,113 @@
+---
+layout: post
+title: "Apache Arrow 5.0.0 Release"
+date: "2020-07-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Update all "XX" values with the appropriate numbers (you can get the resolved issues and contributors count from `_release/5.0.0.md`)
+* Fill in the various sections below. Note that the audience is the broader user community, not Arrow developers, so please write clearly using terms they will understand and care about. Delete any sections that don't have any content (as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the 5.0.0 release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+Since the 4.0.0 release, Daniël Heres, Kazuaki Ishizaki, Dominik Moritz, and Weston Pace
+have been invited as committers to Arrow,
+and Benjamin Kietzman and David Li have joined the Project Management Committee
+(PMC). Thank you for all of your contributions!
+
+## Columnar Format Notes
+
+Official IANA Media types (MIME types) have been registered for Apache
+Arrow IPC protocol data, both [stream]({{ site.baseurl }}/docs/format/Columnar.html#ipc-streaming-format) and [file]({{ site.baseurl }}/docs/format/Columnar.html#ipc-file-format) variants:
+
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.stream
+* https://www.iana.org/assignments/media-types/application/vnd.apache.arrow.file
+
+We recommend ".arrow" as the IPC file format file extension and ".arrows" for the IPC streaming format file extension.
+
+## Arrow Flight RPC notes
+
+The Go implementation now supports custom metadata and middleware, and has been added to integration testing.
+
+In Python, some operations can now be interrupted via Control-C.
+
+## C++ notes
+
+## C# notes
+
+## Go notes
+
+## Java notes

Review comment:
       @liyafan82 @BryanCutler anything to note for Java in this release?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org