You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by jo...@apache.org on 2021/05/15 05:36:59 UTC
[arrow] branch maint-4.0.x updated (f959141 -> cbcdb33)
This is an automated email from the ASF dual-hosted git repository.
jorgecarleitao pushed a change to branch maint-4.0.x
in repository https://gitbox.apache.org/repos/asf/arrow.git.
from f959141 [Release] Update versions for 4.0.0
new 28c1fd6 ARROW-12520: [R] Minor docs updates
new 3ed7fab ARROW-12601: [R][Packaging] Fix pkg-config check in r/configure
new 0bebdbc ARROW-11926: [R] preparations for ucrt toolchains
new 2b5b15d ARROW-12622: [Python] Fix segfault in read_csv when not on main thread
new cbcdb33 ARROW-12617: [Python] Align orc.write_table keyword order with parquet.write_table
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
python/pyarrow/_orc.pyx | 3 ++-
python/pyarrow/error.pxi | 2 +-
python/pyarrow/orc.py | 16 ++++++++++----
python/pyarrow/tests/test_csv.py | 8 +++++++
python/pyarrow/tests/test_orc.py | 10 ++++++++-
r/configure | 2 +-
r/configure.win | 2 +-
.../arrow/gpu/.gitignore => r/src/Makevars.ucrt | 3 ++-
r/vignettes/arrow.Rmd | 2 +-
r/vignettes/developing.Rmd | 25 +++++++++++-----------
10 files changed, 50 insertions(+), 23 deletions(-)
copy cpp/src/arrow/gpu/.gitignore => r/src/Makevars.ucrt (96%)
[arrow] 03/05: ARROW-11926: [R] preparations for ucrt toolchains
Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jorgecarleitao pushed a commit to branch maint-4.0.x
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit 0bebdbc26b41139433d97c23608afc2af4e18976
Author: Jeroen Ooms <je...@gmail.com>
AuthorDate: Fri Apr 30 15:09:28 2021 -0500
ARROW-11926: [R] preparations for ucrt toolchains
Minimal changes to link to ucrt builds specifically once supported (and will be safely redundant until then).
Closes #10217 from jeroen/winucrt
Lead-authored-by: Jeroen Ooms <je...@gmail.com>
Co-authored-by: Neal Richardson <ne...@gmail.com>
Signed-off-by: Jonathan Keane <jk...@gmail.com>
---
r/configure.win | 2 +-
r/src/Makevars.ucrt | 19 +++++++++++++++++++
2 files changed, 20 insertions(+), 1 deletion(-)
diff --git a/r/configure.win b/r/configure.win
index d645834..f31bf8f 100644
--- a/r/configure.win
+++ b/r/configure.win
@@ -50,7 +50,7 @@ AWS_LIBS="-laws-cpp-sdk-config -laws-cpp-sdk-transfer -laws-cpp-sdk-identity-man
# NOTE: If you make changes to the libraries below, you should also change
# ci/scripts/r_windows_build.sh and ci/scripts/PKGBUILD
PKG_CFLAGS="-I${RWINLIB}/include -DARROW_STATIC -DPARQUET_STATIC -DARROW_DS_STATIC -DARROW_R_WITH_ARROW -DARROW_R_WITH_PARQUET -DARROW_R_WITH_DATASET"
-PKG_LIBS="-L${RWINLIB}/lib"'$(subst gcc,,$(COMPILED_BY))$(R_ARCH) '"-L${RWINLIB}/lib"'$(R_ARCH) '"-lparquet -larrow_dataset -larrow -larrow_bundled_dependencies -lutf8proc -lthrift -lsnappy -lz -lzstd -llz4 ${MIMALLOC_LIBS} ${OPENSSL_LIBS}"
+PKG_LIBS="-L${RWINLIB}/lib"'$(subst gcc,,$(COMPILED_BY))$(R_ARCH) '"-L${RWINLIB}/lib"'$(R_ARCH)$(CRT) '"-lparquet -larrow_dataset -larrow -larrow_bundled_dependencies -lutf8proc -lthrift -lsnappy -lz -lzstd -llz4 ${MIMALLOC_LIBS} ${OPENSSL_LIBS}"
# S3 and re2 support only for Rtools40 (i.e. R >= 4.0)
"${R_HOME}/bin${R_ARCH_BIN}/Rscript.exe" -e 'R.version$major >= 4' | grep TRUE >/dev/null 2>&1
diff --git a/r/src/Makevars.ucrt b/r/src/Makevars.ucrt
new file mode 100644
index 0000000..52488eb
--- /dev/null
+++ b/r/src/Makevars.ucrt
@@ -0,0 +1,19 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+CRT=-ucrt
+include Makevars.win
[arrow] 01/05: ARROW-12520: [R] Minor docs updates
Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jorgecarleitao pushed a commit to branch maint-4.0.x
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit 28c1fd6175ef2b925562f4fd5f1afed5a02abd67
Author: Jonathan Keane <jk...@gmail.com>
AuthorDate: Mon Apr 26 12:23:47 2021 -0700
ARROW-12520: [R] Minor docs updates
Closes #10143 from jonkeane/ARROW-12520
Authored-by: Jonathan Keane <jk...@gmail.com>
Signed-off-by: Neal Richardson <ne...@gmail.com>
---
r/vignettes/arrow.Rmd | 2 +-
r/vignettes/developing.Rmd | 25 +++++++++++++------------
2 files changed, 14 insertions(+), 13 deletions(-)
diff --git a/r/vignettes/arrow.Rmd b/r/vignettes/arrow.Rmd
index 21cbbe4..4c5da50 100644
--- a/r/vignettes/arrow.Rmd
+++ b/r/vignettes/arrow.Rmd
@@ -104,7 +104,7 @@ corresponds to a class of the same name in the Arrow C++ library. The `arrow`
package provides a variety of `R6` and S3 methods for interacting with instances
of these classes.
-For convenience, the `arrow package also defines several synthetic classes that
+For convenience, the `arrow` package also defines several synthetic classes that
do not exist in the C++ library, including:
* `ArrowDatum`: inherited by `Scalar`, `Array`, and `ChunkedArray`
diff --git a/r/vignettes/developing.Rmd b/r/vignettes/developing.Rmd
index 38027a9..efb2c80 100644
--- a/r/vignettes/developing.Rmd
+++ b/r/vignettes/developing.Rmd
@@ -116,7 +116,7 @@ export ARROW_HOME=$(pwd)/dist
mkdir $ARROW_HOME
```
-_Special instructions on Linux:_ You will need to set `LD_LIBRARY_PATH` to the `lib` directory that will is under where we set `$ARROW_HOME`, before launching R and using Arrow. One way to do this is to add it to your profile (we use `~/.bash_profile` here, but you might need to put this in a different file depending on your setup, e.g. if you use a shell other than `bash`). On macOS we do not need to do this because the macOS shared library paths are hardcoded to their locations during [...]
+_Special instructions on Linux:_ You will need to set `LD_LIBRARY_PATH` to the `lib` directory that is under where we set `$ARROW_HOME`, before launching R and using Arrow. One way to do this is to add it to your profile (we use `~/.bash_profile` here, but you might need to put this in a different file depending on your setup, e.g. if you use a shell other than `bash`). On macOS we do not need to do this because the macOS shared library paths are hardcoded to their locations during build time.
```{bash, save=run & ubuntu & !sys_install}
export LD_LIBRARY_PATH=$ARROW_HOME/lib:$LD_LIBRARY_PATH
@@ -140,14 +140,14 @@ cmake \
-DARROW_COMPUTE=ON \
-DARROW_CSV=ON \
-DARROW_DATASET=ON \
+ -DARROW_EXTRA_ERROR_CONTEXT=ON \
-DARROW_FILESYSTEM=ON \
+ -DARROW_INSTALL_NAME_RPATH=OFF \
-DARROW_JEMALLOC=ON \
-DARROW_JSON=ON \
-DARROW_PARQUET=ON \
-DARROW_WITH_SNAPPY=ON \
-DARROW_WITH_ZLIB=ON \
- -DARROW_INSTALL_NAME_RPATH=OFF \
- -DARROW_EXTRA_ERROR_CONTEXT=ON \
..
```
@@ -172,14 +172,14 @@ cmake \
-DARROW_COMPUTE=ON \
-DARROW_CSV=ON \
-DARROW_DATASET=ON \
+ -DARROW_EXTRA_ERROR_CONTEXT=ON \
-DARROW_FILESYSTEM=ON \
+ -DARROW_INSTALL_NAME_RPATH=OFF \
-DARROW_JEMALLOC=ON \
-DARROW_JSON=ON \
-DARROW_PARQUET=ON \
-DARROW_WITH_SNAPPY=ON \
-DARROW_WITH_ZLIB=ON \
- -DARROW_INSTALL_NAME_RPATH=OFF \
- -DARROW_EXTRA_ERROR_CONTEXT=ON \
..
```
@@ -191,6 +191,7 @@ To enable optional features including: S3 support, an alternative memory allocat
``` shell
-DARROW_MIMALLOC=ON \
+ -DARROW_S3=ON \
-DARROW_WITH_BROTLI=ON \
-DARROW_WITH_BZ2=ON \
-DARROW_WITH_LZ4=ON \
@@ -252,7 +253,7 @@ export ARROW_R_CXXFLAGS=-fno-omit-frame-pointer
With the setups described here, you should not need to rebuild the Arrow library or even the C++ source in the R package as you iterated and work on the R package. The only time those should need to be rebuilt is if you have changed the C++ in the R package (and even then, `R CMD INSTALL .` should only need to recompile the files that have changed) _or_ if the Arrow library C++ has changed and there is a mismatch between the Arrow Library and the R package. If you find yourself rebuildin [...]
<details>
-<summary>For a full build: a `cmake` command with all of the R-relevant optional dependencies turned on</summary>
+<summary>For a full build: a `cmake` command with all of the R-relevant optional dependencies turned on. Development with other languages might require different flags as well. For example, to develop Python, you would need to also add `-DARROW_PYTHON=ON` (though all of the other flags used for Python are already included here).</summary>
<p>
``` shell
@@ -262,26 +263,26 @@ cmake \
-DARROW_COMPUTE=ON \
-DARROW_CSV=ON \
-DARROW_DATASET=ON \
+ -DARROW_EXTRA_ERROR_CONTEXT=ON \
-DARROW_FILESYSTEM=ON \
+ -DARROW_INSTALL_NAME_RPATH=OFF \
-DARROW_JEMALLOC=ON \
-DARROW_JSON=ON \
- -DARROW_PARQUET=ON \
- -DARROW_WITH_SNAPPY=ON \
- -DARROW_WITH_ZLIB=ON \
- -DARROW_INSTALL_NAME_RPATH=OFF \
- -DARROW_EXTRA_ERROR_CONTEXT=ON \
-DARROW_MIMALLOC=ON \
+ -DARROW_PARQUET=ON \
+ -DARROW_S3=ON \
-DARROW_WITH_BROTLI=ON \
-DARROW_WITH_BZ2=ON \
-DARROW_WITH_LZ4=ON \
-DARROW_WITH_SNAPPY=ON \
+ -DARROW_WITH_ZLIB=ON \
-DARROW_WITH_ZSTD=ON \
..
```
</p>
</details>
-## Troublshooting
+## Troubleshooting
Note that after any change to the C++ library, you must reinstall it and
run `make clean` or `git clean -fdx .` to remove any cached object code
[arrow] 02/05: ARROW-12601: [R][Packaging] Fix pkg-config check in
r/configure
Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jorgecarleitao pushed a commit to branch maint-4.0.x
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit 3ed7fab9eb56a60e45a0de4374616a25c4091a17
Author: Ian Cook <ia...@gmail.com>
AuthorDate: Thu Apr 29 17:32:33 2021 -0400
ARROW-12601: [R][Packaging] Fix pkg-config check in r/configure
Fixes false positives in a check that pkg-config is installed
Closes #10198 from ianmcook/ARROW-12601
Authored-by: Ian Cook <ia...@gmail.com>
Signed-off-by: Ian Cook <ia...@gmail.com>
---
r/configure | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/r/configure b/r/configure
index f6c1013..2da2b9f 100755
--- a/r/configure
+++ b/r/configure
@@ -77,7 +77,7 @@ elif [ "$INCLUDE_DIR" ] && [ "$LIB_DIR" ]; then
else
# Use pkg-config if available and allowed
pkg-config --version >/dev/null 2>&1
- if [ "$ARROW_USE_PKG_CONFIG" != "false" ] && [ $? -eq 0 ]; then
+ if [ $? -eq 0 ] && [ "$ARROW_USE_PKG_CONFIG" != "false" ]; then
PKGCONFIG_CFLAGS=`pkg-config --cflags --silence-errors ${PKG_CONFIG_NAME}`
PKGCONFIG_LIBS=`pkg-config --libs-only-l --silence-errors ${PKG_CONFIG_NAME}`
PKGCONFIG_DIRS=`pkg-config --libs-only-L --silence-errors ${PKG_CONFIG_NAME}`
[arrow] 05/05: ARROW-12617: [Python] Align orc.write_table keyword
order with parquet.write_table
Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jorgecarleitao pushed a commit to branch maint-4.0.x
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit cbcdb33a78763cb39b201241229c0a41cff236e0
Author: Joris Van den Bossche <jo...@gmail.com>
AuthorDate: Mon May 3 17:29:16 2021 +0200
ARROW-12617: [Python] Align orc.write_table keyword order with parquet.write_table
Closes #10223 from jorisvandenbossche/ARROW-12617-orc-write_table-signature
Authored-by: Joris Van den Bossche <jo...@gmail.com>
Signed-off-by: Antoine Pitrou <an...@python.org>
---
python/pyarrow/_orc.pyx | 3 ++-
python/pyarrow/orc.py | 16 ++++++++++++----
python/pyarrow/tests/test_orc.py | 10 +++++++++-
3 files changed, 23 insertions(+), 6 deletions(-)
diff --git a/python/pyarrow/_orc.pyx b/python/pyarrow/_orc.pyx
index 2640057..e56a62d 100644
--- a/python/pyarrow/_orc.pyx
+++ b/python/pyarrow/_orc.pyx
@@ -29,6 +29,7 @@ from pyarrow.lib cimport (check_status, _Weakrefable,
Schema, pyarrow_wrap_schema,
pyarrow_wrap_batch,
RecordBatch,
+ Table,
pyarrow_wrap_table,
pyarrow_unwrap_schema,
pyarrow_unwrap_table,
@@ -127,7 +128,7 @@ cdef class ORCWriter(_Weakrefable):
self.writer = move(GetResultValue[unique_ptr[ORCFileWriter]](
ORCFileWriter.Open(self.rd_handle.get())))
- def write(self, object table):
+ def write(self, Table table):
cdef:
shared_ptr[CTable] sp_table
sp_table = pyarrow_unwrap_table(table)
diff --git a/python/pyarrow/orc.py b/python/pyarrow/orc.py
index e1009bc..13af5a2 100644
--- a/python/pyarrow/orc.py
+++ b/python/pyarrow/orc.py
@@ -18,9 +18,10 @@
from itertools import count
from numbers import Integral
+import warnings
from pyarrow import types
-from pyarrow.lib import Schema
+from pyarrow.lib import Schema, Table
import pyarrow._orc as _orc
@@ -184,19 +185,26 @@ class ORCWriter:
self.writer.close()
-def write_table(where, table):
+def write_table(table, where):
"""
Write a table into an ORC file
Parameters
----------
+ table : pyarrow.lib.Table
+ The table to be written into the ORC file
where : str or pyarrow.io.NativeFile
Writable target. For passing Python file objects or byte buffers,
see pyarrow.io.PythonFileInterface, pyarrow.io.BufferOutputStream
or pyarrow.io.FixedSizeBufferWriter.
- table : pyarrow.lib.Table
- The table to be written into the ORC file
"""
+ if isinstance(where, Table):
+ warnings.warn(
+ "The order of the arguments has changed. Pass as "
+ "'write_table(table, where)' instead. The old order will raise "
+ "an error in the future.", FutureWarning, stacklevel=2
+ )
+ table, where = where, table
writer = ORCWriter(where)
writer.write(table)
writer.close()
diff --git a/python/pyarrow/tests/test_orc.py b/python/pyarrow/tests/test_orc.py
index 14edad8..e71c452 100644
--- a/python/pyarrow/tests/test_orc.py
+++ b/python/pyarrow/tests/test_orc.py
@@ -176,7 +176,15 @@ def test_orcfile_readwrite():
a = pa.array([1, None, 3, None])
b = pa.array([None, "Arrow", None, "ORC"])
table = pa.table({"int64": a, "utf8": b})
- orc.write_table(buffer_output_stream, table)
+ orc.write_table(table, buffer_output_stream)
+ buffer_reader = pa.BufferReader(buffer_output_stream.getvalue())
+ output_table = orc.ORCFile(buffer_reader).read()
+ assert table.equals(output_table)
+
+ # deprecated keyword order
+ buffer_output_stream = pa.BufferOutputStream()
+ with pytest.warns(FutureWarning):
+ orc.write_table(buffer_output_stream, table)
buffer_reader = pa.BufferReader(buffer_output_stream.getvalue())
output_table = orc.ORCFile(buffer_reader).read()
assert table.equals(output_table)
[arrow] 04/05: ARROW-12622: [Python] Fix segfault in read_csv when
not on main thread
Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jorgecarleitao pushed a commit to branch maint-4.0.x
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit 2b5b15dd7c37ec2033f114b3e9c65e70d8dfc38a
Author: David Li <li...@gmail.com>
AuthorDate: Mon May 3 16:53:16 2021 +0200
ARROW-12622: [Python] Fix segfault in read_csv when not on main thread
An uninitialized StopToken caused segfaults if you ever called read_csv with cancellation disabled or when not on the main thread (e.g. if used in a Flight server). If we have a 4.0.1 I think this qualifies as a regression.
Closes #10227 from lidavidm/arrow-12622
Authored-by: David Li <li...@gmail.com>
Signed-off-by: Antoine Pitrou <an...@python.org>
---
python/pyarrow/error.pxi | 2 +-
python/pyarrow/tests/test_csv.py | 8 ++++++++
2 files changed, 9 insertions(+), 1 deletion(-)
diff --git a/python/pyarrow/error.pxi b/python/pyarrow/error.pxi
index f9e45f2..2866848 100644
--- a/python/pyarrow/error.pxi
+++ b/python/pyarrow/error.pxi
@@ -188,8 +188,8 @@ cdef class SignalStopHandler:
if signal.getsignal(sig) not in (signal.SIG_DFL,
signal.SIG_IGN, None)]
+ self._stop_token = StopToken()
if not self._signals.empty():
- self._stop_token = StopToken()
self._stop_token.init(GetResultValue(
SetSignalStopSource()).token())
self._enabled = True
diff --git a/python/pyarrow/tests/test_csv.py b/python/pyarrow/tests/test_csv.py
index 395f948..34ab556 100644
--- a/python/pyarrow/tests/test_csv.py
+++ b/python/pyarrow/tests/test_csv.py
@@ -942,6 +942,14 @@ class BaseTestCSVRead:
assert isinstance(e, pa.ArrowCancelled)
assert e.signum == signal.SIGINT
+ def test_cancellation_disabled(self):
+ # ARROW-12622: reader would segfault when the cancelling signal
+ # handler was not enabled (e.g. if disabled, or if not on the
+ # main thread)
+ t = threading.Thread(target=lambda: self.read_bytes(b"f64\n0.1"))
+ t.start()
+ t.join()
+
class TestSerialCSVRead(BaseTestCSVRead, unittest.TestCase):