You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "mgorny (via GitHub)" <gi...@apache.org> on 2024/02/20 11:44:01 UTC
[I] Test failures on 32-bit x86 [arrow]
mgorny opened a new issue, #40153:
URL: https://github.com/apache/arrow/issues/40153
### Describe the bug, including details regarding any error messages, version, and platform.
When running the test suite on 32-bit x86, I'm getting the following test failures:
```
FAILED tests/test_array.py::test_dictionary_to_numpy - TypeError: Cannot cast array data from dtype('int64') to dtype('int32') according to the rule 'safe'
FAILED tests/test_io.py::test_python_file_large_seeks - assert 5 == ((2 ** 32) + 5)
FAILED tests/test_io.py::test_memory_map_large_seeks - OSError: Read out of bounds (offset = 4294967301, size = 5) in file of size 10
FAILED tests/test_pandas.py::TestConvertStructTypes::test_from_numpy_nested - AssertionError: assert 8 == 12
FAILED tests/test_schema.py::test_schema_sizeof - assert 28 > 30
FAILED tests/interchange/test_conversion.py::test_pandas_roundtrip_string - OverflowError: Python int too large to convert to C ssize_t
FAILED tests/interchange/test_conversion.py::test_pandas_roundtrip_large_string - OverflowError: Python int too large to convert to C ssize_t
FAILED tests/interchange/test_conversion.py::test_pandas_roundtrip_string_with_missing - OverflowError: Python int too large to convert to C ssize_t
FAILED tests/interchange/test_conversion.py::test_pandas_roundtrip_categorical - OverflowError: Python int too large to convert to C ssize_t
FAILED tests/interchange/test_conversion.py::test_empty_dataframe - OverflowError: Python int too large to convert to C ssize_t
```
<details>
<summary>Tracebacks</summary>
```pytb
============================================================== FAILURES ===============================================================
______________________________________________________ test_dictionary_to_numpy _______________________________________________________
obj = array([13.7, 11. ]), method = 'take', args = (array([0, 1, 1, 0], dtype=int64),)
kwds = {'axis': None, 'mode': 'raise', 'out': None}, bound = <built-in method take of numpy.ndarray object at 0xeaca6ad0>
def _wrapfunc(obj, method, *args, **kwds):
bound = getattr(obj, method, None)
if bound is None:
return _wrapit(obj, method, *args, **kwds)
try:
> return bound(*args, **kwds)
E TypeError: Cannot cast array data from dtype('int64') to dtype('int32') according to the rule 'safe'
args = (array([0, 1, 1, 0], dtype=int64),)
bound = <built-in method take of numpy.ndarray object at 0xeaca6ad0>
kwds = {'axis': None, 'mode': 'raise', 'out': None}
method = 'take'
obj = array([13.7, 11. ])
/usr/lib/python3.11/site-packages/numpy/core/fromnumeric.py:59: TypeError
During handling of the above exception, another exception occurred:
def test_dictionary_to_numpy():
expected = pa.array(
["foo", "bar", None, "foo"]
).to_numpy(zero_copy_only=False)
a = pa.DictionaryArray.from_arrays(
pa.array([0, 1, None, 0]),
pa.array(['foo', 'bar'])
)
np.testing.assert_array_equal(a.to_numpy(zero_copy_only=False),
expected)
with pytest.raises(pa.ArrowInvalid):
# If this would be changed to no longer raise in the future,
# ensure to test the actual result because, currently, to_numpy takes
# for granted that when zero_copy_only=True there will be no nulls
# (it's the decoding of the DictionaryArray that handles the nulls and
# this is only activated with zero_copy_only=False)
a.to_numpy(zero_copy_only=True)
anonulls = pa.DictionaryArray.from_arrays(
pa.array([0, 1, 1, 0]),
pa.array(['foo', 'bar'])
)
expected = pa.array(
["foo", "bar", "bar", "foo"]
).to_numpy(zero_copy_only=False)
np.testing.assert_array_equal(anonulls.to_numpy(zero_copy_only=False),
expected)
with pytest.raises(pa.ArrowInvalid):
anonulls.to_numpy(zero_copy_only=True)
afloat = pa.DictionaryArray.from_arrays(
pa.array([0, 1, 1, 0]),
pa.array([13.7, 11.0])
)
expected = pa.array([13.7, 11.0, 11.0, 13.7]).to_numpy()
> np.testing.assert_array_equal(afloat.to_numpy(zero_copy_only=True),
expected)
a = <pyarrow.lib.DictionaryArray object at 0xeafe6ed0>
-- dictionary:
[
"foo",
"bar"
]
-- indices:
[
0,
1,
null,
0
]
afloat = <pyarrow.lib.DictionaryArray object at 0xeafe6fb0>
-- dictionary:
[
13.7,
11
]
-- indices:
[
0,
1,
1,
0
]
anonulls = <pyarrow.lib.DictionaryArray object at 0xeafe6e60>
-- dictionary:
[
"foo",
"bar"
]
-- indices:
[
0,
1,
1,
0
]
expected = array([13.7, 11. , 11. , 13.7])
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/tests/test_array.py:823:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pyarrow/array.pxi:1590: in pyarrow.lib.Array.to_numpy
???
/usr/lib/python3.11/site-packages/numpy/core/fromnumeric.py:192: in take
return _wrapfunc(a, 'take', indices, axis=axis, out=out, mode=mode)
a = array([13.7, 11. ])
axis = None
indices = array([0, 1, 1, 0], dtype=int64)
mode = 'raise'
out = None
/usr/lib/python3.11/site-packages/numpy/core/fromnumeric.py:68: in _wrapfunc
return _wrapit(obj, method, *args, **kwds)
args = (array([0, 1, 1, 0], dtype=int64),)
bound = <built-in method take of numpy.ndarray object at 0xeaca6ad0>
kwds = {'axis': None, 'mode': 'raise', 'out': None}
method = 'take'
obj = array([13.7, 11. ])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
obj = array([13.7, 11. ]), method = 'take', args = (array([0, 1, 1, 0], dtype=int64),)
kwds = {'axis': None, 'mode': 'raise', 'out': None}, wrap = <built-in method __array_wrap__ of numpy.ndarray object at 0xeaca6ad0>
def _wrapit(obj, method, *args, **kwds):
try:
wrap = obj.__array_wrap__
except AttributeError:
wrap = None
> result = getattr(asarray(obj), method)(*args, **kwds)
E TypeError: Cannot cast array data from dtype('int64') to dtype('int32') according to the rule 'safe'
args = (array([0, 1, 1, 0], dtype=int64),)
kwds = {'axis': None, 'mode': 'raise', 'out': None}
method = 'take'
obj = array([13.7, 11. ])
wrap = <built-in method __array_wrap__ of numpy.ndarray object at 0xeaca6ad0>
/usr/lib/python3.11/site-packages/numpy/core/fromnumeric.py:45: TypeError
____________________________________________________ test_python_file_large_seeks _____________________________________________________
def test_python_file_large_seeks():
def factory(filename):
return pa.PythonFile(open(filename, 'rb'))
> check_large_seeks(factory)
factory = <function test_python_file_large_seeks.<locals>.factory at 0xe13b6de8>
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/tests/test_io.py:262:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
file_factory = <function test_python_file_large_seeks.<locals>.factory at 0xe13b6de8>
def check_large_seeks(file_factory):
if sys.platform in ('win32', 'darwin'):
pytest.skip("need sparse file support")
try:
filename = tempfile.mktemp(prefix='test_io')
with open(filename, 'wb') as f:
f.truncate(2 ** 32 + 10)
f.seek(2 ** 32 + 5)
f.write(b'mark\n')
with file_factory(filename) as f:
> assert f.seek(2 ** 32 + 5) == 2 ** 32 + 5
E assert 5 == ((2 ** 32) + 5)
E + where 5 = <bound method NativeFile.seek of <pyarrow.PythonFile closed=False own_file=False is_seekable=True is_writable=False is_readable=True>>(((2 ** 32) + 5))
E + where <bound method NativeFile.seek of <pyarrow.PythonFile closed=False own_file=False is_seekable=True is_writable=False is_readable=True>> = <pyarrow.PythonFile closed=False own_file=False is_seekable=True is_writable=False is_readable=True>.seek
f = <pyarrow.PythonFile closed=True own_file=False is_seekable=True is_writable=False is_readable=True>
file_factory = <function test_python_file_large_seeks.<locals>.factory at 0xe13b6de8>
filename = '/var/tmp/portage/dev-python/pyarrow-15.0.0/temp/test_ioj_p6zuld'
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/tests/test_io.py:49: AssertionError
_____________________________________________________ test_memory_map_large_seeks _____________________________________________________
def test_memory_map_large_seeks():
> check_large_seeks(pa.memory_map)
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/tests/test_io.py:1140:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/tests/test_io.py:51: in check_large_seeks
assert f.read(5) == b'mark\n'
f = <pyarrow.MemoryMappedFile closed=True own_file=False is_seekable=True is_writable=False is_readable=True>
file_factory = <cyfunction memory_map at 0xf228e778>
filename = '/var/tmp/portage/dev-python/pyarrow-15.0.0/temp/test_iozl2wxbou'
pyarrow/io.pxi:409: in pyarrow.lib.NativeFile.read
???
pyarrow/error.pxi:154: in pyarrow.lib.pyarrow_internal_check_status
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E OSError: Read out of bounds (offset = 4294967301, size = 5) in file of size 10
pyarrow/error.pxi:91: OSError
____________________________________________ TestConvertStructTypes.test_from_numpy_nested ____________________________________________
self = <pyarrow.tests.test_pandas.TestConvertStructTypes object at 0xeb535d90>
def test_from_numpy_nested(self):
# Note: an object field inside a struct
dt = np.dtype([('x', np.dtype([('xx', np.int8),
('yy', np.bool_)])),
('y', np.int16),
('z', np.object_)])
# Note: itemsize is not a multiple of sizeof(object)
> assert dt.itemsize == 12
E AssertionError: assert 8 == 12
E + where 8 = dtype([('x', [('xx', 'i1'), ('yy', '?')]), ('y', '<i2'), ('z', 'O')]).itemsize
dt = dtype([('x', [('xx', 'i1'), ('yy', '?')]), ('y', '<i2'), ('z', 'O')])
self = <pyarrow.tests.test_pandas.TestConvertStructTypes object at 0xeb535d90>
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/tests/test_pandas.py:2604: AssertionError
_________________________________________________________ test_schema_sizeof __________________________________________________________
def test_schema_sizeof():
schema = pa.schema([
pa.field('foo', pa.int32()),
pa.field('bar', pa.string()),
])
> assert sys.getsizeof(schema) > 30
E assert 28 > 30
E + where 28 = <built-in function getsizeof>(foo: int32\nbar: string)
E + where <built-in function getsizeof> = sys.getsizeof
schema = foo: int32
bar: string
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/tests/test_schema.py:684: AssertionError
____________________________________________________ test_pandas_roundtrip_string _____________________________________________________
@pytest.mark.pandas
def test_pandas_roundtrip_string():
# See https://github.com/pandas-dev/pandas/issues/50554
if Version(pd.__version__) < Version("1.6"):
pytest.skip("Column.size() bug in pandas")
arr = ["a", "", "c"]
table = pa.table({"a": pa.array(arr)})
from pandas.api.interchange import (
from_dataframe as pandas_from_dataframe
)
pandas_df = pandas_from_dataframe(table)
> result = pi.from_dataframe(pandas_df)
arr = ['a', '', 'c']
pandas_df = a
0 a
1
2 c
pandas_from_dataframe = <function from_dataframe at 0xebbaa398>
table = pyarrow.Table
a: string
----
a: [["a","","c"]]
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/tests/interchange/test_conversion.py:159:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:113: in from_dataframe
return _from_dataframe(df.__dataframe__(allow_copy=allow_copy),
allow_copy = True
df = a
0 a
1
2 c
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:136: in _from_dataframe
batch = protocol_df_chunk_to_pyarrow(chunk, allow_copy)
allow_copy = True
batches = []
chunk = <pandas.core.interchange.dataframe.PandasDataFrameXchg object at 0xda1c61f0>
df = <pandas.core.interchange.dataframe.PandasDataFrameXchg object at 0xda1c61f0>
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:182: in protocol_df_chunk_to_pyarrow
columns[name] = column_to_array(col, allow_copy)
allow_copy = True
col = <pandas.core.interchange.column.PandasColumn object at 0xda1c65b0>
columns = {}
df = <pandas.core.interchange.dataframe.PandasDataFrameXchg object at 0xda1c61f0>
dtype = <DtypeKind.STRING: 21>
name = 'a'
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:214: in column_to_array
data = buffers_to_array(buffers, data_type,
allow_copy = True
buffers = {'data': (PandasBuffer({'bufsize': 2, 'ptr': 3879523528, 'device': 'CPU'}),
(<DtypeKind.STRING: 21>, 8, 'u', '=')),
'offsets': (PandasBuffer({'bufsize': 32, 'ptr': 1530035680, 'device': 'CPU'}),
(<DtypeKind.INT: 0>, 64, 'l', '=')),
'validity': (PandasBuffer({'bufsize': 3, 'ptr': 1529980112, 'device': 'CPU'}),
(<DtypeKind.BOOL: 20>, 8, 'b', '='))}
col = <pandas.core.interchange.column.PandasColumn object at 0xda1c65b0>
data_type = (<DtypeKind.STRING: 21>, 8, 'u', '=')
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:396: in buffers_to_array
data_pa_buffer = pa.foreign_buffer(data_buff.ptr, data_buff.bufsize,
_ = (<DtypeKind.STRING: 21>, 8, 'u', '=')
allow_copy = True
buffers = {'data': (PandasBuffer({'bufsize': 2, 'ptr': 3879523528, 'device': 'CPU'}),
(<DtypeKind.STRING: 21>, 8, 'u', '=')),
'offsets': (PandasBuffer({'bufsize': 32, 'ptr': 1530035680, 'device': 'CPU'}),
(<DtypeKind.INT: 0>, 64, 'l', '=')),
'validity': (PandasBuffer({'bufsize': 3, 'ptr': 1529980112, 'device': 'CPU'}),
(<DtypeKind.BOOL: 20>, 8, 'b', '='))}
data_buff = PandasBuffer({'bufsize': 2, 'ptr': 3879523528, 'device': 'CPU'})
data_type = (<DtypeKind.STRING: 21>, 8, 'u', '=')
describe_null = (<ColumnNullType.USE_BYTEMASK: 4>, 0)
length = 3
offset = 0
offset_buff = PandasBuffer({'bufsize': 32, 'ptr': 1530035680, 'device': 'CPU'})
offset_dtype = (<DtypeKind.INT: 0>, 64, 'l', '=')
validity_buff = PandasBuffer({'bufsize': 3, 'ptr': 1529980112, 'device': 'CPU'})
validity_dtype = (<DtypeKind.BOOL: 20>, 8, 'b', '=')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E OverflowError: Python int too large to convert to C ssize_t
pyarrow/io.pxi:1990: OverflowError
_________________________________________________ test_pandas_roundtrip_large_string __________________________________________________
@pytest.mark.pandas
def test_pandas_roundtrip_large_string():
# See https://github.com/pandas-dev/pandas/issues/50554
if Version(pd.__version__) < Version("1.6"):
pytest.skip("Column.size() bug in pandas")
arr = ["a", "", "c"]
table = pa.table({"a_large": pa.array(arr, type=pa.large_string())})
from pandas.api.interchange import (
from_dataframe as pandas_from_dataframe
)
if Version(pd.__version__) >= Version("2.0.1"):
pandas_df = pandas_from_dataframe(table)
> result = pi.from_dataframe(pandas_df)
arr = ['a', '', 'c']
pandas_df = a_large
0 a
1
2 c
pandas_from_dataframe = <function from_dataframe at 0xebbaa398>
table = pyarrow.Table
a_large: large_string
----
a_large: [["a","","c"]]
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/tests/interchange/test_conversion.py:189:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:113: in from_dataframe
return _from_dataframe(df.__dataframe__(allow_copy=allow_copy),
allow_copy = True
df = a_large
0 a
1
2 c
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:136: in _from_dataframe
batch = protocol_df_chunk_to_pyarrow(chunk, allow_copy)
allow_copy = True
batches = []
chunk = <pandas.core.interchange.dataframe.PandasDataFrameXchg object at 0xda103a10>
df = <pandas.core.interchange.dataframe.PandasDataFrameXchg object at 0xda103a10>
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:182: in protocol_df_chunk_to_pyarrow
columns[name] = column_to_array(col, allow_copy)
allow_copy = True
col = <pandas.core.interchange.column.PandasColumn object at 0xda1033d0>
columns = {}
df = <pandas.core.interchange.dataframe.PandasDataFrameXchg object at 0xda103a10>
dtype = <DtypeKind.STRING: 21>
name = 'a_large'
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:214: in column_to_array
data = buffers_to_array(buffers, data_type,
allow_copy = True
buffers = {'data': (PandasBuffer({'bufsize': 2, 'ptr': 3879522800, 'device': 'CPU'}),
(<DtypeKind.STRING: 21>, 8, 'u', '=')),
'offsets': (PandasBuffer({'bufsize': 32, 'ptr': 1480303312, 'device': 'CPU'}),
(<DtypeKind.INT: 0>, 64, 'l', '=')),
'validity': (PandasBuffer({'bufsize': 3, 'ptr': 1478277616, 'device': 'CPU'}),
(<DtypeKind.BOOL: 20>, 8, 'b', '='))}
col = <pandas.core.interchange.column.PandasColumn object at 0xda1033d0>
data_type = (<DtypeKind.STRING: 21>, 8, 'u', '=')
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:396: in buffers_to_array
data_pa_buffer = pa.foreign_buffer(data_buff.ptr, data_buff.bufsize,
_ = (<DtypeKind.STRING: 21>, 8, 'u', '=')
allow_copy = True
buffers = {'data': (PandasBuffer({'bufsize': 2, 'ptr': 3879522800, 'device': 'CPU'}),
(<DtypeKind.STRING: 21>, 8, 'u', '=')),
'offsets': (PandasBuffer({'bufsize': 32, 'ptr': 1480303312, 'device': 'CPU'}),
(<DtypeKind.INT: 0>, 64, 'l', '=')),
'validity': (PandasBuffer({'bufsize': 3, 'ptr': 1478277616, 'device': 'CPU'}),
(<DtypeKind.BOOL: 20>, 8, 'b', '='))}
data_buff = PandasBuffer({'bufsize': 2, 'ptr': 3879522800, 'device': 'CPU'})
data_type = (<DtypeKind.STRING: 21>, 8, 'u', '=')
describe_null = (<ColumnNullType.USE_BYTEMASK: 4>, 0)
length = 3
offset = 0
offset_buff = PandasBuffer({'bufsize': 32, 'ptr': 1480303312, 'device': 'CPU'})
offset_dtype = (<DtypeKind.INT: 0>, 64, 'l', '=')
validity_buff = PandasBuffer({'bufsize': 3, 'ptr': 1478277616, 'device': 'CPU'})
validity_dtype = (<DtypeKind.BOOL: 20>, 8, 'b', '=')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E OverflowError: Python int too large to convert to C ssize_t
pyarrow/io.pxi:1990: OverflowError
______________________________________________ test_pandas_roundtrip_string_with_missing ______________________________________________
@pytest.mark.pandas
def test_pandas_roundtrip_string_with_missing():
# See https://github.com/pandas-dev/pandas/issues/50554
if Version(pd.__version__) < Version("1.6"):
pytest.skip("Column.size() bug in pandas")
arr = ["a", "", "c", None]
table = pa.table({"a": pa.array(arr),
"a_large": pa.array(arr, type=pa.large_string())})
from pandas.api.interchange import (
from_dataframe as pandas_from_dataframe
)
if Version(pd.__version__) >= Version("2.0.2"):
pandas_df = pandas_from_dataframe(table)
> result = pi.from_dataframe(pandas_df)
arr = ['a', '', 'c', None]
pandas_df = a a_large
0 a a
1
2 c c
3 NaN NaN
pandas_from_dataframe = <function from_dataframe at 0xebbaa398>
table = pyarrow.Table
a: string
a_large: large_string
----
a: [["a","","c",null]]
a_large: [["a","","c",null]]
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/tests/interchange/test_conversion.py:227:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:113: in from_dataframe
return _from_dataframe(df.__dataframe__(allow_copy=allow_copy),
allow_copy = True
df = a a_large
0 a a
1
2 c c
3 NaN NaN
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:136: in _from_dataframe
batch = protocol_df_chunk_to_pyarrow(chunk, allow_copy)
allow_copy = True
batches = []
chunk = <pandas.core.interchange.dataframe.PandasDataFrameXchg object at 0xda15b850>
df = <pandas.core.interchange.dataframe.PandasDataFrameXchg object at 0xda15b850>
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:182: in protocol_df_chunk_to_pyarrow
columns[name] = column_to_array(col, allow_copy)
allow_copy = True
col = <pandas.core.interchange.column.PandasColumn object at 0xda103210>
columns = {}
df = <pandas.core.interchange.dataframe.PandasDataFrameXchg object at 0xda15b850>
dtype = <DtypeKind.STRING: 21>
name = 'a'
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:214: in column_to_array
data = buffers_to_array(buffers, data_type,
allow_copy = True
buffers = {'data': (PandasBuffer({'bufsize': 2, 'ptr': 3879523744, 'device': 'CPU'}),
(<DtypeKind.STRING: 21>, 8, 'u', '=')),
'offsets': (PandasBuffer({'bufsize': 40, 'ptr': 1469510752, 'device': 'CPU'}),
(<DtypeKind.INT: 0>, 64, 'l', '=')),
'validity': (PandasBuffer({'bufsize': 4, 'ptr': 1475420176, 'device': 'CPU'}),
(<DtypeKind.BOOL: 20>, 8, 'b', '='))}
col = <pandas.core.interchange.column.PandasColumn object at 0xda103210>
data_type = (<DtypeKind.STRING: 21>, 8, 'u', '=')
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:396: in buffers_to_array
data_pa_buffer = pa.foreign_buffer(data_buff.ptr, data_buff.bufsize,
_ = (<DtypeKind.STRING: 21>, 8, 'u', '=')
allow_copy = True
buffers = {'data': (PandasBuffer({'bufsize': 2, 'ptr': 3879523744, 'device': 'CPU'}),
(<DtypeKind.STRING: 21>, 8, 'u', '=')),
'offsets': (PandasBuffer({'bufsize': 40, 'ptr': 1469510752, 'device': 'CPU'}),
(<DtypeKind.INT: 0>, 64, 'l', '=')),
'validity': (PandasBuffer({'bufsize': 4, 'ptr': 1475420176, 'device': 'CPU'}),
(<DtypeKind.BOOL: 20>, 8, 'b', '='))}
data_buff = PandasBuffer({'bufsize': 2, 'ptr': 3879523744, 'device': 'CPU'})
data_type = (<DtypeKind.STRING: 21>, 8, 'u', '=')
describe_null = (<ColumnNullType.USE_BYTEMASK: 4>, 0)
length = 4
offset = 0
offset_buff = PandasBuffer({'bufsize': 40, 'ptr': 1469510752, 'device': 'CPU'})
offset_dtype = (<DtypeKind.INT: 0>, 64, 'l', '=')
validity_buff = PandasBuffer({'bufsize': 4, 'ptr': 1475420176, 'device': 'CPU'})
validity_dtype = (<DtypeKind.BOOL: 20>, 8, 'b', '=')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E OverflowError: Python int too large to convert to C ssize_t
pyarrow/io.pxi:1990: OverflowError
__________________________________________________ test_pandas_roundtrip_categorical __________________________________________________
@pytest.mark.pandas
def test_pandas_roundtrip_categorical():
if Version(pd.__version__) < Version("2.0.2"):
pytest.skip("Bitmasks not supported in pandas interchange implementation")
arr = ["Mon", "Tue", "Mon", "Wed", "Mon", "Thu", "Fri", "Sat", None]
table = pa.table(
{"weekday": pa.array(arr).dictionary_encode()}
)
from pandas.api.interchange import (
from_dataframe as pandas_from_dataframe
)
pandas_df = pandas_from_dataframe(table)
> result = pi.from_dataframe(pandas_df)
arr = ['Mon', 'Tue', 'Mon', 'Wed', 'Mon', 'Thu', 'Fri', 'Sat', None]
pandas_df = weekday
0 Mon
1 Tue
2 Mon
3 Wed
4 Mon
5 Thu
6 Fri
7 Sat
8 NaN
pandas_from_dataframe = <function from_dataframe at 0xebbaa398>
table = pyarrow.Table
weekday: dictionary<values=string, indices=int32, ordered=0>
----
weekday: [ -- dictionary:
["Mon","Tue","Wed","Thu","Fri","Sat"] -- indices:
[0,1,0,2,0,3,4,5,null]]
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/tests/interchange/test_conversion.py:257:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:113: in from_dataframe
return _from_dataframe(df.__dataframe__(allow_copy=allow_copy),
allow_copy = True
df = weekday
0 Mon
1 Tue
2 Mon
3 Wed
4 Mon
5 Thu
6 Fri
7 Sat
8 NaN
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:136: in _from_dataframe
batch = protocol_df_chunk_to_pyarrow(chunk, allow_copy)
allow_copy = True
batches = []
chunk = <pandas.core.interchange.dataframe.PandasDataFrameXchg object at 0xd9e217f0>
df = <pandas.core.interchange.dataframe.PandasDataFrameXchg object at 0xd9e217f0>
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:186: in protocol_df_chunk_to_pyarrow
columns[name] = categorical_column_to_dictionary(col, allow_copy)
allow_copy = True
col = <pandas.core.interchange.column.PandasColumn object at 0xda180550>
columns = {}
df = <pandas.core.interchange.dataframe.PandasDataFrameXchg object at 0xd9e217f0>
dtype = <DtypeKind.CATEGORICAL: 23>
name = 'weekday'
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:293: in categorical_column_to_dictionary
dictionary = column_to_array(cat_column)
allow_copy = True
cat_column = <pandas.core.interchange.column.PandasColumn object at 0xda1801d0>
categorical = {'categories': <pandas.core.interchange.column.PandasColumn object at 0xda1801d0>,
'is_dictionary': True,
'is_ordered': False}
col = <pandas.core.interchange.column.PandasColumn object at 0xda180550>
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:214: in column_to_array
data = buffers_to_array(buffers, data_type,
allow_copy = True
buffers = {'data': (PandasBuffer({'bufsize': 18, 'ptr': 3659006432, 'device': 'CPU'}),
(<DtypeKind.STRING: 21>, 8, 'u', '=')),
'offsets': (PandasBuffer({'bufsize': 56, 'ptr': 1466456352, 'device': 'CPU'}),
(<DtypeKind.INT: 0>, 64, 'l', '=')),
'validity': (PandasBuffer({'bufsize': 6, 'ptr': 1477427216, 'device': 'CPU'}),
(<DtypeKind.BOOL: 20>, 8, 'b', '='))}
col = <pandas.core.interchange.column.PandasColumn object at 0xda1801d0>
data_type = (<DtypeKind.STRING: 21>, 8, 'u', '=')
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:396: in buffers_to_array
data_pa_buffer = pa.foreign_buffer(data_buff.ptr, data_buff.bufsize,
_ = (<DtypeKind.STRING: 21>, 8, 'u', '=')
allow_copy = True
buffers = {'data': (PandasBuffer({'bufsize': 18, 'ptr': 3659006432, 'device': 'CPU'}),
(<DtypeKind.STRING: 21>, 8, 'u', '=')),
'offsets': (PandasBuffer({'bufsize': 56, 'ptr': 1466456352, 'device': 'CPU'}),
(<DtypeKind.INT: 0>, 64, 'l', '=')),
'validity': (PandasBuffer({'bufsize': 6, 'ptr': 1477427216, 'device': 'CPU'}),
(<DtypeKind.BOOL: 20>, 8, 'b', '='))}
data_buff = PandasBuffer({'bufsize': 18, 'ptr': 3659006432, 'device': 'CPU'})
data_type = (<DtypeKind.STRING: 21>, 8, 'u', '=')
describe_null = (<ColumnNullType.USE_BYTEMASK: 4>, 0)
length = 6
offset = 0
offset_buff = PandasBuffer({'bufsize': 56, 'ptr': 1466456352, 'device': 'CPU'})
offset_dtype = (<DtypeKind.INT: 0>, 64, 'l', '=')
validity_buff = PandasBuffer({'bufsize': 6, 'ptr': 1477427216, 'device': 'CPU'})
validity_dtype = (<DtypeKind.BOOL: 20>, 8, 'b', '=')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E OverflowError: Python int too large to convert to C ssize_t
pyarrow/io.pxi:1990: OverflowError
________________________________________________________ test_empty_dataframe _________________________________________________________
def test_empty_dataframe():
schema = pa.schema([('col1', pa.int8())])
df = pa.table([[]], schema=schema)
dfi = df.__dataframe__()
> assert pi.from_dataframe(dfi) == df
df = pyarrow.Table
col1: int8
----
col1: [[]]
dfi = <pyarrow.interchange.dataframe._PyArrowDataFrame object at 0xd98381d0>
schema = col1: int8
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/tests/interchange/test_conversion.py:522:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:113: in from_dataframe
return _from_dataframe(df.__dataframe__(allow_copy=allow_copy),
allow_copy = True
df = <pyarrow.interchange.dataframe._PyArrowDataFrame object at 0xd98381d0>
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:140: in _from_dataframe
batch = protocol_df_chunk_to_pyarrow(df)
allow_copy = True
batches = []
df = <pyarrow.interchange.dataframe._PyArrowDataFrame object at 0xd96e41b0>
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:182: in protocol_df_chunk_to_pyarrow
columns[name] = column_to_array(col, allow_copy)
allow_copy = True
col = <pyarrow.interchange.column._PyArrowColumn object at 0xd96a6650>
columns = {}
df = <pyarrow.interchange.dataframe._PyArrowDataFrame object at 0xd96e41b0>
dtype = <DtypeKind.INT: 0>
name = 'col1'
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:214: in column_to_array
data = buffers_to_array(buffers, data_type,
allow_copy = True
buffers = {'data': (PyArrowBuffer({'bufsize': 0, 'ptr': 4122363392, 'device': 'CPU'}),
(<DtypeKind.INT: 0>, 8, 'c', '=')),
'offsets': None,
'validity': None}
col = <pyarrow.interchange.column._PyArrowColumn object at 0xd96a6650>
data_type = (<DtypeKind.INT: 0>, 8, 'c', '=')
../work/apache-arrow-15.0.0/python-python3_11/install/usr/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:396: in buffers_to_array
data_pa_buffer = pa.foreign_buffer(data_buff.ptr, data_buff.bufsize,
_ = (<DtypeKind.INT: 0>, 8, 'c', '=')
allow_copy = True
buffers = {'data': (PyArrowBuffer({'bufsize': 0, 'ptr': 4122363392, 'device': 'CPU'}),
(<DtypeKind.INT: 0>, 8, 'c', '=')),
'offsets': None,
'validity': None}
data_buff = PyArrowBuffer({'bufsize': 0, 'ptr': 4122363392, 'device': 'CPU'})
data_type = (<DtypeKind.INT: 0>, 8, 'c', '=')
describe_null = (<ColumnNullType.NON_NULLABLE: 0>, None)
length = 0
offset = 0
offset_buff = None
validity_buff = None
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E OverflowError: Python int too large to convert to C ssize_t
pyarrow/io.pxi:1990: OverflowError
```
</details>
Full build & test log (2.5M): [pyarrow.txt](https://github.com/apache/arrow/files/14343715/pyarrow.txt)
This is arrow 15.0.0 on Gentoo, with x86 systemd-nspawn container. I've used `-O2 -march=pentium-m -mfpmath=sse -pipe` as compiler flags, to rule out i387-specific issues.
```
>>> pyarrow.show_info()
pyarrow version info
--------------------
Package kind : not indicated
Arrow C++ library version : 15.0.0
Arrow C++ compiler : GNU 13.2.1
Arrow C++ compiler flags : -O2 -march=pentium-m -mfpmath=sse -pipe
Arrow C++ git revision :
Arrow C++ git description :
Arrow C++ build type : relwithdebinfo
Platform:
OS / Arch : Linux x86_64
SIMD Level : avx2
Detected SIMD Level : avx2
Memory:
Default backend : system
Bytes allocated : 0 bytes
Max memory : 0 bytes
Supported Backends : system
Optional modules:
csv : Enabled
cuda : -
dataset : Enabled
feather : Enabled
flight : -
fs : Enabled
gandiva : -
json : Enabled
orc : -
parquet : Enabled
Filesystems:
GcsFileSystem : -
HadoopFileSystem : Enabled
S3FileSystem : -
Compression Codecs:
brotli : Enabled
bz2 : Enabled
gzip : Enabled
lz4_frame : Enabled
lz4 : Enabled
snappy : Enabled
zstd : Enabled
```
Some of these might be problems inside pandas. I'm going to file a bug about the test failures there in a minute, and link it here afterwards.
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Test failures on 32-bit x86 [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou closed issue #40153: [Python] Test failures on 32-bit x86
URL: https://github.com/apache/arrow/issues/40153
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Test failures on 32-bit x86 [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #40153:
URL: https://github.com/apache/arrow/issues/40153#issuecomment-1971449220
Taken together, PRs #40293, #40294 and #40295 should fix the remaining failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Test failures on 32-bit x86 [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #40153:
URL: https://github.com/apache/arrow/issues/40153#issuecomment-1954544737
Also cc @jorisvandenbossche
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Test failures on 32-bit x86 [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #40153:
URL: https://github.com/apache/arrow/issues/40153#issuecomment-1956174802
Oops, sorry, I didn't mean to close it. Let's reopen it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Test failures on 32-bit x86 [arrow]
Posted by "thesamesam (via GitHub)" <gi...@apache.org>.
thesamesam commented on issue #40153:
URL: https://github.com/apache/arrow/issues/40153#issuecomment-1971597380
Thank you for putting the time into this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Test failures on 32-bit x86 [arrow]
Posted by "mgorny (via GitHub)" <gi...@apache.org>.
mgorny commented on issue #40153:
URL: https://github.com/apache/arrow/issues/40153#issuecomment-1954056392
pandas counterpart: https://github.com/pandas-dev/pandas/issues/57523
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Test failures on 32-bit x86 [arrow]
Posted by "mgorny (via GitHub)" <gi...@apache.org>.
mgorny commented on issue #40153:
URL: https://github.com/apache/arrow/issues/40153#issuecomment-1954567780
Looking at the other failures:
> FAILED tests/test_array.py::test_dictionary_to_numpy - TypeError: Cannot cast array data from dtype('int64') to dtype('int32') according to the rule 'safe'
I haven't figured this out yet. It might be a bug in pandas. FWICS we're constructing two arrays, and they end up having different types inside.
> FAILED tests/test_io.py::test_python_file_large_seeks - assert 5 == ((2 ** 32) + 5)
> FAILED tests/test_io.py::test_memory_map_large_seeks - OSError: Read out of bounds (offset = 4294967301, size = 5) in file of size 10
This could be a bug in pyarrow (or arrow itself) — either it didn't build with Large File Support, or it doesn't use correct types somewhere in between.
> FAILED tests/test_pandas.py::TestConvertStructTypes::test_from_numpy_nested - AssertionError: assert 8 == 12
> FAILED tests/test_schema.py::test_schema_sizeof - assert 28 > 30
I think the first one is because `np.object_` is 32-bit rather than 64-bit, but I'm not sure about the second one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Test failures on 32-bit x86 [arrow]
Posted by "mgorny (via GitHub)" <gi...@apache.org>.
mgorny commented on issue #40153:
URL: https://github.com/apache/arrow/issues/40153#issuecomment-1954364444
I can try but I can't promise I'll come up with anything that makes sense. I have almost no knowledge of numpy, and none of arrow/pandas, so it'll be all guesswork.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Test failures on 32-bit x86 [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #40153:
URL: https://github.com/apache/arrow/issues/40153#issuecomment-1978669663
Issue resolved by pull request 40293
https://github.com/apache/arrow/pull/40293
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Test failures on 32-bit x86 [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou closed issue #40153: [Python] Test failures on 32-bit x86
URL: https://github.com/apache/arrow/issues/40153
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Test failures on 32-bit x86 [arrow]
Posted by "mgorny (via GitHub)" <gi...@apache.org>.
mgorny commented on issue #40153:
URL: https://github.com/apache/arrow/issues/40153#issuecomment-1955828200
Should we reopen the bug for the remaining issues or should I file a new one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Test failures on 32-bit x86 [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #40153:
URL: https://github.com/apache/arrow/issues/40153#issuecomment-1956486527
Among the remaining failures, `test_dictionary_to_numpy` should probably be fixed in the test itself, though I'm not sure.
The other failures (the gdb and tensor ones) probably need fixing the code under test, so you don't necessarily have to tackle them (though of course you can if you want!).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Test failures on 32-bit x86 [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #40153:
URL: https://github.com/apache/arrow/issues/40153#issuecomment-1956747712
> Among the remaining failures, `test_dictionary_to_numpy` should probably be fixed in the test itself, though I'm not sure.
Looking at the traceback, it seems this is actually failing in the code itself, here:
https://github.com/apache/arrow/blob/b51c318122ff5db1c6c4c70a69b0c804f9e31704/python/pyarrow/array.pxi#L1588-L1589
We do a take with int64 indices, and for some reason that seems to fail inside numpy's internals when trying to cast the int64 indices to int32 (because it's not "safe" casting).
The numpy `take` docs don't say anything about a limitation of the indices array to int32 on 32bit platforms, so not sure if that's something we should report to them (or if we need to cast to int32 up front when needed).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Test failures on 32-bit x86 [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #40153:
URL: https://github.com/apache/arrow/issues/40153#issuecomment-1956849421
> The numpy `take` docs don't say anything about a limitation of the indices array to int32 on 32bit platforms, so not sure if that's something we should report to them (or if we need to cast to int32 up front when needed).
We should report it to them IMHO, but we should also make sure we don't produce indices that are too large for Numpy (it seems this is done in `CategoricalWriter`?).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Test failures on 32-bit x86 [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou closed issue #40153: [Python] Test failures on 32-bit x86
URL: https://github.com/apache/arrow/issues/40153
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Test failures on 32-bit x86 [arrow]
Posted by "mgorny (via GitHub)" <gi...@apache.org>.
mgorny commented on issue #40153:
URL: https://github.com/apache/arrow/issues/40153#issuecomment-1957047782
By the way, I've gotten a reply that the two cases of `ValueError: putmask: output array is read-only` in https://github.com/pandas-dev/pandas/issues/57523 "could potentially be related to bugs in Arrow".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Test failures on 32-bit x86 [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #40153:
URL: https://github.com/apache/arrow/issues/40153#issuecomment-1956289987
I've opened https://github.com/apache/arrow/pull/40176 for the large file-related failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Test failures on 32-bit x86 [arrow]
Posted by "mgorny (via GitHub)" <gi...@apache.org>.
mgorny commented on issue #40153:
URL: https://github.com/apache/arrow/issues/40153#issuecomment-1971575795
Thanks a lot! I can confirm that they fix the remaining issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Test failures on 32-bit x86 [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #40153:
URL: https://github.com/apache/arrow/issues/40153#issuecomment-1954159546
Perhaps you want to submit a PR for this? From what I can tell, this should be mostly a matter of fixing the tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org