You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "joemarshall (via GitHub)" <gi...@apache.org> on 2023/09/21 14:51:38 UTC
[GitHub] [arrow] joemarshall opened a new pull request, #37822: GH23221 - python changes for pyodide build
joemarshall opened a new pull request, #37822:
URL: https://github.com/apache/arrow/pull/37822
1) pyarrow knows about ARROW_ENABLE_THREADING and doesn't use threads if they are not enabled in libarrow.
2) pyarrow sets defaults for inclusion of submodules based on their inclusion in the arrow build. e.g. pyarrow.parquet is built only if ARROW_PARQUET is set. This makes it possible to build in situations where you don't have access to set the build environment variables (e.g. in cross compiling situations like pyodide). It also seems to me to make sense - I can't see why you'd want to build pyarrow without parquet if you've built arrow with it, and the inverse is obviously not possible.
Split from #37696
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2039474876
@ianmcook #37821 is in ππΊπͺ©!'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2141394234
@kou rebased now. python tests pass now also.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1586242110
##########
python/setup.py:
##########
@@ -133,8 +143,68 @@ def run(self):
'bundle the Arrow C++ headers')] +
_build_ext.user_options)
+ def get_arrow_build_options(self):
Review Comment:
> You can use https://github.com/apache/arrow/pull/37822/files#r1344887227 for this approach.
As @joemarshall mentioned, right now that is not directly possible because we also use those variables from within `setup.py` for other reasons.
However, I have been looking a bit into that, and to me it seems this is only used for the "failure_permitted" logic, which I am not sure if that is something we necessarily need to keep.
So if we remove that logic, then we can indeed move the environment variable handling fully to the CMakeLists file, making this a lot simpler. Given this is already a big PR, I think this is something we can discuss and review separately, so I opened https://github.com/apache/arrow/issues/41480 for this.
(@joemarshall I am also happy to take this on if you want, and ensure to move that forward so this doesn't need to block this PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1636301971
##########
python/setup.py:
##########
@@ -40,6 +40,12 @@
# Check if we're running 64-bit Python
is_64_bit = sys.maxsize > 2**32
+is_emscripten = (
+ sysconfig.get_config_var("SOABI")
+ and sysconfig.get_config_var("SOABI").find("emscripten") != -1
+)
Review Comment:
OK, thanks for the explanation. Can you add something like that as a comment here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1651063494
##########
python/pyarrow/conftest.py:
##########
@@ -55,6 +58,7 @@
'sockets',
'substrait',
'threading',
+ 'timezone_data'
Review Comment:
```suggestion
'timezone_data',
```
##########
python/pyarrow/conftest.py:
##########
@@ -55,6 +58,7 @@
'sockets',
'substrait',
'threading',
+ 'timezone_data'
Review Comment:
```suggestion
'timezone_data',
```
This should fix the CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1639997184
##########
python/pyarrow/tests/test_compute.py:
##########
@@ -2481,6 +2493,10 @@ def _check_temporal_rounding(ts, values, unit):
@pytest.mark.skipif(sys.platform == "win32" and not util.windows_has_tzdata(),
reason="Timezone database is not installed on Windows")
+@pytest.mark.xfail(
+ sys.platform == "emscripten",
+ reason="Emscripten datetime is implemented in Javascript and works differently"
+)
Review Comment:
Ah, good catch. I should have thought about this, because we have discussed related issues in the past. We explicitly don't use the `tzdata` package because of the compatibility issue. Although the discussion at https://github.com/apache/arrow/issues/31472 / https://github.com/apache/arrow/issues/28868#issuecomment-1378032344 is specifically for Windows (where getting the tzdata through a package would be especially useful, given we now require users to download the text database manually), and from re-reading that it seems that the binary format not being supported is only a Windows issue, and for linux it should be fine. But generally for linux the system provides the tz data, so we don't actually test that with the `tzdata` package. I am wondering if we would have similar problems as you encountered here for Linux with `tzdata` .. (given the `slim` / `fat` difference)
> I'm inclined to bundle pre-built timezones into pyarrow build as resources?
I am not entirely sure this would actually be possible.
For Windows, we do provide a helper function to both set the path to the tz database manually to a custom path (`pyarrow.set_timezone_db_path`), and then also a helper to download the data (`pyarrow.util.download_tzdata_on_windows`).
But that requires you to be able to point to a custom path, and right now that is only possible on Windows AFAIK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2020994422
@joemarshall is this blocking https://github.com/pyodide/pyodide/issues/2933?
Could you please reply to comments, rebase, etc? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1631252322
##########
python/pyarrow/lib.pyx:
##########
@@ -79,6 +79,15 @@ def set_cpu_count(int count):
check_status(SetCpuThreadPoolCapacity(count))
+def is_threading_enabled() -> bool:
+ """
+ Returns true if threading is enabled in libarrow. If it isn't enabled, then python shouldn't
Review Comment:
```suggestion
Return True if threading is enabled in pyarrow.
If it isn't enabled, then python shouldn't
```
And a styling nitpick, but can you limit the line length a bit when you reflow the second paragraph?
##########
python/setup.py:
##########
@@ -40,6 +40,12 @@
# Check if we're running 64-bit Python
is_64_bit = sys.maxsize > 2**32
+is_emscripten = (
+ sysconfig.get_config_var("SOABI")
+ and sysconfig.get_config_var("SOABI").find("emscripten") != -1
+)
Review Comment:
Is there a specific reason we use this check here, but `sys.platform == 'emscripten':` in other places?
##########
python/scripts/run_emscripten_tests.py:
##########
Review Comment:
I haven't looked in detail at this file, but so certainly would second your comment
> 2\. The script (in `python/scripts` which runs the tests is non-trivial; at some point it would be worth upstreaming this code to somewhere pyodide related, but I can't work out where right now. I'm talking to pyodide people about it, but for now it needs to live in arrow still.
Just a question about this: is there a reason this is more complicated for us? (I see that eg numpy is just calling pytest from the github action workflow)
##########
python/pyarrow/io.pxi:
##########
@@ -788,11 +847,28 @@ cdef class NativeFile(_Weakrefable):
write_queue.put_nowait(buf)
finally:
done = True
-
writer_thread.join()
if exc_info is not None:
raise exc_info[0], exc_info[1], exc_info[2]
+ def _upload_nothreads(self, stream, buffer_size=None):
+ """
+ Internal method to do an upload without separate threads, queues etc.
+ Called by upload above if is_threading_enabled() == False
Review Comment:
```suggestion
Internal method to do an upload without separate threads, queues etc.
Called by upload above if is_threading_enabled() == False
```
##########
python/pyarrow/conftest.py:
##########
@@ -76,14 +80,21 @@
'pandas': False,
'parquet': False,
'parquet_encryption': False,
+ 'processes': True,
'requires_testing_data': True,
's3': False,
'slow': False,
'snappy': Codec.is_available('snappy'),
'substrait': False,
+ 'threading': is_threading_enabled(),
'zstd': Codec.is_available('zstd'),
}
+if sys.platform == "emscripten":
+ # emscripten doesn't support multiprocessing or gdb
Review Comment:
```suggestion
# emscripten doesn't support multiprocessing/subprocess or gdb
```
From seeing the tests that are marked with `processes`, it's more the use of `subprocess` module than `multiprocessing`? (which in practice has the same effect of spawning multiple processes, but I found the mention of multiprocessing (thinking about the stdlib module) here confusing for a moment)
##########
python/pyarrow/error.pxi:
##########
@@ -217,7 +218,10 @@ cdef class SignalStopHandler:
maybe_source.status().Warn()
else:
self._stop_token.init(deref(maybe_source).token())
- self._enabled = True
+ if not is_threading_enabled():
+ self._enabled = False
Review Comment:
The signal handler (and so ctrl-c?) just doesn't work on emscripten, or this is actually threading related?
(potentially dumb question, I am not familiar with this part of our code base; this might be obvious, but it could maybe also use a clarifying code comment)
##########
python/pyarrow/_emscripten_timezones.py:
##########
@@ -0,0 +1,48 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from importlib import resources
+from pathlib import Path
+from shutil import copytree
+
+try:
+ import js
+
+ def setup_emscripten_timezone_database():
+ """Setup the timezone database for use on browser based
+ emscripten environments.
+ """
+ if not Path("/usr/share/zoneinfo").exists():
+ Path("/usr/share/").mkdir(parents=True, exist_ok=True)
+ try:
+ tzpath = resources.files("tzdata").joinpath("zoneinfo")
+ copytree(tzpath, "/usr/share/zoneinfo", dirs_exist_ok=True)
+ localtime_path = Path("/etc/localtime")
+ if not localtime_path.exists():
+ # get local timezone from browser js object
+ timezone = js.Intl.DateTimeFormat().resolvedOptions().timeZone
+ if timezone and str(timezone) != "":
+ timezone = str(timezone)
+ # make symbolic link to local time
+ Path("/etc/").mkdir(parents=True, exist_ok=True)
+ localtime_path.symlink_to(tzpath / timezone)
+ except (ImportError, IOError):
+ print("Arrow couldn't install timezone db to /usr/share/zoneinfo")
Review Comment:
Should we maybe use a warning.warn instead of print?
##########
python/pyarrow/io.pxi:
##########
@@ -733,11 +734,66 @@ cdef class NativeFile(_Weakrefable):
finally:
free(buf)
done = True
-
writer_thread.join()
if exc_info is not None:
raise exc_info[0], exc_info[1], exc_info[2]
+ def _download_nothreads(self, stream_or_path, buffer_size=None):
+ """
+ Internal method to do a download without separate threads, queues etc.
+ Called by download above if is_threading_enabled() == False
Review Comment:
```suggestion
Internal method to do a download without separate threads, queues etc.
Called by download above if is_threading_enabled() == False
```
##########
python/pyarrow/tests/test_compute.py:
##########
@@ -2481,6 +2493,10 @@ def _check_temporal_rounding(ts, values, unit):
@pytest.mark.skipif(sys.platform == "win32" and not util.windows_has_tzdata(),
reason="Timezone database is not installed on Windows")
+@pytest.mark.xfail(
+ sys.platform == "emscripten",
+ reason="Emscripten datetime is implemented in Javascript and works differently"
+)
Review Comment:
Out of curiosity, do you remember what exactly was failing here? Because this test is rounding timestamps, which I would think is mostly implemented by our own code and not relying on something platform specific
##########
python/pyarrow/tests/test_memory.py:
##########
@@ -118,6 +119,7 @@ def test_release_unused():
pool.release_unused()
+@pytest.mark.processes
Review Comment:
You added the file top-level mark, so those test specific ones can be removed?
##########
python/pyarrow/tests/test_array.py:
##########
@@ -34,6 +34,9 @@
from pyarrow.vendored.version import Version
+@pytest.mark.skipif(
+ sys.platform == "emscripten", reason="Emscripten can't run processes"
+)
Review Comment:
You added a `processes` marker I saw in contest.py. Could/should that be used here instead?
##########
ci/docker/conda-python-emscripten.dockerfile:
##########
Review Comment:
You are adding this docker file, but where is it used? Because the changes in docker-compose.yml (still) seem to be based on an ubuntu image?
##########
python/pyarrow/tests/test_io.py:
##########
@@ -1073,6 +1073,8 @@ def _try_delete(path):
def test_memory_map_writer(tmpdir):
+ if sys.platform == "emscripten":
+ pytest.xfail("Multiple memory maps to same file don't work on emscripten")
Review Comment:
Use a marker like in other places?
##########
python/pyarrow/tests/test_io.py:
##########
@@ -1073,6 +1073,8 @@ def _try_delete(path):
def test_memory_map_writer(tmpdir):
+ if sys.platform == "emscripten":
+ pytest.xfail("Multiple memory maps to same file don't work on emscripten")
Review Comment:
And can maybe also be a skip instead of xfail (or are there prospects that this would someday work?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #37822: GH-23221: [Python] python changes for pyodide build
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-1733622941
@joemarshall thanks for the PR!
We might want to expose `is_threading_enabled()` in pyarrow publicly (it might be useful for downstream packages as well?), in `__init__.py`
> 2\. pyarrow sets defaults for inclusion of submodules based on their inclusion in the arrow build. e.g. pyarrow.parquet is built only if ARROW_PARQUET is set. This makes it possible to build in situations where you don't have access to set the build environment variables (e.g. in cross compiling situations like pyodide).
This part is not actually included here? (or I don't understand the sentence)
(and it might also make sense to leave that for a third PR since it is changing the build process beyond emscriptem?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1633262513
##########
python/pyarrow/error.pxi:
##########
@@ -217,7 +218,10 @@ cdef class SignalStopHandler:
maybe_source.status().Warn()
else:
self._stop_token.init(deref(maybe_source).token())
- self._enabled = True
+ if not is_threading_enabled():
+ self._enabled = False
Review Comment:
The tests for this handling all rely on threads also.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2191514463
Revision: 135ecf666fe716e238756fb77ce242fb528a9c49
Submitted crossbow builds: [ursacomputing/crossbow @ actions-569b73f722](https://github.com/ursacomputing/crossbow/branches/all?query=actions-569b73f722)
|Task|Status|
|----|------|
|test-conda-python-emscripten|[](https://github.com/ursacomputing/crossbow/actions/runs/9679189658/job/26704639176)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2211065815
Revision: 71a2f6ae009c25dae91309e1a7079037375d0c45
Submitted crossbow builds: [ursacomputing/crossbow @ actions-8f6301a1d5](https://github.com/ursacomputing/crossbow/branches/all?query=actions-8f6301a1d5)
|Task|Status|
|----|------|
|wheel-macos-big-sur-cp310-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508555/job/27090789375)|
|wheel-macos-big-sur-cp311-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508742/job/27090790905)|
|wheel-macos-big-sur-cp312-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508529/job/27090789692)|
|wheel-macos-big-sur-cp38-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508845/job/27090791177)|
|wheel-macos-big-sur-cp39-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810509066/job/27090792527)|
|wheel-macos-catalina-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810509050/job/27090792546)|
|wheel-macos-catalina-cp311-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508761/job/27090791171)|
|wheel-macos-catalina-cp312-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508653/job/27090790391)|
|wheel-macos-catalina-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508528/job/27090789383)|
|wheel-macos-catalina-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810509056/job/27090792532)|
|wheel-manylinux-2-28-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508743/job/27090790688)|
|wheel-manylinux-2-28-cp310-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508550/job/27090789949)|
|wheel-manylinux-2-28-cp311-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508897/job/27090791520)|
|wheel-manylinux-2-28-cp311-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508794/job/27090791150)|
|wheel-manylinux-2-28-cp312-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508771/job/27090790914)|
|wheel-manylinux-2-28-cp312-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508931/job/27090791706)|
|wheel-manylinux-2-28-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508912/job/27090791655)|
|wheel-manylinux-2-28-cp38-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508894/job/27090791537)|
|wheel-manylinux-2-28-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508657/job/27090790387)|
|wheel-manylinux-2-28-cp39-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508911/job/27090791544)|
|wheel-manylinux-2014-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508504/job/27090789578)|
|wheel-manylinux-2014-cp310-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508552/job/27090789716)|
|wheel-manylinux-2014-cp311-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508691/job/27090790747)|
|wheel-manylinux-2014-cp311-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508646/job/27090790405)|
|wheel-manylinux-2014-cp312-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508909/job/27090791514)|
|wheel-manylinux-2014-cp312-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508920/job/27090791689)|
|wheel-manylinux-2014-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508519/job/27090789388)|
|wheel-manylinux-2014-cp38-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508662/job/27090790674)|
|wheel-manylinux-2014-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508934/job/27090791675)|
|wheel-manylinux-2014-cp39-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508512/job/27090789395)|
|wheel-windows-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508600/job/27090790122)|
|wheel-windows-cp311-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508946/job/27090791841)|
|wheel-windows-cp312-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508521/job/27090789588)|
|wheel-windows-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508839/job/27090790883)|
|wheel-windows-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9810508694/job/27090790793)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2211057567
> @joemarshall could you please sync your fork with upstream main? I think that will resolve the CI failures.
Try now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1554370483
##########
python/setup.py:
##########
@@ -133,8 +143,68 @@ def run(self):
'bundle the Arrow C++ headers')] +
_build_ext.user_options)
+ def get_arrow_build_options(self):
Review Comment:
@pitrou finally getting round to messing with this now c++ is in - I discover that there is actually an official way to query arrow options as JSON. I'll look at that next.
The reason for needing this for pyodide is that pyodide builds everything in an isolated environment which loses the environment variables that are passed in to set these things up. It's either that or make a file to load settings in, but even if one supported a settings file it would be good to default to building a pyarrow with the same options as the linked in libarrow.
@kou - I think it has to be done before the cmake build is run because the value of the options alters some things in setup.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2024007561
> @joemarshall is this blocking https://github.com/pyodide/pyodide/issues/2933?
>
> Could you please reply to comments, rebase, etc? Thanks
Like @kou says, this one is reliant on the c++ library changes for emscripten, so is on hold until that is merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "cpcloud (via GitHub)" <gi...@apache.org>.
cpcloud commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2066916483
Nice ππ» that makes the import work, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] Add support for Pyodide [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1622131589
##########
python/pyarrow/_dataset_parquet.pyx:
##########
@@ -737,6 +737,8 @@ cdef class ParquetFragmentScanOptions(FragmentScanOptions):
new CParquetFragmentScanOptions()))
self.use_buffered_stream = use_buffered_stream
self.buffer_size = buffer_size
+ if pre_buffer and not is_threading_enabled():
+ pre_buffer=False
Review Comment:
@jorisvandenbossche Is there a linter that can detect this style mismatch?
```suggestion
pre_buffer = False
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1620107963
##########
python/setup.py:
##########
@@ -133,8 +143,68 @@ def run(self):
'bundle the Arrow C++ headers')] +
_build_ext.user_options)
+ def get_arrow_build_options(self):
Review Comment:
FWIW this has been merged in the meantime, so you should be able to simplify this again now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1344275196
##########
python/setup.py:
##########
@@ -133,8 +143,68 @@ def run(self):
'bundle the Arrow C++ headers')] +
_build_ext.user_options)
+ def get_arrow_build_options(self):
Review Comment:
I have to say I find this solution quite horrible. Enabling or disabling PyArrow components manually is a bit cumbersome, but not terribly so. If there's no better way to query Arrow CMake arguments than parsing some adhoc output format from a hidden CMake option, then I would rather not have this.
@kou Any thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1344268982
##########
python/pyarrow/lib.pyx:
##########
@@ -79,6 +79,10 @@ def set_cpu_count(int count):
check_status(SetCpuThreadPoolCapacity(count))
+def is_threading_enabled():
+ return libarrow_python.IsThreadingEnabled()
Review Comment:
Need a docstring here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] joemarshall commented on pull request #37822: GH23221 - python changes for pyodide build
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-1729748852
@kou And these are the python changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2142397513
@kou I've gone through those comments I think - there's two open discussions - one about what platforms to test on, and one about CMAKE_INSTALL_LIBDIR.
Of these - the testing platforms, I think it could go down to just node and chrome. Going down to just node is a possibility, but I think slightly risky, because system calls including files, dates and times etc. are handled differently between node and browser.
I think the INSTALL_LIBDIR is right, because with it disabled, we install into emscripten sysroot, which makes python build work nicely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] Add support for Pyodide [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1622945294
##########
python/pyarrow/_dataset_parquet.pyx:
##########
@@ -737,6 +737,8 @@ cdef class ParquetFragmentScanOptions(FragmentScanOptions):
new CParquetFragmentScanOptions()))
self.use_buffered_stream = use_buffered_stream
self.buffer_size = buffer_size
+ if pre_buffer and not is_threading_enabled():
+ pre_buffer=False
Review Comment:
OK. We specify `--no-pycodestyle` explicitly. It may be related.
I've opened a new issue for it: GH-41908
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1651056439
##########
ci/docker/conda-python-emscripten.dockerfile:
##########
@@ -0,0 +1,83 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+ARG repo
+ARG arch
+ARG python="3.12"
+FROM ${repo}:${arch}-conda-python-${python}
+
+ARG selenium_version="4.15.2"
+ARG pyodide_version="0.26.0"
+ARG emscripten_version="3.1.58"
+ARG chrome_version="latest"
+ARG required_python_min="(3,12)"
+# fail if python version < 3.12
+RUN echo "check PYTHON>=${required_python_min}" && python -c "import sys;sys.exit(0 if sys.version_info>=${required_python_min} else 1)"
+
+# install selenium and pyodide-build and recent python
+
+# needs to be a login shell so ~/.profile is read
+SHELL ["/bin/bash","--login","-c"]
+
+RUN python --version && \
+python -m pip install selenium==${selenium_version} &&\
+ python -m pip install --upgrade --pre pyodide-build==${pyodide_version}
+
+RUN cd ~ && (ls emsdk || git clone https://github.com/emscripten-core/emsdk.git) && \
+ cd emsdk && \
+ ./emsdk install ${emscripten_version} && \
+ ./emsdk activate ${emscripten_version} && \
+ echo "Installed emsdk to:" ~/emsdk
+
+# make sure zlib is cached in the EMSDK folder
+RUN source ~/emsdk/emsdk_env.sh && embuilder --pic build zlib
+
+# install pyodide dist directory to /pyodide
+RUN cd / \
+ && pyodide_dist_url="https://github.com/pyodide/pyodide/releases/download/${pyodide_version}/pyodide-${pyodide_version}.tar.bz2"\
+ && wget ${pyodide_dist_url} -O- |tar -xj
+
+# install chrome for testing browser based runner
+# zip needed for chrome, libpthread stubs installs pthread module to cmake, build-essential makes
+# sure that unix make is available in isolated python environments
+RUN apt-get update && apt-get install -y -q zip libpthread-stubs0-dev build-essential
+
+RUN if [ $chrome_version = "latest" ]; \
+ then CHROME_VERSION_FULL=$(wget --no-verbose -O - "https://googlechromelabs.github.io/chrome-for-testing/LATEST_RELEASE_STABLE"); \
+ else CHROME_VERSION_FULL=$(wget --no-verbose -O - "https://googlechromelabs.github.io/chrome-for-testing/LATEST_RELEASE_${chrome_version}"); \
+ fi \
+ && CHROME_DOWNLOAD_URL="https://dl.google.com/linux/chrome/deb/pool/main/g/google-chrome-stable/google-chrome-stable_${CHROME_VERSION_FULL}-1_amd64.deb" \
+ && CHROMEDRIVER_DOWNLOAD_URL="https://storage.googleapis.com/chrome-for-testing-public/${CHROME_VERSION_FULL}/linux64/chromedriver-linux64.zip" \
+ && wget --no-verbose -O /tmp/google-chrome.deb ${CHROME_DOWNLOAD_URL} \
+ && apt-get update \
+ && apt install -qqy /tmp/google-chrome.deb \
+ && rm -f /tmp/google-chrome.deb \
+ && rm -rf /var/lib/apt/lists/* \
+ && wget --no-verbose -O /tmp/chromedriver-linux64.zip ${CHROMEDRIVER_DOWNLOAD_URL} \
+ && unzip /tmp/chromedriver-linux64.zip -d /opt/ \
+ && rm /tmp/chromedriver-linux64.zip \
+ && ln -fs /opt/chromedriver-linux64/chromedriver /usr/local/bin/chromedriver \
+ && echo "Using Chrome version: $(google-chrome --version)" \
+ && echo "Using Chrome Driver version: $(chromedriver --version)"
+
+
+
+SHELL ["/bin/bash", "--login", "-i", "-c"]
+# install node 18 (needed for async call support)
+RUN curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash -
+RUN nvm install 18
Review Comment:
This should probably be a separate script as well?
##########
ci/docker/conda-python-emscripten.dockerfile:
##########
@@ -0,0 +1,83 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+ARG repo
+ARG arch
+ARG python="3.12"
+FROM ${repo}:${arch}-conda-python-${python}
+
+ARG selenium_version="4.15.2"
+ARG pyodide_version="0.26.0"
+ARG emscripten_version="3.1.58"
+ARG chrome_version="latest"
+ARG required_python_min="(3,12)"
+# fail if python version < 3.12
+RUN echo "check PYTHON>=${required_python_min}" && python -c "import sys;sys.exit(0 if sys.version_info>=${required_python_min} else 1)"
+
+# install selenium and pyodide-build and recent python
+
+# needs to be a login shell so ~/.profile is read
+SHELL ["/bin/bash","--login","-c"]
+
+RUN python --version && \
+python -m pip install selenium==${selenium_version} &&\
+ python -m pip install --upgrade --pre pyodide-build==${pyodide_version}
+
+RUN cd ~ && (ls emsdk || git clone https://github.com/emscripten-core/emsdk.git) && \
+ cd emsdk && \
+ ./emsdk install ${emscripten_version} && \
+ ./emsdk activate ${emscripten_version} && \
+ echo "Installed emsdk to:" ~/emsdk
Review Comment:
Should this be a dedicated install script? Also, why the `cd ~` specifically here?
##########
ci/docker/conda-python-emscripten.dockerfile:
##########
@@ -0,0 +1,83 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+ARG repo
+ARG arch
+ARG python="3.12"
+FROM ${repo}:${arch}-conda-python-${python}
+
+ARG selenium_version="4.15.2"
+ARG pyodide_version="0.26.0"
+ARG emscripten_version="3.1.58"
+ARG chrome_version="latest"
+ARG required_python_min="(3,12)"
+# fail if python version < 3.12
+RUN echo "check PYTHON>=${required_python_min}" && python -c "import sys;sys.exit(0 if sys.version_info>=${required_python_min} else 1)"
+
+# install selenium and pyodide-build and recent python
+
+# needs to be a login shell so ~/.profile is read
+SHELL ["/bin/bash","--login","-c"]
+
+RUN python --version && \
+python -m pip install selenium==${selenium_version} &&\
+ python -m pip install --upgrade --pre pyodide-build==${pyodide_version}
+
+RUN cd ~ && (ls emsdk || git clone https://github.com/emscripten-core/emsdk.git) && \
+ cd emsdk && \
+ ./emsdk install ${emscripten_version} && \
+ ./emsdk activate ${emscripten_version} && \
+ echo "Installed emsdk to:" ~/emsdk
+
+# make sure zlib is cached in the EMSDK folder
+RUN source ~/emsdk/emsdk_env.sh && embuilder --pic build zlib
+
+# install pyodide dist directory to /pyodide
+RUN cd / \
+ && pyodide_dist_url="https://github.com/pyodide/pyodide/releases/download/${pyodide_version}/pyodide-${pyodide_version}.tar.bz2"\
+ && wget ${pyodide_dist_url} -O- |tar -xj
+
+# install chrome for testing browser based runner
+# zip needed for chrome, libpthread stubs installs pthread module to cmake, build-essential makes
+# sure that unix make is available in isolated python environments
+RUN apt-get update && apt-get install -y -q zip libpthread-stubs0-dev build-essential
+
+RUN if [ $chrome_version = "latest" ]; \
Review Comment:
Can we create a script for this instead of having this as a gigantic one-liner in the dockerfile?
##########
docker-compose.yml:
##########
@@ -875,6 +876,46 @@ services:
/arrow/ci/scripts/python_build.sh /arrow /build &&
/arrow/ci/scripts/python_test.sh /arrow"]
+ conda-python-emscripten:
+ # Usage:
+ # docker-compose build conda-python-emscripten
+ # docker-compose run --rm conda-python-emscripten
+ # Parameters:
+ # ARCH: amd64, arm64v8, ...
+ # UBUNTU: 22.04
+ image: ${REPO}:${ARCH}-conda-python-emscripten
+ build:
+ context: .
+ dockerfile: ci/docker/conda-python-emscripten.dockerfile
+ cache_from:
+ - ${REPO}:${ARCH}-conda-python-${PYTHON}
+ args:
+ repo: ${REPO}
+ arch: ${ARCH}
+ clang_tools: ${CLANG_TOOLS}
+ llvm: ${LLVM}
+ emscripten_version: "3.1.58" # must use correct emscripten for pyodide version
+ pyodide_version: "0.26.0"
+ chrome_version: "122"
+ selenium_version: "4.15.2"
+ required_python_min: "(3,12)"
+ python: ${PYTHON}
+ shm_size: *shm-size
+ volumes: *ubuntu-volumes
+ environment:
+ <<: [*common, *ccache, *sccache, *cpp]
+ ARROW_EMSCRIPTEN: "ON"
+ ARROW_DEPENDENCY_SOURCE: "BUNDLED"
+ UBUNTU: "22.04"
+ PYODIDE_VERSION: "0.26.0"
+ ARROW_BUILD_TYPE: "release"
+ ARROW_BUILD_TESTS: "off"
+ command: ["
+ source ~/.nvm/nvm.sh &&
Review Comment:
Shouldn't the `source` command be part of the script(s) below wherever necessary?
##########
python/pyarrow/_dataset_parquet.pyx:
##########
@@ -787,6 +789,8 @@ cdef class ParquetFragmentScanOptions(FragmentScanOptions):
@pre_buffer.setter
def pre_buffer(self, bint pre_buffer):
+ if pre_buffer and not is_threading_enabled():
Review Comment:
Same question here
##########
ci/scripts/python_test_emscripten.sh:
##########
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# run tests against Chrome and node.js as representative
+# WebAssembly platforms (i.e. one browser, one non-browser).
+
+set -ex
+build_dir=${1}/python
+cd ${build_dir}
+
+pyodide_dist_dir=${2}
+
+# note: this assumes that there is only one wheel built into dist
+# (which is true if you build using emscripten_python_build.sh)
Review Comment:
This may turn out false in developers' environment, as opposed to CI. Can we make this stricter and less error-prone?
##########
ci/docker/conda-python-emscripten.dockerfile:
##########
@@ -0,0 +1,83 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+ARG repo
+ARG arch
+ARG python="3.12"
+FROM ${repo}:${arch}-conda-python-${python}
+
+ARG selenium_version="4.15.2"
+ARG pyodide_version="0.26.0"
+ARG emscripten_version="3.1.58"
+ARG chrome_version="latest"
+ARG required_python_min="(3,12)"
+# fail if python version < 3.12
+RUN echo "check PYTHON>=${required_python_min}" && python -c "import sys;sys.exit(0 if sys.version_info>=${required_python_min} else 1)"
+
+# install selenium and pyodide-build and recent python
+
+# needs to be a login shell so ~/.profile is read
+SHELL ["/bin/bash","--login","-c"]
+
+RUN python --version && \
Review Comment:
Why the `python --version`?
##########
python/pyarrow/_dataset_parquet.pyx:
##########
@@ -737,6 +737,8 @@ cdef class ParquetFragmentScanOptions(FragmentScanOptions):
new CParquetFragmentScanOptions()))
self.use_buffered_stream = use_buffered_stream
self.buffer_size = buffer_size
+ if pre_buffer and not is_threading_enabled():
+ pre_buffer = False
Review Comment:
Shouldn't we instead raise an error if the setting is incompatible with emscripten? cc @jorisvandenbossche
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2151092478
@kou those changes are in now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2210815015
@github-actions crossbow submit -g wheel
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2210964634
@joemarshall could you please sync your fork with upstream main? I think that will resolve the CI failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1666524682
##########
python/pyarrow/io.pxi:
##########
@@ -733,11 +734,66 @@ cdef class NativeFile(_Weakrefable):
finally:
free(buf)
done = True
-
writer_thread.join()
if exc_info is not None:
raise exc_info[0], exc_info[1], exc_info[2]
+ def _download_nothreads(self, stream_or_path, buffer_size=None):
+ """
+ Internal method to do a download without separate threads, queues etc.
+ Called by download above if is_threading_enabled() == False
+ """
+ cdef:
+ int64_t bytes_read = 0
+ uint8_t* buf
+
+ handle = self.get_input_stream()
+
+ buffer_size = buffer_size or DEFAULT_BUFFER_SIZE
+
+ if not hasattr(stream_or_path, 'read'):
+ stream = open(stream_or_path, 'wb')
+
+ def cleanup():
+ stream.close()
Review Comment:
called by download line 662
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2206590836
@kou I believe Joris and Antoine are mostly away this week. Joris reviewed this earlier and communicated that he would like this to be merged in time for 17.0.0 if possible. Let's please not block waiting for additional reviews of the Python parts, unless you have concerns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2211640439
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 2de8008748e2dde383047e47580c69b96fccb35c.
There were no benchmark performance regressions. π
The [full Conbench report](https://github.com/apache/arrow/runs/27107257829) has more details. It also includes information about 31 possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1575363633
##########
cpp/src/arrow/acero/CMakeLists.txt:
##########
@@ -173,13 +173,8 @@ add_arrow_acero_test(hash_join_node_test SOURCES hash_join_node_test.cc
bloom_filter_test.cc)
add_arrow_acero_test(pivot_longer_node_test SOURCES pivot_longer_node_test.cc)
-# asof_join_node and sorted_merge_node use std::thread internally
-# and doesn't use ThreadPool so it will
-# be broken if threading is turned off
-if(ARROW_ENABLE_THREADING)
- add_arrow_acero_test(asof_join_node_test SOURCES asof_join_node_test.cc)
- add_arrow_acero_test(sorted_merge_node_test SOURCES sorted_merge_node_test.cc)
-endif()
+add_arrow_acero_test(asof_join_node_test SOURCES asof_join_node_test.cc)
+add_arrow_acero_test(sorted_merge_node_test SOURCES sorted_merge_node_test.cc)
Review Comment:
It seems that this change is #41125. Let's work on #41125 before this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1575363633
##########
cpp/src/arrow/acero/CMakeLists.txt:
##########
@@ -173,13 +173,8 @@ add_arrow_acero_test(hash_join_node_test SOURCES hash_join_node_test.cc
bloom_filter_test.cc)
add_arrow_acero_test(pivot_longer_node_test SOURCES pivot_longer_node_test.cc)
-# asof_join_node and sorted_merge_node use std::thread internally
-# and doesn't use ThreadPool so it will
-# be broken if threading is turned off
-if(ARROW_ENABLE_THREADING)
- add_arrow_acero_test(asof_join_node_test SOURCES asof_join_node_test.cc)
- add_arrow_acero_test(sorted_merge_node_test SOURCES sorted_merge_node_test.cc)
-endif()
+add_arrow_acero_test(asof_join_node_test SOURCES asof_join_node_test.cc)
+add_arrow_acero_test(sorted_merge_node_test SOURCES sorted_merge_node_test.cc)
Review Comment:
It seems that this change is #41125. Let's work on #41125 before this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1619566702
##########
cpp/src/arrow/acero/CMakeLists.txt:
##########
@@ -173,13 +173,8 @@ add_arrow_acero_test(hash_join_node_test SOURCES hash_join_node_test.cc
bloom_filter_test.cc)
add_arrow_acero_test(pivot_longer_node_test SOURCES pivot_longer_node_test.cc)
-# asof_join_node and sorted_merge_node use std::thread internally
-# and doesn't use ThreadPool so it will
-# be broken if threading is turned off
-if(ARROW_ENABLE_THREADING)
- add_arrow_acero_test(asof_join_node_test SOURCES asof_join_node_test.cc)
- add_arrow_acero_test(sorted_merge_node_test SOURCES sorted_merge_node_test.cc)
-endif()
+add_arrow_acero_test(asof_join_node_test SOURCES asof_join_node_test.cc)
+add_arrow_acero_test(sorted_merge_node_test SOURCES sorted_merge_node_test.cc)
Review Comment:
#41125 is in! This branch likely needs a refresh,but all related testing and threading issues seem to be addressed!
Thanks @joemarshall for your continued work here! Home stretch!
@kou can you do an initial review or would you rather wait until Joe does one more pass with upstream changes completed?
##########
cpp/src/arrow/acero/CMakeLists.txt:
##########
@@ -173,13 +173,8 @@ add_arrow_acero_test(hash_join_node_test SOURCES hash_join_node_test.cc
bloom_filter_test.cc)
add_arrow_acero_test(pivot_longer_node_test SOURCES pivot_longer_node_test.cc)
-# asof_join_node and sorted_merge_node use std::thread internally
-# and doesn't use ThreadPool so it will
-# be broken if threading is turned off
-if(ARROW_ENABLE_THREADING)
- add_arrow_acero_test(asof_join_node_test SOURCES asof_join_node_test.cc)
- add_arrow_acero_test(sorted_merge_node_test SOURCES sorted_merge_node_test.cc)
-endif()
+add_arrow_acero_test(asof_join_node_test SOURCES asof_join_node_test.cc)
+add_arrow_acero_test(sorted_merge_node_test SOURCES sorted_merge_node_test.cc)
Review Comment:
> It seems that this change is #41125. Let's work on #41125 before this PR.
#41125 is in! This branch likely needs a refresh,but all related testing and threading issues seem to be addressed!
Thanks @joemarshall for your continued work here! Home stretch!
@kou can you do an initial review or would you rather wait until Joe does one more pass with upstream changes completed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] Add support for Pyodide [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2142976887
> Of these - the testing platforms, I think it could go down to just node and chrome. Going down to just node is a possibility, but I think slightly risky, because system calls including files, dates and times etc. are handled differently between node and browser.
It makes sense. Let's choose the approach.
> I think the INSTALL_LIBDIR is right, because with it disabled, we install into emscripten sysroot, which makes python build work nicely.
It means that we should use `INSTALL_LIBDIR` not `CMAKE_INSTALL_LIBDIR`, right?
I think that it's not related.
Can we specify Emscripten sysroot as `ARROW_HOME` instead of removing `-DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX:-${ARROW_HOME}}`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1631186815
##########
docker-compose.yml:
##########
@@ -679,6 +680,45 @@ services:
UBUNTU:
command: *cpp-command
+ ubuntu-cpp-emscripten-python:
+ # Usage:
+ # docker-compose build ubuntu-cpp-emscripten-python
+ # docker-compose run --rm ubuntu-cpp-emscripten-python
+ # Parameters:
+ # ARCH: amd64, arm64v8, ...
+ # UBUNTU: 22.04
+ image: ${REPO}:${ARCH}-ubuntu-22.04-cpp-emscripten
+ build:
+ context: .
+ dockerfile: ci/docker/ubuntu-22.04-cpp-emscripten.dockerfile
Review Comment:
okay, done the conda way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1633292319
##########
python/pyarrow/tests/test_io.py:
##########
@@ -1073,6 +1073,8 @@ def _try_delete(path):
def test_memory_map_writer(tmpdir):
+ if sys.platform == "emscripten":
+ pytest.xfail("Multiple memory maps to same file don't work on emscripten")
Review Comment:
And I'm not sure if it is worth a marker for a single weirdness
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1640047599
##########
python/pyarrow/tests/test_compute.py:
##########
@@ -2481,6 +2493,10 @@ def _check_temporal_rounding(ts, values, unit):
@pytest.mark.skipif(sys.platform == "win32" and not util.windows_has_tzdata(),
reason="Timezone database is not installed on Windows")
+@pytest.mark.xfail(
+ sys.platform == "emscripten",
+ reason="Emscripten datetime is implemented in Javascript and works differently"
+)
Review Comment:
> because we get our timezone files using the python tzdata package, which builds using the -b slim option, which is not supported yet by HowardHinnant/date
It's still a bit bizarre.
I checked locally on my Linux in a conda env, and the zoneinfo files provided in `/env/share/zoneinfo` ("system" provided) or `env/lib/python3.11/site-packages/tzdata/zoneinfo` (so from tzdata package) are exactly the same. So if `txdata` provides the slim binaries, I don't understand how this is working ..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2167102635
@github-actions crossbow submit test-conda-python-emscripten
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2163389975
@jorisvandenbossche I've added it to tasks.yml, but I'm flying slightly blind on that, so hopefully I've done it right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2186619151
@joemarshall thanks, that sounds good! (I was planning to ask if you were OK leaving out the tz stuff for now, to ensure we would get this in for the 17.0 release freeze next week, but so that's not needed anymore ;))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1575363633
##########
cpp/src/arrow/acero/CMakeLists.txt:
##########
@@ -173,13 +173,8 @@ add_arrow_acero_test(hash_join_node_test SOURCES hash_join_node_test.cc
bloom_filter_test.cc)
add_arrow_acero_test(pivot_longer_node_test SOURCES pivot_longer_node_test.cc)
-# asof_join_node and sorted_merge_node use std::thread internally
-# and doesn't use ThreadPool so it will
-# be broken if threading is turned off
-if(ARROW_ENABLE_THREADING)
- add_arrow_acero_test(asof_join_node_test SOURCES asof_join_node_test.cc)
- add_arrow_acero_test(sorted_merge_node_test SOURCES sorted_merge_node_test.cc)
-endif()
+add_arrow_acero_test(asof_join_node_test SOURCES asof_join_node_test.cc)
+add_arrow_acero_test(sorted_merge_node_test SOURCES sorted_merge_node_test.cc)
Review Comment:
It seems that this is #41125. Let's work on #41125 before this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1574511165
##########
python/CMakeLists.txt:
##########
@@ -590,28 +616,42 @@ endif()
# Acero
if(PYARROW_BUILD_ACERO)
- if(PYARROW_BUNDLE_ARROW_CPP)
- bundle_arrow_lib(${ARROW_ACERO_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
- if(MSVC)
- bundle_arrow_import_lib(${ARROW_ACERO_IMPORT_LIB})
+ if(ARROW_BUILD_SHARED)
+
+ if(PYARROW_BUNDLE_ARROW_CPP)
+ bundle_arrow_lib(${ARROW_ACERO_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
+ if(MSVC)
+ bundle_arrow_import_lib(${ARROW_ACERO_IMPORT_LIB})
+ endif()
endif()
- endif()
- set(ACERO_LINK_LIBS ArrowAcero::arrow_acero_shared)
- list(APPEND CYTHON_EXTENSIONS _acero)
+ set(ACERO_LINK_LIBS ArrowAcero::arrow_acero_shared)
+ list(APPEND CYTHON_EXTENSIONS _acero)
+ else()
+ # ACERO is statically linked into libarrow_python already
+ set(ACERO_LINK_LIBS)
Review Comment:
If we do that, then we have libarrow_acero included in libarrow_python.so and also in _acero.so and bad things happen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #37822: GH23221 - python changes for pyodide build
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-1729746176
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
See also:
* [Other pull requests](https://github.com/apache/arrow/pulls/)
* [Contribution Guidelines - How to contribute patches](https://arrow.apache.org/docs/developers/contributing.html#how-to-contribute-patches)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2191510594
@github-actions crossbow submit test-conda-python-emscripten
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2191658424
Revision: 5fef3a95c85a1a3f27dafc6cedd33ee9f9951412
Submitted crossbow builds: [ursacomputing/crossbow @ actions-7629007087](https://github.com/ursacomputing/crossbow/branches/all?query=actions-7629007087)
|Task|Status|
|----|------|
|test-conda-python-emscripten|[](https://github.com/ursacomputing/crossbow/actions/runs/9680295839/job/26708119611)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2205449975
@jorisvandenbossche @pitrou Do you want to review the Python part?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1344265320
##########
python/CMakeLists.txt:
##########
@@ -68,6 +68,25 @@ if(POLICY CMP0095)
cmake_policy(SET CMP0095 NEW)
endif()
+# this option is used to auto-set defaults for pyarrow build
+option(DUMP_ARROW_ARGUMENTS "Dump the arrow arguments then quit" OFF)
+
+if($ENV{PYODIDE})
+ # these variables are needed for building pyarrow on Emscripten
+ # if they aren't set, cmake cross compiling fails for python
+ # modules (at least under pyodide it does)
+ set(Python3_INCLUDE_DIR $ENV{PYTHONINCLUDE})
+ set(Python3_LIBRARY $ENV{CPYTHONLIB})
+ set(Python3_NumPy_INCLUDE_DIR $ENV{NUMPY_LIB}/core/include)
+ set(Python3_EXECUTABLE)
+ set(ENV{_PYTHON_SYSCONFIGDATA_NAME} $ENV{SYSCONFIG_NAME})
+
+ # remove any c++ standard settings from cflags and let cmake set it.
+ # n.b. this is here because pyodide sets -stdc++=14, which breaks pyarrow builds
+ string(REGEX REPLACE "-std=c\\+\\+[0-9]+" "" CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS})
Review Comment:
@kou Could you take a look at the CMake changes here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1344278839
##########
python/pyarrow/parquet/core.py:
##########
@@ -329,6 +330,9 @@ def __init__(self, source, *, metadata=None, common_metadata=None,
decryption_properties=None, thrift_string_size_limit=None,
thrift_container_size_limit=None, filesystem=None):
+ if pre_buffer and not is_threading_enabled():
+ pre_buffer = False
Review Comment:
I am not sure why we are doing this in many different places. Presumably, this could only be done somewhere at the bottom of the call stack.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1651076210
##########
python/pyarrow/_dataset_parquet.pyx:
##########
@@ -737,6 +737,8 @@ cdef class ParquetFragmentScanOptions(FragmentScanOptions):
new CParquetFragmentScanOptions()))
self.use_buffered_stream = use_buffered_stream
self.buffer_size = buffer_size
+ if pre_buffer and not is_threading_enabled():
+ pre_buffer = False
Review Comment:
One problem is that `pre_buffer` defaults to True in some cases, and we don't want to raise an error by default but we also want to keep that default, I think.
Adding a documentation note that this keyword only has effect when running with threading enabled sounds sufficient to me.
(you can argue the same for the `use_threads` keyword)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2209424099
@joemarshall if you're able to react to Kou's review in the next day or so, then I think we can get this merged in time for the 17.0.0 release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1657171747
##########
ci/docker/conda-python-emscripten.dockerfile:
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+ARG repo
+ARG arch
+ARG python="3.12"
+FROM ${repo}:${arch}-conda-python-${python}
+
+ARG selenium_version="4.15.2"
+ARG pyodide_version="0.26.0"
+ARG emscripten_version="3.1.58"
+ARG chrome_version="latest"
+ARG required_python_min="(3,12)"
+# fail if python version < 3.12
+RUN echo "check PYTHON>=${required_python_min}" && python -c "import sys;sys.exit(0 if sys.version_info>=${required_python_min} else 1)"
+
+# install selenium and pyodide-build and recent python
+
+# needs to be a login shell so ~/.profile is read
+SHELL ["/bin/bash", "--login", "-c", "-o", "pipefail"]
+
+RUN python -m pip install --no-cache-dir selenium==${selenium_version} &&\
+ python -m pip install --no-cache-dir --upgrade --pre pyodide-build==${pyodide_version}
+
+COPY ci/scripts/install_emscripten.sh /arrow/ci/scripts/
+RUN bash /arrow/ci/scripts/install_emscripten.sh ~ "${emscripten_version}"
+
+# make sure zlib is cached in the EMSDK folder
+RUN source ~/emsdk/emsdk_env.sh && embuilder --pic build zlib
+
+# install pyodide dist directory to /pyodide
+WORKDIR /
+RUN pyodide_dist_url="https://github.com/pyodide/pyodide/releases/download/${pyodide_version}/pyodide-${pyodide_version}.tar.bz2" && \
+ wget --progress=dot:giga "${pyodide_dist_url}" -O- |tar -xj
+
+# install chrome for testing browser based runner
+# zip needed for chrome, libpthread stubs installs pthread module to cmake, build-essential makes
+# sure that unix make is available in isolated python environments
+
+# Install basic build stuff, don't pin versions, hence ignore lint
+# hadolint ignore=DL3008
+RUN apt-get update && apt-get install --no-install-recommends -y -q unzip zip libpthread-stubs0-dev build-essential && \
+ apt-get clean && \
+ rm -rf /var/lib/apt/lists/*
+
+COPY ci/scripts/install_chromedriver.sh /arrow/ci/scripts/
+RUN /arrow/ci/scripts/install_chromedriver.sh "${chrome_version}"
+
+SHELL ["/bin/bash", "--login", "-i", "-o", "pipefail", "-c"]
+# install node 18 (needed for async call support)
+COPY ci/scripts/install_node.sh /arrow/ci/scripts/
+RUN /arrow/ci/scripts/install_node.sh 18
Review Comment:
good point
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2191652692
> those test fixes are in
There are still some of the timezone related ones that need the mark (i.e. places where we still use `sys.platform == "win32" and not util.windows_has_tzdata()` to skip a test)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2191653122
@github-actions crossbow submit test-conda-python-emscripten
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2210573984
@ianmcook changes merged in, running tests locally right now, could probably fire off the emscripten tests here to be sure also
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1622204491
##########
ci/scripts/cpp_build.sh:
##########
@@ -105,8 +105,6 @@ if [ "${ARROW_EMSCRIPTEN:-OFF}" = "ON" ]; then
-DCMAKE_C_FLAGS="${CFLAGS:-}" \
-DCMAKE_CXX_FLAGS="${CXXFLAGS:-}" \
-DCMAKE_CXX_STANDARD="${CMAKE_CXX_STANDARD:-17}" \
- -DCMAKE_INSTALL_LIBDIR=${CMAKE_INSTALL_LIBDIR:-lib} \
- -DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX:-${ARROW_HOME}} \
Review Comment:
because we want it to install into emscripten sysroot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2143451349
:warning: GitHub issue #41910 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1633281502
##########
python/pyarrow/tests/test_io.py:
##########
@@ -1073,6 +1073,8 @@ def _try_delete(path):
def test_memory_map_writer(tmpdir):
+ if sys.platform == "emscripten":
+ pytest.xfail("Multiple memory maps to same file don't work on emscripten")
Review Comment:
I was hoping that some day memory mapped files would actually work properly on emscripten. Right now they involve a copy, so multiple maps don't work right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1633265350
##########
docker-compose.yml:
##########
@@ -679,6 +680,45 @@ services:
UBUNTU:
command: *cpp-command
+ ubuntu-cpp-emscripten-python:
Review Comment:
done it hopefully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2168184954
Hopefully fixed crossbow build now, or at least made it not fail in the way it failed last time...
@jorisvandenbossche I made it embed the timezone data in the python wheel rather than downloading it using the tzdata library, because that uses a new timezone format which isn't compatible with the timezone parser used in arrow ( working around this bug: https://github.com/HowardHinnant/date/issues/614 ). This makes the wheel file slightly larger on emscripten, but saves the second download of tzdata which would be required otherwise, so it evens out overall.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2066874678
If you load it with pyodide.loadPackage from JavaScript it should work.
- Or in python call:
```
import pyodide_js
pyodide_js.loadPackage(URL)
```
It's because micropip doesn't preload native libraries in the same way loadPackage does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2021820522
We need to complete #37821 before this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1344268464
##########
python/pyarrow/io.pxi:
##########
@@ -677,16 +676,19 @@ cdef class NativeFile(_Weakrefable):
self.seek(0)
- writer_thread = threading.Thread(target=bg_write)
+ if is_threading_enabled():
+ writer_thread = threading.Thread(target=bg_write)
+ else:
+ writer_thread = None
# This isn't ideal -- PyBytes_FromStringAndSize copies the data from
# the passed buffer, so it's hard for us to avoid doubling the memory
buf = <uint8_t*> malloc(buffer_size)
if buf == NULL:
raise MemoryError("Failed to allocate {0} bytes"
.format(buffer_size))
-
- writer_thread.start()
+ if writer_thread:
Review Comment:
Instead of sprinkling `if writer_thread` conditionals everywhere, I think this would be clearer if there were two separate implementations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1344279839
##########
python/pyarrow/pandas_compat.py:
##########
@@ -608,7 +608,7 @@ def _can_definitely_zero_copy(arr):
arr.flags.contiguous and
issubclass(arr.dtype.type, np.integer))
- if nthreads == 1:
+ if nthreads == 1 or not is_threading_enabled():
Review Comment:
Instead, why not fix the `nthreads` initialization above?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1345504032
##########
python/pyarrow/parquet/core.py:
##########
@@ -329,6 +330,9 @@ def __init__(self, source, *, metadata=None, common_metadata=None,
decryption_properties=None, thrift_string_size_limit=None,
thrift_container_size_limit=None, filesystem=None):
+ if pre_buffer and not is_threading_enabled():
+ pre_buffer = False
Review Comment:
This one can indeed be removed, because it is passed to `ParquetReader.open()` from `_parquet.pyx`, where you added a similar check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2191508427
> The compile bug with orc is a one line fix in thirdpartytoolchain.cpp
Ah, if that's an easy fix, you can certainly do that and re-enable ORC
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1619566702
##########
cpp/src/arrow/acero/CMakeLists.txt:
##########
@@ -173,13 +173,8 @@ add_arrow_acero_test(hash_join_node_test SOURCES hash_join_node_test.cc
bloom_filter_test.cc)
add_arrow_acero_test(pivot_longer_node_test SOURCES pivot_longer_node_test.cc)
-# asof_join_node and sorted_merge_node use std::thread internally
-# and doesn't use ThreadPool so it will
-# be broken if threading is turned off
-if(ARROW_ENABLE_THREADING)
- add_arrow_acero_test(asof_join_node_test SOURCES asof_join_node_test.cc)
- add_arrow_acero_test(sorted_merge_node_test SOURCES sorted_merge_node_test.cc)
-endif()
+add_arrow_acero_test(asof_join_node_test SOURCES asof_join_node_test.cc)
+add_arrow_acero_test(sorted_merge_node_test SOURCES sorted_merge_node_test.cc)
Review Comment:
#41125 is in! This branch likely needs a refresh,but all related testing and threading issues seem to be addressed!
Thanks @joemarshall for your continued work here! Home stretch!
@kou can you do an initial review or would you rather wait until Joe does one more pass with upstream changes completed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] Add support for Pyodide [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1622374286
##########
ci/docker/ubuntu-22.04-cpp-emscripten.dockerfile:
##########
Review Comment:
Other related questions: do we "need" to test this in all those different ways (node + different browsers)?
Do we expect there might be code issues on our side that would only be seen in one of those testing platforms? Because if not testing on one might be sufficient? (for example, my understanding from looking at the CI config is that numpy is only testing with node: https://github.com/numpy/numpy/blob/main/.github/workflows/emscripten.yml)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2210646485
Tests pass here, so hopefully we're ready to merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1666524682
##########
python/pyarrow/io.pxi:
##########
@@ -733,11 +734,66 @@ cdef class NativeFile(_Weakrefable):
finally:
free(buf)
done = True
-
writer_thread.join()
if exc_info is not None:
raise exc_info[0], exc_info[1], exc_info[2]
+ def _download_nothreads(self, stream_or_path, buffer_size=None):
+ """
+ Internal method to do a download without separate threads, queues etc.
+ Called by download above if is_threading_enabled() == False
+ """
+ cdef:
+ int64_t bytes_read = 0
+ uint8_t* buf
+
+ handle = self.get_input_stream()
+
+ buffer_size = buffer_size or DEFAULT_BUFFER_SIZE
+
+ if not hasattr(stream_or_path, 'read'):
+ stream = open(stream_or_path, 'wb')
+
+ def cleanup():
+ stream.close()
Review Comment:
Oh, I see what you mean, should be called in finally below I think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2211062127
@github-actions crossbow submit -g wheel
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2250939388
Thanks @joemarshall for the amazing effort and everyone for the reviews!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1640047599
##########
python/pyarrow/tests/test_compute.py:
##########
@@ -2481,6 +2493,10 @@ def _check_temporal_rounding(ts, values, unit):
@pytest.mark.skipif(sys.platform == "win32" and not util.windows_has_tzdata(),
reason="Timezone database is not installed on Windows")
+@pytest.mark.xfail(
+ sys.platform == "emscripten",
+ reason="Emscripten datetime is implemented in Javascript and works differently"
+)
Review Comment:
> because we get our timezone files using the python tzdata package, which builds using the -b slim option, which is not supported yet by HowardHinnant/date
It's still a bit bizarre.
I checked locally on my Linux in a conda env, and the zoneinfo files provided in `/env/share/zoneinfo` ("system" provided) or `env/lib/python3.11/site-packages/tzdata/zoneinfo` (so from tzdata package) are exactly the same. So if `tzdata` provides the slim binaries, I don't understand how this is working ..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1622368110
##########
ci/docker/ubuntu-22.04-cpp-emscripten.dockerfile:
##########
Review Comment:
I couldn't work out how to - it needs python 3.12 and then too much other stuff, browser test runners, browsers, emscripten etc. that are all emscripten specific, so I think the easiest way is to have it in its own dockerfile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] Add support for Pyodide [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1622233872
##########
python/pyarrow/_dataset_parquet.pyx:
##########
@@ -737,6 +737,8 @@ cdef class ParquetFragmentScanOptions(FragmentScanOptions):
new CParquetFragmentScanOptions()))
self.use_buffered_stream = use_buffered_stream
self.buffer_size = buffer_size
+ if pre_buffer and not is_threading_enabled():
+ pre_buffer=False
Review Comment:
> Is there a linter that can detect this style mismatch?
We use `cython-lint` which I would assume to catch this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1622519260
##########
ci/docker/ubuntu-22.04-cpp-emscripten.dockerfile:
##########
Review Comment:
@jorisvandenbossche I think it is worth testing node + chrome at minimum just because things work differently on node, especially when you do things like handle time, use files etc. I think numpy only needs to test node because they don't do as much that makes system calls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1554224826
##########
python/pyarrow/io.pxi:
##########
@@ -677,16 +676,19 @@ cdef class NativeFile(_Weakrefable):
self.seek(0)
- writer_thread = threading.Thread(target=bg_write)
+ if is_threading_enabled():
+ writer_thread = threading.Thread(target=bg_write)
+ else:
+ writer_thread = None
# This isn't ideal -- PyBytes_FromStringAndSize copies the data from
# the passed buffer, so it's hard for us to avoid doubling the memory
buf = <uint8_t*> malloc(buffer_size)
if buf == NULL:
raise MemoryError("Failed to allocate {0} bytes"
.format(buffer_size))
-
- writer_thread.start()
+ if writer_thread:
Review Comment:
good idea, have done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1570294288
##########
python/CMakeLists.txt:
##########
@@ -365,6 +388,9 @@ if(NOT PYARROW_CPP_LINK_LIBS)
endif()
add_library(arrow_python SHARED ${PYARROW_CPP_SRCS})
+target_compile_options(arrow_python PUBLIC "-fexceptions")
Review Comment:
Could you do this only when building with Emscripten?
##########
python/CMakeLists.txt:
##########
@@ -590,28 +616,42 @@ endif()
# Acero
if(PYARROW_BUILD_ACERO)
- if(PYARROW_BUNDLE_ARROW_CPP)
- bundle_arrow_lib(${ARROW_ACERO_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
- if(MSVC)
- bundle_arrow_import_lib(${ARROW_ACERO_IMPORT_LIB})
+ if(ARROW_BUILD_SHARED)
+
+ if(PYARROW_BUNDLE_ARROW_CPP)
+ bundle_arrow_lib(${ARROW_ACERO_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
+ if(MSVC)
+ bundle_arrow_import_lib(${ARROW_ACERO_IMPORT_LIB})
+ endif()
endif()
- endif()
- set(ACERO_LINK_LIBS ArrowAcero::arrow_acero_shared)
- list(APPEND CYTHON_EXTENSIONS _acero)
+ set(ACERO_LINK_LIBS ArrowAcero::arrow_acero_shared)
+ list(APPEND CYTHON_EXTENSIONS _acero)
+ else()
+ # ACERO is statically linked into libarrow_python already
+ set(ACERO_LINK_LIBS)
+ list(APPEND CYTHON_EXTENSIONS _acero)
Review Comment:
Could you move `list(APPEND CYTHON_EXTENSIONS _acero)` to out of `if(ARROW_BUILD_SHARED)`?
##########
python/CMakeLists.txt:
##########
@@ -763,8 +813,12 @@ foreach(module ${CYTHON_EXTENSIONS})
set_target_properties(${module_name} PROPERTIES COMPILE_DEFINITIONS
"CYTHON_TRACE=1;CYTHON_TRACE_NOGIL=1")
endif()
-
- target_link_libraries(${module_name} PRIVATE ${LINK_LIBS})
+ # user this rather roundabout way of including, otherwise the
+ # in static linking mode the target will depend on libarrow and
+ # build it into every cython extension
+ target_include_directories(${module_name} PUBLIC $<TARGET_PROPERTY:arrow_python,INTERFACE_INCLUDE_DIRECTORIES>)
+ target_link_libraries(${module_name} PUBLIC $<TARGET_FILE:arrow_python>)
+ add_dependencies(${module_name} arrow_python)
Review Comment:
We want to use CMake target directly instead of each parameter separately...
##########
python/CMakeLists.txt:
##########
@@ -590,28 +616,42 @@ endif()
# Acero
if(PYARROW_BUILD_ACERO)
- if(PYARROW_BUNDLE_ARROW_CPP)
- bundle_arrow_lib(${ARROW_ACERO_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
- if(MSVC)
- bundle_arrow_import_lib(${ARROW_ACERO_IMPORT_LIB})
+ if(ARROW_BUILD_SHARED)
+
+ if(PYARROW_BUNDLE_ARROW_CPP)
+ bundle_arrow_lib(${ARROW_ACERO_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
+ if(MSVC)
+ bundle_arrow_import_lib(${ARROW_ACERO_IMPORT_LIB})
+ endif()
endif()
- endif()
- set(ACERO_LINK_LIBS ArrowAcero::arrow_acero_shared)
- list(APPEND CYTHON_EXTENSIONS _acero)
+ set(ACERO_LINK_LIBS ArrowAcero::arrow_acero_shared)
+ list(APPEND CYTHON_EXTENSIONS _acero)
+ else()
+ # ACERO is statically linked into libarrow_python already
+ set(ACERO_LINK_LIBS)
Review Comment:
Hmm. I think that we need `ArrowAcero::arrow_acero_static`...
##########
python/CMakeLists.txt:
##########
@@ -148,6 +151,8 @@ include(BuildUtils)
# Cython generated code emits way to many warnings at CHECKIN and EVERYTHING
set(BUILD_WARNING_LEVEL "PRODUCTION")
+
+
Review Comment:
Could you revert needless changes?
##########
python/setup.py:
##########
@@ -311,10 +367,25 @@ def append_cmake_bool(value, varname):
if parallel:
build_tool_args.append(f'-j{parallel}')
- # Generate the build files
- print("-- Running cmake for PyArrow")
- self.spawn(['cmake'] + extra_cmake_args + cmake_options + [source])
- print("-- Finished cmake for PyArrow")
+ if "PYODIDE" in os.environ and os.environ["PYODIDE"] == "1":
+ # remove c++ standard setting which pyodide
+ # (emscripten python) adds to its build variables
+ import json
+ pyodide_args = json.loads(os.environ["PYWASMCROSS_ARGS"])
+ pyodide_args["cxxflags"] = pyodide_args["cxxflags"].replace(
+ "-std=c++14", "")
Review Comment:
Can we remove similar code in `CMakeLists.txt`?
##########
python/setup.py:
##########
@@ -133,8 +143,68 @@ def run(self):
'bundle the Arrow C++ headers')] +
_build_ext.user_options)
+ def get_arrow_build_options(self):
Review Comment:
It seems that you want to enable PyArrow components if corresponding Arrow C++ components are enabled.
If so, you don't need to specify `PYARROW_BUILD_*` CMake variables by `setup.py` or can ignore CMake variables specified by`setup.py` in `CMakeLists.txt`. You can use https://github.com/apache/arrow/pull/37822/files#r1344887227 for this approach.
##########
python/CMakeLists.txt:
##########
@@ -683,14 +725,22 @@ if(PYARROW_BUILD_SUBSTRAIT)
message(FATAL_ERROR "You must build Arrow C++ with ARROW_SUBSTRAIT=ON")
endif()
find_package(ArrowSubstrait REQUIRED)
- if(PYARROW_BUNDLE_ARROW_CPP)
- bundle_arrow_lib(${ARROW_SUBSTRAIT_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
- if(MSVC)
- bundle_arrow_import_lib(${ARROW_SUBSTRAIT_IMPORT_LIB})
+
+ if(ARROW_BUILD_SHARED)
+ if(PYARROW_BUNDLE_ARROW_CPP)
+ bundle_arrow_lib(${ARROW_SUBSTRAIT_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
+ if(MSVC)
+ bundle_arrow_import_lib(${ARROW_SUBSTRAIT_IMPORT_LIB})
+ endif()
endif()
+ set(SUBSTRAIT_LINK_LIBS ArrowSubstrait::arrow_substrait_shared)
+ else()
+ # only link to the output of substrait, not to libarrow etc.
+ # if we link to the target directly we end up with multiple copies
+ # of libarrow / parquet etc.
+ set(SUBSTRAIT_LINK_LIBS $<TARGET_FILE:ArrowSubstrait::arrow_substrait_static> )
Review Comment:
Hmm. I think that CMake removes duplicated libraries automatically when we use CMake target.
##########
python/pyarrow/conftest.py:
##########
@@ -81,9 +85,16 @@
'slow': False,
'snappy': Codec.is_available('snappy'),
'substrait': False,
+ 'threading': is_threading_enabled(),
'zstd': Codec.is_available('zstd'),
+ 'processes': True
Review Comment:
Could you keep this list in alphabetical order?
##########
python/pyarrow/conftest.py:
##########
@@ -51,6 +53,8 @@
'slow',
'requires_testing_data',
'zstd',
+ 'threading',
+ 'processes',
Review Comment:
Could you keep this list in alphabetical order?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1574512095
##########
python/CMakeLists.txt:
##########
@@ -590,28 +616,42 @@ endif()
# Acero
if(PYARROW_BUILD_ACERO)
- if(PYARROW_BUNDLE_ARROW_CPP)
- bundle_arrow_lib(${ARROW_ACERO_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
- if(MSVC)
- bundle_arrow_import_lib(${ARROW_ACERO_IMPORT_LIB})
+ if(ARROW_BUILD_SHARED)
+
+ if(PYARROW_BUNDLE_ARROW_CPP)
+ bundle_arrow_lib(${ARROW_ACERO_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
+ if(MSVC)
+ bundle_arrow_import_lib(${ARROW_ACERO_IMPORT_LIB})
+ endif()
endif()
- endif()
- set(ACERO_LINK_LIBS ArrowAcero::arrow_acero_shared)
- list(APPEND CYTHON_EXTENSIONS _acero)
+ set(ACERO_LINK_LIBS ArrowAcero::arrow_acero_shared)
+ list(APPEND CYTHON_EXTENSIONS _acero)
+ else()
+ # ACERO is statically linked into libarrow_python already
+ set(ACERO_LINK_LIBS)
+ list(APPEND CYTHON_EXTENSIONS _acero)
Review Comment:
oops, done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1574522230
##########
python/CMakeLists.txt:
##########
@@ -365,6 +388,9 @@ if(NOT PYARROW_CPP_LINK_LIBS)
endif()
add_library(arrow_python SHARED ${PYARROW_CPP_SRCS})
+target_compile_options(arrow_python PUBLIC "-fexceptions")
Review Comment:
yep
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1633254489
##########
python/pyarrow/_emscripten_timezones.py:
##########
@@ -0,0 +1,48 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from importlib import resources
+from pathlib import Path
+from shutil import copytree
+
+try:
+ import js
+
+ def setup_emscripten_timezone_database():
+ """Setup the timezone database for use on browser based
+ emscripten environments.
+ """
+ if not Path("/usr/share/zoneinfo").exists():
+ Path("/usr/share/").mkdir(parents=True, exist_ok=True)
+ try:
+ tzpath = resources.files("tzdata").joinpath("zoneinfo")
+ copytree(tzpath, "/usr/share/zoneinfo", dirs_exist_ok=True)
+ localtime_path = Path("/etc/localtime")
+ if not localtime_path.exists():
+ # get local timezone from browser js object
+ timezone = js.Intl.DateTimeFormat().resolvedOptions().timeZone
+ if timezone and str(timezone) != "":
+ timezone = str(timezone)
+ # make symbolic link to local time
+ Path("/etc/").mkdir(parents=True, exist_ok=True)
+ localtime_path.symlink_to(tzpath / timezone)
+ except (ImportError, IOError):
+ print("Arrow couldn't install timezone db to /usr/share/zoneinfo")
Review Comment:
can do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1639678156
##########
python/pyarrow/tests/test_compute.py:
##########
@@ -2481,6 +2493,10 @@ def _check_temporal_rounding(ts, values, unit):
@pytest.mark.skipif(sys.platform == "win32" and not util.windows_has_tzdata(),
reason="Timezone database is not installed on Windows")
+@pytest.mark.xfail(
+ sys.platform == "emscripten",
+ reason="Emscripten datetime is implemented in Javascript and works differently"
+)
Review Comment:
Aha, nailed it down -
We're hitting this:
https://github.com/HowardHinnant/date/issues/614
because we get our timezone files using the python tzdata package, which builds using the -b slim option, which is not supported yet by HowardHinnant/date
I'm inclined to bundle pre-built timezones into pyarrow build as resources?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1666534622
##########
python/pyarrow/tests/test_io.py:
##########
@@ -1334,6 +1336,9 @@ def test_native_file_modes(tmpdir):
assert f.seekable()
+@pytest.mark.xfail(
Review Comment:
It's dependent on the underlying file system and there are bugs filed against it so it should at some point in the future work, so I left it as xfail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2210757530
Revision: fa0e497fb22062a613d801d4d9e4e85add3084c4
Submitted crossbow builds: [ursacomputing/crossbow @ actions-e5764a79dd](https://github.com/ursacomputing/crossbow/branches/all?query=actions-e5764a79dd)
|Task|Status|
|----|------|
|test-conda-python-emscripten|[](https://github.com/ursacomputing/crossbow/actions/runs/9807996652/job/27082901781)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1636745634
##########
python/pyarrow/tests/test_compute.py:
##########
@@ -2481,6 +2493,10 @@ def _check_temporal_rounding(ts, values, unit):
@pytest.mark.skipif(sys.platform == "win32" and not util.windows_has_tzdata(),
reason="Timezone database is not installed on Windows")
+@pytest.mark.xfail(
+ sys.platform == "emscripten",
+ reason="Emscripten datetime is implemented in Javascript and works differently"
+)
Review Comment:
Ah, I've got it, it's something to do with timezones and daylight saving, I think it isn't actually in the emscripten javascript that the bug is happening. I think there's an error with DST.
I can't work out where the bug is though, whether it is a bug in arrow that only happens in emscripten, or an emscripten libc++ bug or what.
If I run the below code, which takes a timestamp near the daylight savings time change, then pandas and arrow disagree as to whether some dates should be dst or not.
```
import pyarrow as pa
import pyarrow.compute as pc
import pandas as pd
for x in range(0,20):
t=pd.to_datetime("%d-10-26 09:39:00.316686848+01:00 "%(x+1990)).tz_convert("Europe/London")
s=pa.scalar(t)
print(t,s,t.dst(),pc.is_dst(s))
```
Output looks like this
|Pandas_timestamp|Arrow_timestamp|pandasDST_offset|pyarrow_dst_offset|
|-----------|------------|--------------|-----------|
|1990-10-26 09:39:00.316686848+01:00|1990-10-26 09:39:00.316686+01:00|1:00:00|True|
|1991-10-26 09:39:00.316686848+01:00|1991-10-26 09:39:00.316686+01:00|1:00:00|True|
|1992-10-26 08:39:00.316686848+00:00|1992-10-26 08:39:00.316686+00:00|0:00:00|False|
|1993-10-26 08:39:00.316686848+00:00|1993-10-26 08:39:00.316686+00:00|0:00:00|False|
|1994-10-26 08:39:00.316686848+00:00|1994-10-26 08:39:00.316686+00:00|0:00:00|False|
|1995-10-26 08:39:00.316686848+00:00|1995-10-26 08:39:00.316686+00:00|0:00:00|False|
|1996-10-26 09:39:00.316686848+01:00|1996-10-26 09:39:00.316686+01:00|1:00:00|False|
|1997-10-26 08:39:00.316686848+00:00|1997-10-26 08:39:00.316686+00:00|0:00:00|False|
|1998-10-26 08:39:00.316686848+00:00|1998-10-26 08:39:00.316686+00:00|0:00:00|False|
|1999-10-26 09:39:00.316686848+01:00|1999-10-26 09:39:00.316686+01:00|1:00:00|False|
|2000-10-26 09:39:00.316686848+01:00|2000-10-26 09:39:00.316686+01:00|1:00:00|False|
|2001-10-26 09:39:00.316686848+01:00|2001-10-26 09:39:00.316686+01:00|1:00:00|False|
|2002-10-26 09:39:00.316686848+01:00|2002-10-26 09:39:00.316686+01:00|1:00:00|False|
|2003-10-26 08:39:00.316686848+00:00|2003-10-26 08:39:00.316686+00:00|0:00:00|False|
|2004-10-26 09:39:00.316686848+01:00|2004-10-26 09:39:00.316686+01:00|1:00:00|False|
|2005-10-26 09:39:00.316686848+01:00|2005-10-26 09:39:00.316686+01:00|1:00:00|False|
|2006-10-26 09:39:00.316686848+01:00|2006-10-26 09:39:00.316686+01:00|1:00:00|False|
|2007-10-26 09:39:00.316686848+01:00|2007-10-26 09:39:00.316686+01:00|1:00:00|False|
|2008-10-26 08:39:00.316686848+00:00|2008-10-26 08:39:00.316686+00:00|0:00:00|False|
|2009-10-26 08:39:00.316686848+00:00|2009-10-26 08:39:00.316686+00:00|0:00:00|False|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2209463287
> @joemarshall if you're able to react to Kou's review in the next day or so, then I think we can get this merged in time for the 17.0.0 release
Brilliant, I'll get onto it either later on today or first thing tomorrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1620053099
##########
cpp/src/arrow/acero/CMakeLists.txt:
##########
@@ -173,13 +173,8 @@ add_arrow_acero_test(hash_join_node_test SOURCES hash_join_node_test.cc
bloom_filter_test.cc)
add_arrow_acero_test(pivot_longer_node_test SOURCES pivot_longer_node_test.cc)
-# asof_join_node and sorted_merge_node use std::thread internally
-# and doesn't use ThreadPool so it will
-# be broken if threading is turned off
-if(ARROW_ENABLE_THREADING)
- add_arrow_acero_test(asof_join_node_test SOURCES asof_join_node_test.cc)
- add_arrow_acero_test(sorted_merge_node_test SOURCES sorted_merge_node_test.cc)
-endif()
+add_arrow_acero_test(asof_join_node_test SOURCES asof_join_node_test.cc)
+add_arrow_acero_test(sorted_merge_node_test SOURCES sorted_merge_node_test.cc)
Review Comment:
Could you rebase on main to remove needless changes from this PR?
It'll be easy to review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] Add support for Pyodide [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1622374286
##########
ci/docker/ubuntu-22.04-cpp-emscripten.dockerfile:
##########
Review Comment:
Other related questions: do we "need" to test this in all those different ways (node + different browsers)?
Do we expect there might be code issues on our side that would only be seen in one of those testing platforms? Because if not, testing on one might be sufficient? (for example, my understanding from looking at the CI config is that numpy is only testing with node: https://github.com/numpy/numpy/blob/main/.github/workflows/emscripten.yml)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2143451472
> Could you open a new issue instead of reusing closed GH-23221?
Opened https://github.com/apache/arrow/issues/41910
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1633262076
##########
python/pyarrow/error.pxi:
##########
@@ -217,7 +218,10 @@ cdef class SignalStopHandler:
maybe_source.status().Warn()
else:
self._stop_token.init(deref(maybe_source).token())
- self._enabled = True
+ if not is_threading_enabled():
+ self._enabled = False
Review Comment:
I'm not sure of other platforms, but yes on Emscripten signal handling doesn't really work without another thread to fire off the signals and CTRL+C doesn't work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1632112288
##########
docker-compose.yml:
##########
@@ -679,6 +680,45 @@ services:
UBUNTU:
command: *cpp-command
+ ubuntu-cpp-emscripten-python:
Review Comment:
Could you update this?
##########
docker-compose.yml:
##########
@@ -679,6 +680,45 @@ services:
UBUNTU:
command: *cpp-command
+ ubuntu-cpp-emscripten-python:
+ # Usage:
+ # docker-compose build ubuntu-cpp-emscripten-python
+ # docker-compose run --rm ubuntu-cpp-emscripten-python
+ # Parameters:
+ # ARCH: amd64, arm64v8, ...
+ # UBUNTU: 22.04
+ image: ${REPO}:${ARCH}-ubuntu-22.04-cpp-emscripten
+ build:
+ context: .
+ dockerfile: ci/docker/ubuntu-22.04-cpp-emscripten.dockerfile
Review Comment:
Could you update the path, `image:` and `cache_from:`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2162791842
@joemarshall thanks for the updates!
You added the `conda-python-emscripten` docker build, but is this actually used somewhere? (I don't see any equivalent changes to the CI setup to run this, or to tasks.yml to enable it as nightly build (like is done for `ubuntu-cpp-emscripten`))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2069199736
@kou I've fixed most of those things you mentioned above - there are a couple of points where it looks weird as per your comments but is right in this case -
1) I don't include `arrow_acero_static` as a dependency for `_acero`, because libarrow_acero is already in `libarrow_python.so`
2) For substrait, we use ` set(SUBSTRAIT_LINK_LIBS $<TARGET_FILE:ArrowSubstrait::arrow_substrait_static> )` to link just `libarrow_substrait.a` into `_substrait.so`. If we link directly to `arrow_substrait_static`, it will link libarrow.a into `_substrait.so`, leading to a duplicate of `libarrow.a` (which is already included in `libarrow_python.so`.
I did try building without the separate `libarrow_python.so`, i.e. linking libarrow_python as a static library then linking statically into each cython extension, but (a) it is too much bigger, even with dead code elimination, (b) bad things happened in testing because libarrow expects to only exist once.
I've also added a docker configuration to build emscripten python and run the test suite on node.js, chrome and firefox. Some notes on this:
1) Right now it has a patch for pyodide, to fix the bug https://github.com/emscripten-core/emscripten/pull/21759/ which is applied in the python test script. Once pyodide 0.26 releases this can be dropped.
2) The script (in `python/scripts` which runs the tests is non-trivial; at some point it would be worth upstreaming this code to somewhere pyodide related, but I can't work out where right now. I'm talking to pyodide people about it, but for now it needs to live in arrow still.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2069229762
Oh, also, about the `get_env_options` stuff in setup.py. The problem with your proposed solution is that setup.py depends on those options internally. So if we do everything in CMakelists.txt, setup.py can't know what they are set to there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2047952431
Is this a real issue or just a flaky test?
```
File "/home/runner/.cache/pre-commit/repoubvtufj5/py_env-python3.12/lib/python3.12/site-packages/flake8/plugins/finder.py", line 293, in _load_plugin
raise FailedToLoadPlugin(plugin.package, e)
flake8.exceptions.FailedToLoadPlugin: Flake8 failed to load plugin "pyflakes" due to source code string cannot contain null bytes.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2168294883
Revision: 3c08fa89239326fa432865dcea37f4597cba9e6c
Submitted crossbow builds: [ursacomputing/crossbow @ actions-e0b66a7419](https://github.com/ursacomputing/crossbow/branches/all?query=actions-e0b66a7419)
|Task|Status|
|----|------|
|test-conda-python-emscripten|[](https://github.com/ursacomputing/crossbow/actions/runs/9518743828/job/26240325843)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2186632187
Revision: 3b4bd02a66cd9d2a229de5e0889a619e04805509
Submitted crossbow builds: [ursacomputing/crossbow @ actions-97f5efbe4d](https://github.com/ursacomputing/crossbow/branches/all?query=actions-97f5efbe4d)
|Task|Status|
|----|------|
|test-conda-python-emscripten|[](https://github.com/ursacomputing/crossbow/actions/runs/9646571174/job/26603304193)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2191641250
I think cross-compiling of orc should work now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2211420126
Checks look good to me. The CI failures are unrelated.
@kou do you think this is good to merge now? If so please go ahead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2169176893
> > I made it embed the timezone data in the python wheel rather than downloading it using the tzdata library
>
> I started writing my comment at https://github.com/apache/arrow/pull/37822#discussion_r1639997184 an hour ago, but got interrupted and only finished it now before seeing your update. So you clearly did succeed to include it, which I thought might not be possible for non-Windows platforms.
>
> I am a bit wary of including that data (for Windows we decided to not do that). But apart from that, the reason this works is because you actually copy the data to the location in "/usr/share/zoneinfo" (where tz.h expects it to be), right? (I was thinking about the windows case where we can point to a custom download location, which we don't support on other platforms).
> Are there situations that this folder could already exist? Or should that generally not be the case for emscripten (node or browser) environments? And if it would already exist, are we sure it has the same format?
The ideal would be to be able to point at the folder the same as on windows. I'm not sure how much work that would be?
In terms of whether to bundle it, the ideal would be for it to be downloaded as a python module as standard, but thanks to that bug with slim timezone format, we can't yet do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2210819324
Revision: fa0e497fb22062a613d801d4d9e4e85add3084c4
Submitted crossbow builds: [ursacomputing/crossbow @ actions-a1c5cd96da](https://github.com/ursacomputing/crossbow/branches/all?query=actions-a1c5cd96da)
|Task|Status|
|----|------|
|wheel-macos-big-sur-cp310-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808533166/job/27084530913)|
|wheel-macos-big-sur-cp311-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808532550/job/27084528177)|
|wheel-macos-big-sur-cp312-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808532842/job/27084529351)|
|wheel-macos-big-sur-cp38-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808532638/job/27084528230)|
|wheel-macos-big-sur-cp39-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808533066/job/27084530069)|
|wheel-macos-catalina-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808533156/job/27084530803)|
|wheel-macos-catalina-cp311-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808533062/job/27084530079)|
|wheel-macos-catalina-cp312-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808532653/job/27084528497)|
|wheel-macos-catalina-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808532739/job/27084528740)|
|wheel-macos-catalina-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808533450/job/27084531899)|
|wheel-manylinux-2-28-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808533014/job/27084529656)|
|wheel-manylinux-2-28-cp310-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808533194/job/27084531256)|
|wheel-manylinux-2-28-cp311-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808533073/job/27084530083)|
|wheel-manylinux-2-28-cp311-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808533382/job/27084531570)|
|wheel-manylinux-2-28-cp312-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808533261/job/27084531265)|
|wheel-manylinux-2-28-cp312-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808532766/job/27084529029)|
|wheel-manylinux-2-28-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808532925/job/27084529477)|
|wheel-manylinux-2-28-cp38-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808532596/job/27084528175)|
|wheel-manylinux-2-28-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808533083/job/27084530085)|
|wheel-manylinux-2-28-cp39-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808533009/job/27084529703)|
|wheel-manylinux-2014-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808533573/job/27084532502)|
|wheel-manylinux-2014-cp310-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808533477/job/27084531959)|
|wheel-manylinux-2014-cp311-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808532895/job/27084529524)|
|wheel-manylinux-2014-cp311-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808533187/job/27084530810)|
|wheel-manylinux-2014-cp312-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808533672/job/27084532985)|
|wheel-manylinux-2014-cp312-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808533437/job/27084531886)|
|wheel-manylinux-2014-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808533241/job/27084531237)|
|wheel-manylinux-2014-cp38-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808533280/job/27084531432)|
|wheel-manylinux-2014-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808532775/job/27084528806)|
|wheel-manylinux-2014-cp39-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808533530/job/27084532496)|
|wheel-windows-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808533196/job/27084531242)|
|wheel-windows-cp311-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808532835/job/27084528947)|
|wheel-windows-cp312-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808532942/job/27084529478)|
|wheel-windows-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808532729/job/27084528803)|
|wheel-windows-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/9808533115/job/27084530540)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2186627024
@github-actions crossbow submit test-conda-python-emscripten
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "cpcloud (via GitHub)" <gi...@apache.org>.
cpcloud commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2066776793
@joemarshall Thanks for powering through this odyssey.
Building the wheel from your PR, I loaded this `index.html` into Brave:
```html
<html>
<head>
<meta charset="utf-8" />
</head>
<body>
<script type="text/javascript" src="https://cdn.jsdelivr.net/pyodide/v0.25.1/full/pyodide.js"></script>
<script type="text/javascript">
async function main() {
let pyodide = await loadPyodide();
await pyodide.loadPackage("micropip");
const micropip = pyodide.pyimport("micropip");
await micropip.install([
"http://localhost:8001/pyarrow-16.0.0.dev2661+g9bddb87fd-cp311-cp311-emscripten_3_1_46_wasm32.whl"
])
await pyodide.runPython(`import pyarrow as pa`);
}
main();
</script>
</body>
</html>
```
And I get this error:
<details>
<summary>Build script</summary>
```bash
#!/usr/bin/env bash
set -euo pipefail
apt update -y
apt upgrade -y
apt install -y software-properties-common build-essential git curl cmake ninja-build
add-apt-repository -y ppa:deadsnakes/ppa
apt update -y
DEBIAN_FRONTEND=noninteractive TZ=America/New_York apt install -y python3.11 python3.11-dev python3.11-venv
rm -rf .venv* .pyodide*
python3.11 -m venv /scripts/.venv
source /scripts/.venv/bin/activate
pip install 'pyodide-build==0.25.1' 'pydantic<2'
rm -rf /scripts/emsdk /scripts/duckdb /scripts/arrow
git clone https://github.com/emscripten-core/emsdk.git /scripts/emsdk
pushd /scripts/emsdk
PYODIDE_EMSCRIPTEN_VERSION="$(pyodide config get emscripten_version)"
./emsdk install "${PYODIDE_EMSCRIPTEN_VERSION}"
./emsdk activate "${PYODIDE_EMSCRIPTEN_VERSION}"
source /scripts/emsdk/emsdk_env.sh
popd
git clone https://github.com/joemarshall/arrow.git
pushd /scripts/arrow/cpp
git checkout "${1:-emscripten_python_changes}"
emcmake cmake --preset ninja-release-emscripten
ninja install
popd
pushd /scripts/arrow/python
pyodide build --exports=whole_archive
popd
```
I also tried running without `--exports=whole_archive` but I get the same error.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1622515131
##########
ci/scripts/cpp_build.sh:
##########
@@ -105,8 +105,6 @@ if [ "${ARROW_EMSCRIPTEN:-OFF}" = "ON" ]; then
-DCMAKE_C_FLAGS="${CFLAGS:-}" \
-DCMAKE_CXX_FLAGS="${CXXFLAGS:-}" \
-DCMAKE_CXX_STANDARD="${CMAKE_CXX_STANDARD:-17}" \
- -DCMAKE_INSTALL_LIBDIR=${CMAKE_INSTALL_LIBDIR:-lib} \
- -DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX:-${ARROW_HOME}} \
Review Comment:
(otherwise it can't find them when building e.g. python or other things that depend on it)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1633244752
##########
python/setup.py:
##########
@@ -40,6 +40,12 @@
# Check if we're running 64-bit Python
is_64_bit = sys.maxsize > 2**32
+is_emscripten = (
+ sysconfig.get_config_var("SOABI")
+ and sysconfig.get_config_var("SOABI").find("emscripten") != -1
+)
Review Comment:
Yes, because in cross-compiling situations it appears `sys.platform` contains the build platform (i.e. linux / windows or whatever), whereas `sysconfig` contains the target platform (i.e. emscripten). So in setup.py, which is running on a host platform, you have to look as `sysconfig`, whereas in code that will actually run on emscripten, you can use sys.platform.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1636304550
##########
python/pyarrow/tests/test_io.py:
##########
@@ -1073,6 +1073,8 @@ def _try_delete(path):
def test_memory_map_writer(tmpdir):
+ if sys.platform == "emscripten":
+ pytest.xfail("Multiple memory maps to same file don't work on emscripten")
Review Comment:
Yes, fine to leave as is then
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1636611796
##########
python/scripts/run_emscripten_tests.py:
##########
Review Comment:
Oh, and the other pain was running pytest on the host to gather tests requires gathering of tests to work, which requires conftest.py to work on the host machine as well as on the target pyodide. I could never get that working, because on the host conftest.py tries to import pyarrow.lib which is a compiled module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2167104539
Revision: 210f928467506661f5ba3a2ba39c57c07b93937a
Submitted crossbow builds: [ursacomputing/crossbow @ actions-365bcc7e10](https://github.com/ursacomputing/crossbow/branches/all?query=actions-365bcc7e10)
|Task|Status|
|----|------|
|test-conda-python-emscripten|[](https://github.com/ursacomputing/crossbow/actions/runs/9509756019/job/26213185634)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1586242110
##########
python/setup.py:
##########
@@ -133,8 +143,68 @@ def run(self):
'bundle the Arrow C++ headers')] +
_build_ext.user_options)
+ def get_arrow_build_options(self):
Review Comment:
> You can use https://github.com/apache/arrow/pull/37822/files#r1344887227 for this approach.
As @joemarshall mentioned, right now that is not directly possible because we also use those variables from within `setup.py` for other reasons.
However, I have been looking a bit into that, and to me it seems this is only used for the "failure_permitted" logic, which I am not sure if that is something we necessarily need to keep.
So if we remove that logic, then we can indeed move the environment variable handling fully to the CMakeLists file. Given this is already a big PR, I think this is something we can discuss and review separately, so I opened https://github.com/apache/arrow/issues/41480 for this.
(@joemarshall I am also happy to take this on if you want, and ensure to move that forward so this doesn't need to block this PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1344280632
##########
python/CMakeLists.txt:
##########
@@ -570,28 +605,42 @@ endif()
# Acero
if(PYARROW_BUILD_ACERO)
- if(PYARROW_BUNDLE_ARROW_CPP)
- bundle_arrow_lib(${ARROW_ACERO_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
- if(MSVC)
- bundle_arrow_import_lib(${ARROW_ACERO_IMPORT_LIB})
+ if(ARROW_BUILD_SHARED)
+
+ if(PYARROW_BUNDLE_ARROW_CPP)
+ bundle_arrow_lib(${ARROW_ACERO_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
+ if(MSVC)
+ bundle_arrow_import_lib(${ARROW_ACERO_IMPORT_LIB})
+ endif()
endif()
- endif()
- set(ACERO_LINK_LIBS ArrowAcero::arrow_acero_shared)
- list(APPEND CYTHON_EXTENSIONS _acero)
+ set(ACERO_LINK_LIBS ArrowAcero::arrow_acero_shared)
+ list(APPEND CYTHON_EXTENSIONS _acero)
+ else()
+ # ACERO is statically linked into libarrow_python already
+ set(ACERO_LINK_LIBS)
+ list(APPEND CYTHON_EXTENSIONS _acero)
+ endif()
endif()
# Dataset
if(PYARROW_BUILD_DATASET)
- if(PYARROW_BUNDLE_ARROW_CPP)
- bundle_arrow_lib(${ARROW_DATASET_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
- if(MSVC)
- bundle_arrow_import_lib(${ARROW_DATASET_IMPORT_LIB})
+ if(ARROW_BUILD_SHARED)
+
+ if(PYARROW_BUNDLE_ARROW_CPP)
+ bundle_arrow_lib(${ARROW_DATASET_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
+ if(MSVC)
+ bundle_arrow_import_lib(${ARROW_DATASET_IMPORT_LIB})
+ endif()
endif()
- endif()
- set(DATASET_LINK_LIBS ArrowDataset::arrow_dataset_shared)
+ set(DATASET_LINK_LIBS ArrowDataset::arrow_dataset_shared)
+ else()
+ # dataset is sttically linked into libarrow_python already
Review Comment:
```suggestion
# dataset is statically linked into libarrow_python already
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1344887227
##########
python/setup.py:
##########
@@ -133,8 +143,68 @@ def run(self):
'bundle the Arrow C++ headers')] +
_build_ext.user_options)
+ def get_arrow_build_options(self):
Review Comment:
We can do this in `python/CMakeLists.txt`.
We can refer all `ARROW_*` variables after `find_package(Arrow)`.
We can use these values as the default value of `PYARROW_BUILD_*` (Note that the following example isn't completed. It breaks at least `PYARROW_CXXFLAGS`.):
```diff
diff --git a/python/CMakeLists.txt b/python/CMakeLists.txt
index 29f8d2da7..a002b4e4e 100644
--- a/python/CMakeLists.txt
+++ b/python/CMakeLists.txt
@@ -108,25 +108,6 @@ if(UNIX)
endif()
endif()
-# Top level cmake dir
-if("${CMAKE_SOURCE_DIR}" STREQUAL "${CMAKE_CURRENT_SOURCE_DIR}")
- option(PYARROW_BUILD_ACERO "Build the PyArrow Acero integration" OFF)
- option(PYARROW_BUILD_CUDA "Build the PyArrow CUDA support" OFF)
- option(PYARROW_BUILD_DATASET "Build the PyArrow Dataset integration" OFF)
- option(PYARROW_BUILD_FLIGHT "Build the PyArrow Flight integration" OFF)
- option(PYARROW_BUILD_GANDIVA "Build the PyArrow Gandiva integration" OFF)
- option(PYARROW_BUILD_ORC "Build the PyArrow ORC integration" OFF)
- option(PYARROW_BUILD_PARQUET "Build the PyArrow Parquet integration" OFF)
- option(PYARROW_BUILD_PARQUET_ENCRYPTION
- "Build the PyArrow Parquet encryption integration" OFF)
- option(PYARROW_BUNDLE_ARROW_CPP "Bundle the Arrow C++ libraries" OFF)
- option(PYARROW_BUNDLE_CYTHON_CPP "Bundle the C++ files generated by Cython" OFF)
- option(PYARROW_GENERATE_COVERAGE "Build with Cython code coverage enabled" OFF)
- set(PYARROW_CXXFLAGS
- ""
- CACHE STRING "Compiler flags to append when compiling Arrow")
-endif()
-
find_program(CCACHE_FOUND ccache)
if(CCACHE_FOUND
AND NOT CMAKE_C_COMPILER_LAUNCHER
@@ -267,6 +248,25 @@ include(GNUInstallDirs)
find_package(Arrow REQUIRED)
+# Top level cmake dir
+if("${CMAKE_SOURCE_DIR}" STREQUAL "${CMAKE_CURRENT_SOURCE_DIR}")
+ option(PYARROW_BUILD_ACERO "Build the PyArrow Acero integration" ${ARROW_ACERO})
+ option(PYARROW_BUILD_CUDA "Build the PyArrow CUDA support" ${ARROW_CUDA})
+ option(PYARROW_BUILD_DATASET "Build the PyArrow Dataset integration" ${ARROW_DATASET})
+ option(PYARROW_BUILD_FLIGHT "Build the PyArrow Flight integration" ${ARROW_FLIGHT})
+ option(PYARROW_BUILD_GANDIVA "Build the PyArrow Gandiva integration" ${ARROW_GANDIVA})
+ option(PYARROW_BUILD_ORC "Build the PyArrow ORC integration" ${ARROW_ORC})
+ option(PYARROW_BUILD_PARQUET "Build the PyArrow Parquet integration" ${ARROW_PARQUET})
+ option(PYARROW_BUILD_PARQUET_ENCRYPTION
+ "Build the PyArrow Parquet encryption integration" ${PARQUET_REQUIRE_ENCRYPTION})
+ option(PYARROW_BUNDLE_ARROW_CPP "Bundle the Arrow C++ libraries" OFF)
+ option(PYARROW_BUNDLE_CYTHON_CPP "Bundle the C++ files generated by Cython" OFF)
+ option(PYARROW_GENERATE_COVERAGE "Build with Cython code coverage enabled" OFF)
+ set(PYARROW_CXXFLAGS
+ ""
+ CACHE STRING "Compiler flags to append when compiling Arrow")
+endif()
+
set(PYARROW_CPP_ROOT_DIR pyarrow/src)
set(PYARROW_CPP_SOURCE_DIR ${PYARROW_CPP_ROOT_DIR}/arrow/python)
set(PYARROW_CPP_SRCS
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1344881620
##########
python/CMakeLists.txt:
##########
@@ -68,6 +68,25 @@ if(POLICY CMP0095)
cmake_policy(SET CMP0095 NEW)
endif()
+# this option is used to auto-set defaults for pyarrow build
+option(DUMP_ARROW_ARGUMENTS "Dump the arrow arguments then quit" OFF)
+
+if($ENV{PYODIDE})
+ # these variables are needed for building pyarrow on Emscripten
+ # if they aren't set, cmake cross compiling fails for python
+ # modules (at least under pyodide it does)
+ set(Python3_INCLUDE_DIR $ENV{PYTHONINCLUDE})
+ set(Python3_LIBRARY $ENV{CPYTHONLIB})
+ set(Python3_NumPy_INCLUDE_DIR $ENV{NUMPY_LIB}/core/include)
+ set(Python3_EXECUTABLE)
+ set(ENV{_PYTHON_SYSCONFIGDATA_NAME} $ENV{SYSCONFIG_NAME})
+
+ # remove any c++ standard settings from cflags and let cmake set it.
+ # n.b. this is here because pyodide sets -stdc++=14, which breaks pyarrow builds
+ string(REGEX REPLACE "-std=c\\+\\+[0-9]+" "" CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS})
Review Comment:
Can we review this after #37821 is merged?
This PR depends on #37821.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2194759607
@kou I've updated based on those comments - it now gets emscripten version from pyodide, and uses conda to install nodejs etc. I also removed the use of multiprocessing in the python test runner, to see if that is causing the weird hang in node tests, which I just can't reproduce in docker here even using archery. If you could run tests again that would be great.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] joemarshall commented on pull request #37822: GH-23221: [Python] python changes for pyodide build
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-1738966008
Oh and for now I have blocked the auto-setting of PYARROW_* to happen only on emscripten - I don't know if that makes sense or not?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] joemarshall commented on pull request #37822: GH-23221: [Python] python changes for pyodide build
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-1738965309
> @joemarshall thanks for the PR!
>
> We might want to expose `is_threading_enabled()` in pyarrow publicly (it might be useful for downstream packages as well?), in `__init__.py`
>
> > 2. pyarrow sets defaults for inclusion of submodules based on their inclusion in the arrow build. e.g. pyarrow.parquet is built only if ARROW_PARQUET is set. This makes it possible to build in situations where you don't have access to set the build environment variables (e.g. in cross compiling situations like pyodide).
>
> This part is not actually included here? (or I don't understand the sentence) (and it might also make sense to leave that for a third PR since it is changing the build process beyond emscriptem?)
Sorry, I missed out putting in the setup.py changes. They're in now.
About is_threading_enabled(), it is currently in `pyarrow.lib.is_threading_enabled()`. Does it need to be top-level?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1345510888
##########
python/pyarrow/parquet/core.py:
##########
@@ -329,6 +330,9 @@ def __init__(self, source, *, metadata=None, common_metadata=None,
decryption_properties=None, thrift_string_size_limit=None,
thrift_container_size_limit=None, filesystem=None):
+ if pre_buffer and not is_threading_enabled():
+ pre_buffer = False
Review Comment:
And looking at the other occurences of the check in this file, I think all of those can be removed, because they either eventually pass the `pre_buffer` keyword to `ParquetFile` (which is handled in _parquet.pyx ParquetReader) or to the pyarrow.dataset ParquetFileFormat/ ParquetFragmentScanOptions, which you already handled in _dataset_parquet.pyx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2062848787
Could you run `pre-commit run -a` to fix lint failures?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1574508173
##########
python/CMakeLists.txt:
##########
@@ -763,8 +813,12 @@ foreach(module ${CYTHON_EXTENSIONS})
set_target_properties(${module_name} PROPERTIES COMPILE_DEFINITIONS
"CYTHON_TRACE=1;CYTHON_TRACE_NOGIL=1")
endif()
-
- target_link_libraries(${module_name} PRIVATE ${LINK_LIBS})
+ # user this rather roundabout way of including, otherwise the
+ # in static linking mode the target will depend on libarrow and
+ # build it into every cython extension
+ target_include_directories(${module_name} PUBLIC $<TARGET_PROPERTY:arrow_python,INTERFACE_INCLUDE_DIRECTORIES>)
+ target_link_libraries(${module_name} PUBLIC $<TARGET_FILE:arrow_python>)
+ add_dependencies(${module_name} arrow_python)
Review Comment:
The reason for this is:
1) libarrow.a is a static library
2) arrow_python links to libarrow.a
3) If we depend on the arrow_python target, we also inherit the libarrow.a dependencies, and libarrow.a gets linked into every cython module, as well as in arrow_python shared library.
4) Two bad things happen - (a) the extension size is massive, (b) there are multiple copies of libarrow in the source, which means two copies of things that are supposed to be unique, which breaks some of compute.
I tried messing with private / public / interface targets, but I couldn't get past the fact that we need the include directories from arrow_python dependencies, but we need to not link the static libraries that it links in. If I make the static library targets private dependencies for arrow_python, then we lose them from the linker command line to cython modules, but we also lose the include directories, so cython can't call anything in arrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1574510160
##########
python/CMakeLists.txt:
##########
@@ -683,14 +725,22 @@ if(PYARROW_BUILD_SUBSTRAIT)
message(FATAL_ERROR "You must build Arrow C++ with ARROW_SUBSTRAIT=ON")
endif()
find_package(ArrowSubstrait REQUIRED)
- if(PYARROW_BUNDLE_ARROW_CPP)
- bundle_arrow_lib(${ARROW_SUBSTRAIT_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
- if(MSVC)
- bundle_arrow_import_lib(${ARROW_SUBSTRAIT_IMPORT_LIB})
+
+ if(ARROW_BUILD_SHARED)
+ if(PYARROW_BUNDLE_ARROW_CPP)
+ bundle_arrow_lib(${ARROW_SUBSTRAIT_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
+ if(MSVC)
+ bundle_arrow_import_lib(${ARROW_SUBSTRAIT_IMPORT_LIB})
+ endif()
endif()
+ set(SUBSTRAIT_LINK_LIBS ArrowSubstrait::arrow_substrait_shared)
+ else()
+ # only link to the output of substrait, not to libarrow etc.
+ # if we link to the target directly we end up with multiple copies
+ # of libarrow / parquet etc.
+ set(SUBSTRAIT_LINK_LIBS $<TARGET_FILE:ArrowSubstrait::arrow_substrait_static> )
Review Comment:
Not across multiple shared libraries it doesn't. If you use the target directly, cmake links libarrow into substrait itself, in addition to the copy in libarrow_python and we have two copies of libarrow in the final python module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] python changes for pyodide build [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1574602705
##########
python/CMakeLists.txt:
##########
@@ -763,8 +813,12 @@ foreach(module ${CYTHON_EXTENSIONS})
set_target_properties(${module_name} PROPERTIES COMPILE_DEFINITIONS
"CYTHON_TRACE=1;CYTHON_TRACE_NOGIL=1")
endif()
-
- target_link_libraries(${module_name} PRIVATE ${LINK_LIBS})
+ # user this rather roundabout way of including, otherwise the
+ # in static linking mode the target will depend on libarrow and
+ # build it into every cython extension
+ target_include_directories(${module_name} PUBLIC $<TARGET_PROPERTY:arrow_python,INTERFACE_INCLUDE_DIRECTORIES>)
+ target_link_libraries(${module_name} PUBLIC $<TARGET_FILE:arrow_python>)
+ add_dependencies(${module_name} arrow_python)
Review Comment:
Ah, hang on, if I make only ARROW_CPP_LINK_LIBS private, I think it works now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1664941286
##########
python/pyarrow/_parquet.pyx:
##########
@@ -1453,6 +1453,9 @@ cdef class ParquetReader(_Weakrefable):
default_arrow_reader_properties())
FileReaderBuilder builder
+ if pre_buffer and not is_threading_enabled():
+ pre_buffer=False
Review Comment:
```suggestion
pre_buffer = False
```
##########
python/pyarrow/io.pxi:
##########
@@ -702,7 +704,6 @@ cdef class NativeFile(_Weakrefable):
if buf == NULL:
raise MemoryError("Failed to allocate {0} bytes"
.format(buffer_size))
-
Review Comment:
Could you revert a needless change?
##########
python/pyarrow/error.pxi:
##########
@@ -217,7 +218,12 @@ cdef class SignalStopHandler:
maybe_source.status().Warn()
else:
self._stop_token.init(deref(maybe_source).token())
- self._enabled = True
+ # signals don't work on Emscripten without threads.
+ # and possibly other single-thread environments.
+ if not is_threading_enabled():
+ self._enabled = False
+ else:
+ self._enabled = True
Review Comment:
We don't need `if` here:
```suggestion
self._enabled = is_threading_enabled()
```
##########
python/pyarrow/tests/test_csv.py:
##########
@@ -1406,7 +1406,9 @@ def test_stress_convert_options_blowup(self):
assert table.num_rows == 0
assert table.column_names == col_names
+ @pytest.mark.threading
def test_cancellation(self):
+
Review Comment:
Could you revert a needless change?
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
+ if('results' in evt.data){
Review Comment:
```suggestion
if ('results' in evt.data) {
```
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
Review Comment:
```suggestion
window.python_done_callback = undefined;
window.python_logs = [];
```
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
+ if('results' in evt.data){
+ if(window.python_done_callback){
+ let callback=window.python_done_callback;
+ window.python_done_callback=undefined;
Review Comment:
```suggestion
window.python_done_callback = undefined;
```
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
+ if('results' in evt.data){
+ if(window.python_done_callback){
+ let callback=window.python_done_callback;
+ window.python_done_callback=undefined;
+ callback({result:evt.data.results});
+ }
+ }
+ if('print' in evt.data){
+ evt.data.print.forEach((x)=>{window.python_logs.push(x)});
+ }
+ }
+ window.pyworker = new Worker("worker.js");
+ window.pyworker.onmessage=capturelogs;
+ </script>
+ </head>
+ <body></body>
+ </html>
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "text/html")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+ elif self.path.endswith("/worker.js"):
+ body = b"""
+ importScripts("./pyodide.js");
+ onmessage = async function (e) {
+ const data = e.data;
+ if(!self.pyodide){
+ self.pyodide = await loadPyodide()
+ }
+ function do_print(arg){
+ let databytes = Array.from(arg)
+ self.postMessage({print:databytes})
+ return databytes.length
+ }
+ self.pyodide.setStdout({write:do_print,isatty:data.isatty});
+ self.pyodide.setStderr({write:do_print,isatty:data.isatty});
+
+ await self.pyodide.loadPackagesFromImports(data.python);
+ let results = await self.pyodide.runPythonAsync(data.python);
+ self.postMessage({results});
+ console.log('FINISHED_WEBWORKER')
+ }
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "application/javascript")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+
+ else:
+ return super().do_GET()
+
+ def end_headers(self):
+ # Enable Cross-Origin Resource Sharing (CORS)
+ self.send_header("Access-Control-Allow-Origin", "*")
+ super().end_headers()
+
+
+def run_server_thread(dist_dir, q):
+ global _SERVER_ADDRESS
+ os.chdir(dist_dir)
+ server = http.server.HTTPServer(("", 0), TemplateOverrider)
+ q.put(server.server_address)
+ print(f"Starting server for {dist_dir} at: {server.server_address}")
+ server.serve_forever()
+
+
+@contextlib.contextmanager
+def launch_server(dist_dir):
+ q = queue.Queue()
+ p = threading.Thread(target=run_server_thread, args=[dist_dir, q], daemon=True)
+ p.start()
+ address = q.get(timeout=50)
+ time.sleep(0.1) # wait to make sure server is started
+ yield address
+ p.terminate()
+
+
+class NodeDriver:
+ import subprocess
+
+ def __init__(self, hostname, port):
+ self.process = subprocess.Popen(
+ [shutil.which("script"), "-c", f'"{shutil.which("node")}"'],
+ stdin=subprocess.PIPE,
+ shell=False,
+ bufsize=0,
+ )
+ print(self.process)
+ time.sleep(0.1) # wait for node to start
+ self.hostname = hostname
+ self.port = port
+ self.last_ret_code = None
+
+ def load_pyodide(self, dist_dir):
+ self.execute_js(
+ f"""
+ const {{ loadPyodide }} = require('{dist_dir}/pyodide.js')
+ let pyodide = await loadPyodide()
+ """
+ )
+
+ def clear_logs(self):
+ pass # we don't handle logs for node
+
+ def write_stdin(self, buffer):
+ # because we use unbuffered IO for
+ # stdout, stdin.write is also unbuffered
+ # so might under-run on writes
+ while len(buffer) > 0 and self.process.poll() is None:
+ written = self.process.stdin.write(buffer)
+ if written == len(buffer):
+ break
+ elif written == 0:
+ # full buffer - wait
+ time.sleep(0.01)
+ else:
+ buffer = buffer[written:]
+
+ def execute_js(self, code, wait_for_terminate=True):
+ self.write_stdin((code + "\n").encode("utf-8"))
+
+ def load_arrow(self):
+ self.execute_js(f"await pyodide.loadPackage('{PYARROW_WHEEL_PATH}')")
+
+ def execute_python(self, code, wait_for_terminate=True):
+ js_code = f"""
+ python = `{code}`;
+ await pyodide.loadPackagesFromImports(python);
+ python_output = await pyodide.runPythonAsync(python);
+ """
+ self.last_ret_code = self.execute_js(js_code, wait_for_terminate)
+ return self.last_ret_code
+
+ def wait_for_done(self):
+ # in node we just let it run above
+ # then send EOF and join process
+ self.write_stdin(b"process.exit(python_output)\n")
+ return self.process.wait()
+
+
+class BrowserDriver:
+ def __init__(self, hostname, port, driver):
+ self.driver = driver
+ self.driver.get(f"http://{hostname}:{port}/test.html")
+ self.driver.set_script_timeout(100)
+
+ def load_pyodide(self, dist_dir):
+ pass
+
+ def load_arrow(self):
+ self.execute_python(
+ f"import pyodide_js as pjs\n"
+ f"await pjs.loadPackage('{PYARROW_WHEEL_PATH.name}')\n"
+ )
+
+ def execute_python(self, code, wait_for_terminate=True):
+ if wait_for_terminate:
+ self.driver.execute_async_script(
+ f"""
+ let callback=arguments[arguments.length-1];
+ python = `{code}`;
+ window.python_done_callback=callback
+ window.pyworker.postMessage(
+ {{python,isatty:{'true' if sys.stdout.isatty() else 'false'}}})
+ """
+ )
+ else:
+ self.driver.execute_script(
+ f"""
+ let python = `{code}`;
+ window.python_done_callback= (x) => {{window.python_script_done=x;}};
+ window.pyworker.postMessage(
+ {{python,isatty:{'true' if sys.stdout.isatty() else 'false'}}});
+ """
+ )
+
+ def clear_logs(self):
+ self.driver.execute_script("window.python_logs = []")
+
+ def wait_for_done(self):
+ while True:
+ # poll for console.log messages from our webworker
+ # which are the output of pytest
+ lines = self.driver.execute_script(
+ "let temp = window.python_logs;window.python_logs=[];return temp;"
+ )
+ if len(lines) > 0:
+ sys.stdout.buffer.write(bytes(lines))
+ done = self.driver.execute_script("return window.python_script_done")
Review Comment:
```suggestion
done = self.driver.execute_script("return window.python_script_done;")
```
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
+ if('results' in evt.data){
+ if(window.python_done_callback){
+ let callback=window.python_done_callback;
+ window.python_done_callback=undefined;
+ callback({result:evt.data.results});
+ }
+ }
+ if('print' in evt.data){
+ evt.data.print.forEach((x)=>{window.python_logs.push(x)});
+ }
+ }
+ window.pyworker = new Worker("worker.js");
+ window.pyworker.onmessage=capturelogs;
+ </script>
+ </head>
+ <body></body>
+ </html>
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "text/html")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+ elif self.path.endswith("/worker.js"):
+ body = b"""
+ importScripts("./pyodide.js");
+ onmessage = async function (e) {
+ const data = e.data;
+ if(!self.pyodide){
+ self.pyodide = await loadPyodide()
+ }
+ function do_print(arg){
+ let databytes = Array.from(arg)
+ self.postMessage({print:databytes})
+ return databytes.length
+ }
+ self.pyodide.setStdout({write:do_print,isatty:data.isatty});
+ self.pyodide.setStderr({write:do_print,isatty:data.isatty});
+
+ await self.pyodide.loadPackagesFromImports(data.python);
+ let results = await self.pyodide.runPythonAsync(data.python);
+ self.postMessage({results});
+ console.log('FINISHED_WEBWORKER')
+ }
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "application/javascript")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+
+ else:
+ return super().do_GET()
+
+ def end_headers(self):
+ # Enable Cross-Origin Resource Sharing (CORS)
+ self.send_header("Access-Control-Allow-Origin", "*")
+ super().end_headers()
+
+
+def run_server_thread(dist_dir, q):
+ global _SERVER_ADDRESS
+ os.chdir(dist_dir)
+ server = http.server.HTTPServer(("", 0), TemplateOverrider)
+ q.put(server.server_address)
+ print(f"Starting server for {dist_dir} at: {server.server_address}")
+ server.serve_forever()
+
+
+@contextlib.contextmanager
+def launch_server(dist_dir):
+ q = queue.Queue()
+ p = threading.Thread(target=run_server_thread, args=[dist_dir, q], daemon=True)
+ p.start()
+ address = q.get(timeout=50)
+ time.sleep(0.1) # wait to make sure server is started
+ yield address
+ p.terminate()
+
+
+class NodeDriver:
+ import subprocess
+
+ def __init__(self, hostname, port):
+ self.process = subprocess.Popen(
+ [shutil.which("script"), "-c", f'"{shutil.which("node")}"'],
+ stdin=subprocess.PIPE,
+ shell=False,
+ bufsize=0,
+ )
+ print(self.process)
+ time.sleep(0.1) # wait for node to start
+ self.hostname = hostname
+ self.port = port
+ self.last_ret_code = None
+
+ def load_pyodide(self, dist_dir):
+ self.execute_js(
+ f"""
+ const {{ loadPyodide }} = require('{dist_dir}/pyodide.js')
+ let pyodide = await loadPyodide()
+ """
+ )
+
+ def clear_logs(self):
+ pass # we don't handle logs for node
+
+ def write_stdin(self, buffer):
+ # because we use unbuffered IO for
+ # stdout, stdin.write is also unbuffered
+ # so might under-run on writes
+ while len(buffer) > 0 and self.process.poll() is None:
+ written = self.process.stdin.write(buffer)
+ if written == len(buffer):
+ break
+ elif written == 0:
+ # full buffer - wait
+ time.sleep(0.01)
+ else:
+ buffer = buffer[written:]
+
+ def execute_js(self, code, wait_for_terminate=True):
+ self.write_stdin((code + "\n").encode("utf-8"))
+
+ def load_arrow(self):
+ self.execute_js(f"await pyodide.loadPackage('{PYARROW_WHEEL_PATH}')")
+
+ def execute_python(self, code, wait_for_terminate=True):
+ js_code = f"""
+ python = `{code}`;
+ await pyodide.loadPackagesFromImports(python);
+ python_output = await pyodide.runPythonAsync(python);
+ """
+ self.last_ret_code = self.execute_js(js_code, wait_for_terminate)
+ return self.last_ret_code
+
+ def wait_for_done(self):
+ # in node we just let it run above
+ # then send EOF and join process
+ self.write_stdin(b"process.exit(python_output)\n")
+ return self.process.wait()
+
+
+class BrowserDriver:
+ def __init__(self, hostname, port, driver):
+ self.driver = driver
+ self.driver.get(f"http://{hostname}:{port}/test.html")
+ self.driver.set_script_timeout(100)
+
+ def load_pyodide(self, dist_dir):
+ pass
+
+ def load_arrow(self):
+ self.execute_python(
+ f"import pyodide_js as pjs\n"
+ f"await pjs.loadPackage('{PYARROW_WHEEL_PATH.name}')\n"
+ )
+
+ def execute_python(self, code, wait_for_terminate=True):
+ if wait_for_terminate:
+ self.driver.execute_async_script(
+ f"""
+ let callback=arguments[arguments.length-1];
+ python = `{code}`;
+ window.python_done_callback=callback
+ window.pyworker.postMessage(
+ {{python,isatty:{'true' if sys.stdout.isatty() else 'false'}}})
+ """
+ )
+ else:
+ self.driver.execute_script(
+ f"""
+ let python = `{code}`;
+ window.python_done_callback= (x) => {{window.python_script_done=x;}};
+ window.pyworker.postMessage(
+ {{python,isatty:{'true' if sys.stdout.isatty() else 'false'}}});
+ """
+ )
+
+ def clear_logs(self):
+ self.driver.execute_script("window.python_logs = []")
+
+ def wait_for_done(self):
+ while True:
+ # poll for console.log messages from our webworker
+ # which are the output of pytest
+ lines = self.driver.execute_script(
+ "let temp = window.python_logs;window.python_logs=[];return temp;"
+ )
+ if len(lines) > 0:
+ sys.stdout.buffer.write(bytes(lines))
+ done = self.driver.execute_script("return window.python_script_done")
+ if done is not None:
+ value = done["result"]
+ self.driver.execute_script("delete window.python_script_done")
+ return value
+ time.sleep(0.1)
+
+
+class ChromeDriver(BrowserDriver):
+ def __init__(self, hostname, port):
+ from selenium.webdriver.chrome.options import Options
+
+ options = Options()
+ options.add_argument("--headless")
+ options.add_argument("--no-sandbox")
+ super().__init__(hostname, port, webdriver.Chrome(options=options))
+
+
+class FirefoxDriver(BrowserDriver):
+ def __init__(self, hostname, port):
+ from selenium.webdriver.firefox.options import Options
+
+ options = Options()
+ options.add_argument("--headless")
+
+ super().__init__(hostname, port, webdriver.Firefox(options=options))
+
+
+def _load_pyarrow_in_runner(driver, wheel_name):
+ driver.load_arrow()
+ driver.execute_python(
+ """import sys
+import micropip
+if "pyarrow" not in sys.modules:
+ await micropip.install("hypothesis")
+ import pyodide_js as pjs
+ await pjs.loadPackage("numpy")
+ await pjs.loadPackage("pandas")
+ import pytest
+ import pandas # import pandas after pyarrow package load for pandas/pyarrow
+ # functions to work
+import pyarrow
+ """,
+ wait_for_terminate=True,
+ )
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument(
+ "-d",
+ "--dist-dir",
+ type=str,
+ help="Pyodide distribution directory",
+ default="./pyodide",
+)
+parser.add_argument("wheel", type=str, help="Wheel to run tests from")
+parser.add_argument(
+ "-t", "--test-submodule", help="Submodule that tests live in", default="test"
+)
+parser.add_argument(
+ "-r",
+ "--runtime",
+ type=str,
+ choices=["chrome", "node", "firefox"],
+ help="Runtime to run tests in ",
+ default="chrome",
+)
+args = parser.parse_args()
+
+PYARROW_WHEEL_PATH = Path(args.wheel).resolve()
+
+dist_dir = Path(os.getcwd(), args.dist_dir).resolve()
+print(f"dist dir={dist_dir}")
+with launch_server(dist_dir) as (hostname, port):
+ if args.runtime == "chrome":
+ driver = ChromeDriver(hostname, port)
+ elif args.runtime == "node":
+ driver = NodeDriver(hostname, port)
+ elif args.runtime == "firefox":
+ driver = FirefoxDriver(hostname, port)
+
+ print("Load pyodide in browser")
+ driver.load_pyodide(dist_dir)
+ print("Done\n")
+
+ print("Load pyarrow in browser")
+ _load_pyarrow_in_runner(driver, Path(args.wheel).name)
+ print("Done\n")
+ driver.clear_logs()
+ print("Run pytest in browser")
+ driver.execute_python(
+ """
+import pyarrow,pathlib
+pyarrow_dir = pathlib.Path(pyarrow.__file__).parent
+pytest.main([pyarrow_dir,'-v'])
Review Comment:
```suggestion
pytest.main([pyarrow_dir, '-v'])
```
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
+ if('results' in evt.data){
+ if(window.python_done_callback){
+ let callback=window.python_done_callback;
+ window.python_done_callback=undefined;
+ callback({result:evt.data.results});
+ }
+ }
+ if('print' in evt.data){
+ evt.data.print.forEach((x)=>{window.python_logs.push(x)});
+ }
+ }
+ window.pyworker = new Worker("worker.js");
+ window.pyworker.onmessage=capturelogs;
+ </script>
+ </head>
+ <body></body>
+ </html>
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "text/html")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+ elif self.path.endswith("/worker.js"):
+ body = b"""
+ importScripts("./pyodide.js");
+ onmessage = async function (e) {
+ const data = e.data;
+ if(!self.pyodide){
+ self.pyodide = await loadPyodide()
+ }
+ function do_print(arg){
+ let databytes = Array.from(arg)
+ self.postMessage({print:databytes})
+ return databytes.length
+ }
+ self.pyodide.setStdout({write:do_print,isatty:data.isatty});
+ self.pyodide.setStderr({write:do_print,isatty:data.isatty});
+
+ await self.pyodide.loadPackagesFromImports(data.python);
+ let results = await self.pyodide.runPythonAsync(data.python);
+ self.postMessage({results});
+ console.log('FINISHED_WEBWORKER')
+ }
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "application/javascript")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+
+ else:
+ return super().do_GET()
+
+ def end_headers(self):
+ # Enable Cross-Origin Resource Sharing (CORS)
+ self.send_header("Access-Control-Allow-Origin", "*")
+ super().end_headers()
+
+
+def run_server_thread(dist_dir, q):
+ global _SERVER_ADDRESS
+ os.chdir(dist_dir)
+ server = http.server.HTTPServer(("", 0), TemplateOverrider)
+ q.put(server.server_address)
+ print(f"Starting server for {dist_dir} at: {server.server_address}")
+ server.serve_forever()
+
+
+@contextlib.contextmanager
+def launch_server(dist_dir):
+ q = queue.Queue()
+ p = threading.Thread(target=run_server_thread, args=[dist_dir, q], daemon=True)
+ p.start()
+ address = q.get(timeout=50)
+ time.sleep(0.1) # wait to make sure server is started
+ yield address
+ p.terminate()
+
+
+class NodeDriver:
+ import subprocess
+
+ def __init__(self, hostname, port):
+ self.process = subprocess.Popen(
+ [shutil.which("script"), "-c", f'"{shutil.which("node")}"'],
+ stdin=subprocess.PIPE,
+ shell=False,
+ bufsize=0,
+ )
+ print(self.process)
+ time.sleep(0.1) # wait for node to start
+ self.hostname = hostname
+ self.port = port
+ self.last_ret_code = None
+
+ def load_pyodide(self, dist_dir):
+ self.execute_js(
+ f"""
+ const {{ loadPyodide }} = require('{dist_dir}/pyodide.js')
+ let pyodide = await loadPyodide()
+ """
+ )
+
+ def clear_logs(self):
+ pass # we don't handle logs for node
+
+ def write_stdin(self, buffer):
+ # because we use unbuffered IO for
+ # stdout, stdin.write is also unbuffered
+ # so might under-run on writes
+ while len(buffer) > 0 and self.process.poll() is None:
+ written = self.process.stdin.write(buffer)
+ if written == len(buffer):
+ break
+ elif written == 0:
+ # full buffer - wait
+ time.sleep(0.01)
+ else:
+ buffer = buffer[written:]
+
+ def execute_js(self, code, wait_for_terminate=True):
+ self.write_stdin((code + "\n").encode("utf-8"))
+
+ def load_arrow(self):
+ self.execute_js(f"await pyodide.loadPackage('{PYARROW_WHEEL_PATH}')")
+
+ def execute_python(self, code, wait_for_terminate=True):
+ js_code = f"""
+ python = `{code}`;
+ await pyodide.loadPackagesFromImports(python);
+ python_output = await pyodide.runPythonAsync(python);
+ """
+ self.last_ret_code = self.execute_js(js_code, wait_for_terminate)
+ return self.last_ret_code
+
+ def wait_for_done(self):
+ # in node we just let it run above
+ # then send EOF and join process
+ self.write_stdin(b"process.exit(python_output)\n")
+ return self.process.wait()
+
+
+class BrowserDriver:
+ def __init__(self, hostname, port, driver):
+ self.driver = driver
+ self.driver.get(f"http://{hostname}:{port}/test.html")
+ self.driver.set_script_timeout(100)
+
+ def load_pyodide(self, dist_dir):
+ pass
+
+ def load_arrow(self):
+ self.execute_python(
+ f"import pyodide_js as pjs\n"
+ f"await pjs.loadPackage('{PYARROW_WHEEL_PATH.name}')\n"
+ )
+
+ def execute_python(self, code, wait_for_terminate=True):
+ if wait_for_terminate:
+ self.driver.execute_async_script(
+ f"""
+ let callback=arguments[arguments.length-1];
+ python = `{code}`;
+ window.python_done_callback=callback
+ window.pyworker.postMessage(
+ {{python,isatty:{'true' if sys.stdout.isatty() else 'false'}}})
+ """
+ )
+ else:
+ self.driver.execute_script(
+ f"""
+ let python = `{code}`;
+ window.python_done_callback= (x) => {{window.python_script_done=x;}};
+ window.pyworker.postMessage(
+ {{python,isatty:{'true' if sys.stdout.isatty() else 'false'}}});
+ """
+ )
+
+ def clear_logs(self):
+ self.driver.execute_script("window.python_logs = []")
+
+ def wait_for_done(self):
+ while True:
+ # poll for console.log messages from our webworker
+ # which are the output of pytest
+ lines = self.driver.execute_script(
+ "let temp = window.python_logs;window.python_logs=[];return temp;"
+ )
+ if len(lines) > 0:
+ sys.stdout.buffer.write(bytes(lines))
+ done = self.driver.execute_script("return window.python_script_done")
+ if done is not None:
+ value = done["result"]
+ self.driver.execute_script("delete window.python_script_done")
+ return value
+ time.sleep(0.1)
+
+
+class ChromeDriver(BrowserDriver):
+ def __init__(self, hostname, port):
+ from selenium.webdriver.chrome.options import Options
+
+ options = Options()
+ options.add_argument("--headless")
+ options.add_argument("--no-sandbox")
+ super().__init__(hostname, port, webdriver.Chrome(options=options))
+
+
+class FirefoxDriver(BrowserDriver):
+ def __init__(self, hostname, port):
+ from selenium.webdriver.firefox.options import Options
+
+ options = Options()
+ options.add_argument("--headless")
+
+ super().__init__(hostname, port, webdriver.Firefox(options=options))
+
+
+def _load_pyarrow_in_runner(driver, wheel_name):
+ driver.load_arrow()
+ driver.execute_python(
+ """import sys
+import micropip
+if "pyarrow" not in sys.modules:
+ await micropip.install("hypothesis")
+ import pyodide_js as pjs
+ await pjs.loadPackage("numpy")
+ await pjs.loadPackage("pandas")
+ import pytest
+ import pandas # import pandas after pyarrow package load for pandas/pyarrow
+ # functions to work
+import pyarrow
+ """,
+ wait_for_terminate=True,
+ )
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument(
+ "-d",
+ "--dist-dir",
+ type=str,
+ help="Pyodide distribution directory",
+ default="./pyodide",
+)
+parser.add_argument("wheel", type=str, help="Wheel to run tests from")
+parser.add_argument(
+ "-t", "--test-submodule", help="Submodule that tests live in", default="test"
+)
+parser.add_argument(
+ "-r",
+ "--runtime",
+ type=str,
+ choices=["chrome", "node", "firefox"],
+ help="Runtime to run tests in ",
+ default="chrome",
+)
+args = parser.parse_args()
+
+PYARROW_WHEEL_PATH = Path(args.wheel).resolve()
+
+dist_dir = Path(os.getcwd(), args.dist_dir).resolve()
+print(f"dist dir={dist_dir}")
+with launch_server(dist_dir) as (hostname, port):
+ if args.runtime == "chrome":
+ driver = ChromeDriver(hostname, port)
+ elif args.runtime == "node":
+ driver = NodeDriver(hostname, port)
+ elif args.runtime == "firefox":
+ driver = FirefoxDriver(hostname, port)
+
+ print("Load pyodide in browser")
+ driver.load_pyodide(dist_dir)
+ print("Done\n")
Review Comment:
Is `\n` needed?
##########
python/pyarrow/conftest.py:
##########
@@ -116,7 +142,13 @@
try:
import pyarrow.orc # noqa
- defaults['orc'] = True
+ if sys.platform != "win32":
+ # orc tests on non-Windows platforms only work
+ # if timezone data exists, so skip them if
+ # not.
+ defaults['orc'] = defaults['timezone_data']
+ else:
+ defaults['orc'] = True
Review Comment:
`if XXX == YYY: else:` is easier to read than `if XXX != YYY: else:`:
```suggestion
if sys.platform == "win32":
defaults['orc'] = True
else:
# orc tests on non-Windows platforms only work
# if timezone data exists, so skip them if
# not.
defaults['orc'] = defaults['timezone_data']
```
##########
python/pyarrow/io.pxi:
##########
@@ -733,11 +734,66 @@ cdef class NativeFile(_Weakrefable):
finally:
free(buf)
done = True
-
writer_thread.join()
if exc_info is not None:
raise exc_info[0], exc_info[1], exc_info[2]
+ def _download_nothreads(self, stream_or_path, buffer_size=None):
+ """
+ Internal method to do a download without separate threads, queues etc.
+ Called by download above if is_threading_enabled() == False
+ """
+ cdef:
+ int64_t bytes_read = 0
+ uint8_t* buf
+
+ handle = self.get_input_stream()
+
+ buffer_size = buffer_size or DEFAULT_BUFFER_SIZE
+
+ if not hasattr(stream_or_path, 'read'):
+ stream = open(stream_or_path, 'wb')
+
+ def cleanup():
+ stream.close()
Review Comment:
It seems that this isn't called.
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
+ if('results' in evt.data){
+ if(window.python_done_callback){
Review Comment:
```suggestion
if (window.python_done_callback) {
```
##########
python/pyarrow/tests/test_io.py:
##########
@@ -1334,6 +1336,9 @@ def test_native_file_modes(tmpdir):
assert f.seekable()
+@pytest.mark.xfail(
Review Comment:
Should we use `skip` here?
Is there any plan that Emscripten supports umask?
##########
python/pyarrow/io.pxi:
##########
@@ -733,11 +734,66 @@ cdef class NativeFile(_Weakrefable):
finally:
free(buf)
done = True
-
Review Comment:
Could you revert a needless change?
##########
python/pyarrow/tests/test_dataset.py:
##########
@@ -808,29 +809,35 @@ def test_parquet_scan_options():
assert opts1.use_buffered_stream is False
assert opts1.buffer_size == 2**13
- assert opts1.pre_buffer is True
+
Review Comment:
```suggestion
```
##########
python/pyarrow/io.pxi:
##########
@@ -33,7 +33,6 @@ from queue import Queue, Empty as QueueEmpty
from pyarrow.lib cimport check_status, HaveLibHdfs
from pyarrow.util import _is_path_like, _stringify_path
-
Review Comment:
Could you revert a needless change?
##########
python/pyarrow/io.pxi:
##########
@@ -788,11 +847,28 @@ cdef class NativeFile(_Weakrefable):
write_queue.put_nowait(buf)
finally:
done = True
-
Review Comment:
Could you revert a needless change?
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
+ if('results' in evt.data){
+ if(window.python_done_callback){
+ let callback=window.python_done_callback;
+ window.python_done_callback=undefined;
+ callback({result:evt.data.results});
+ }
+ }
+ if('print' in evt.data){
+ evt.data.print.forEach((x)=>{window.python_logs.push(x)});
+ }
+ }
+ window.pyworker = new Worker("worker.js");
+ window.pyworker.onmessage=capturelogs;
+ </script>
+ </head>
+ <body></body>
+ </html>
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "text/html")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+ elif self.path.endswith("/worker.js"):
+ body = b"""
+ importScripts("./pyodide.js");
+ onmessage = async function (e) {
+ const data = e.data;
+ if(!self.pyodide){
+ self.pyodide = await loadPyodide()
Review Comment:
```suggestion
self.pyodide = await loadPyodide();
```
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
+ if('results' in evt.data){
+ if(window.python_done_callback){
+ let callback=window.python_done_callback;
+ window.python_done_callback=undefined;
+ callback({result:evt.data.results});
+ }
+ }
+ if('print' in evt.data){
Review Comment:
```suggestion
if ('print' in evt.data) {
```
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
Review Comment:
Should we close an opened file explicitly?
```suggestion
with PYARROW_WHEEL_PATH.open(mode="rb") as wheel:
self.copyfile(wheel, self.wfile)
```
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
Review Comment:
```suggestion
function capturelogs(evt) {
```
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
+ if('results' in evt.data){
+ if(window.python_done_callback){
+ let callback=window.python_done_callback;
+ window.python_done_callback=undefined;
+ callback({result:evt.data.results});
+ }
+ }
+ if('print' in evt.data){
+ evt.data.print.forEach((x)=>{window.python_logs.push(x)});
+ }
+ }
+ window.pyworker = new Worker("worker.js");
+ window.pyworker.onmessage=capturelogs;
Review Comment:
```suggestion
window.pyworker.onmessage = capturelogs;
```
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
+ if('results' in evt.data){
+ if(window.python_done_callback){
+ let callback=window.python_done_callback;
+ window.python_done_callback=undefined;
+ callback({result:evt.data.results});
+ }
+ }
+ if('print' in evt.data){
+ evt.data.print.forEach((x)=>{window.python_logs.push(x)});
+ }
+ }
+ window.pyworker = new Worker("worker.js");
+ window.pyworker.onmessage=capturelogs;
+ </script>
+ </head>
+ <body></body>
+ </html>
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "text/html")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+ elif self.path.endswith("/worker.js"):
+ body = b"""
+ importScripts("./pyodide.js");
+ onmessage = async function (e) {
+ const data = e.data;
+ if(!self.pyodide){
Review Comment:
```suggestion
if (!self.pyodide) {
```
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
+ if('results' in evt.data){
+ if(window.python_done_callback){
+ let callback=window.python_done_callback;
+ window.python_done_callback=undefined;
+ callback({result:evt.data.results});
+ }
+ }
+ if('print' in evt.data){
+ evt.data.print.forEach((x)=>{window.python_logs.push(x)});
+ }
+ }
+ window.pyworker = new Worker("worker.js");
+ window.pyworker.onmessage=capturelogs;
+ </script>
+ </head>
+ <body></body>
+ </html>
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "text/html")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+ elif self.path.endswith("/worker.js"):
+ body = b"""
+ importScripts("./pyodide.js");
+ onmessage = async function (e) {
+ const data = e.data;
+ if(!self.pyodide){
+ self.pyodide = await loadPyodide()
+ }
+ function do_print(arg){
Review Comment:
```suggestion
function do_print(arg) {
```
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
+ if('results' in evt.data){
+ if(window.python_done_callback){
+ let callback=window.python_done_callback;
Review Comment:
```suggestion
let callback = window.python_done_callback;
```
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
+ if('results' in evt.data){
+ if(window.python_done_callback){
+ let callback=window.python_done_callback;
+ window.python_done_callback=undefined;
+ callback({result:evt.data.results});
+ }
+ }
+ if('print' in evt.data){
+ evt.data.print.forEach((x)=>{window.python_logs.push(x)});
+ }
+ }
+ window.pyworker = new Worker("worker.js");
+ window.pyworker.onmessage=capturelogs;
+ </script>
+ </head>
+ <body></body>
+ </html>
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "text/html")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+ elif self.path.endswith("/worker.js"):
+ body = b"""
+ importScripts("./pyodide.js");
+ onmessage = async function (e) {
+ const data = e.data;
+ if(!self.pyodide){
+ self.pyodide = await loadPyodide()
+ }
+ function do_print(arg){
+ let databytes = Array.from(arg)
+ self.postMessage({print:databytes})
+ return databytes.length
+ }
+ self.pyodide.setStdout({write:do_print,isatty:data.isatty});
+ self.pyodide.setStderr({write:do_print,isatty:data.isatty});
+
+ await self.pyodide.loadPackagesFromImports(data.python);
+ let results = await self.pyodide.runPythonAsync(data.python);
+ self.postMessage({results});
+ console.log('FINISHED_WEBWORKER')
Review Comment:
Do we need this?
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
+ if('results' in evt.data){
+ if(window.python_done_callback){
+ let callback=window.python_done_callback;
+ window.python_done_callback=undefined;
+ callback({result:evt.data.results});
+ }
+ }
+ if('print' in evt.data){
+ evt.data.print.forEach((x)=>{window.python_logs.push(x)});
+ }
+ }
+ window.pyworker = new Worker("worker.js");
+ window.pyworker.onmessage=capturelogs;
+ </script>
+ </head>
+ <body></body>
+ </html>
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "text/html")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+ elif self.path.endswith("/worker.js"):
+ body = b"""
+ importScripts("./pyodide.js");
+ onmessage = async function (e) {
+ const data = e.data;
+ if(!self.pyodide){
+ self.pyodide = await loadPyodide()
+ }
+ function do_print(arg){
+ let databytes = Array.from(arg)
+ self.postMessage({print:databytes})
+ return databytes.length
+ }
+ self.pyodide.setStdout({write:do_print,isatty:data.isatty});
+ self.pyodide.setStderr({write:do_print,isatty:data.isatty});
+
+ await self.pyodide.loadPackagesFromImports(data.python);
+ let results = await self.pyodide.runPythonAsync(data.python);
+ self.postMessage({results});
+ console.log('FINISHED_WEBWORKER')
Review Comment:
```suggestion
console.log('FINISHED_WEBWORKER');
```
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
+ if('results' in evt.data){
+ if(window.python_done_callback){
+ let callback=window.python_done_callback;
+ window.python_done_callback=undefined;
+ callback({result:evt.data.results});
+ }
+ }
+ if('print' in evt.data){
+ evt.data.print.forEach((x)=>{window.python_logs.push(x)});
+ }
+ }
+ window.pyworker = new Worker("worker.js");
+ window.pyworker.onmessage=capturelogs;
+ </script>
+ </head>
+ <body></body>
+ </html>
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "text/html")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+ elif self.path.endswith("/worker.js"):
+ body = b"""
+ importScripts("./pyodide.js");
+ onmessage = async function (e) {
+ const data = e.data;
+ if(!self.pyodide){
+ self.pyodide = await loadPyodide()
+ }
+ function do_print(arg){
+ let databytes = Array.from(arg)
+ self.postMessage({print:databytes})
+ return databytes.length
+ }
+ self.pyodide.setStdout({write:do_print,isatty:data.isatty});
+ self.pyodide.setStderr({write:do_print,isatty:data.isatty});
+
+ await self.pyodide.loadPackagesFromImports(data.python);
+ let results = await self.pyodide.runPythonAsync(data.python);
+ self.postMessage({results});
+ console.log('FINISHED_WEBWORKER')
+ }
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "application/javascript")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+
+ else:
+ return super().do_GET()
+
+ def end_headers(self):
+ # Enable Cross-Origin Resource Sharing (CORS)
+ self.send_header("Access-Control-Allow-Origin", "*")
+ super().end_headers()
+
+
+def run_server_thread(dist_dir, q):
+ global _SERVER_ADDRESS
+ os.chdir(dist_dir)
+ server = http.server.HTTPServer(("", 0), TemplateOverrider)
+ q.put(server.server_address)
+ print(f"Starting server for {dist_dir} at: {server.server_address}")
+ server.serve_forever()
+
+
+@contextlib.contextmanager
+def launch_server(dist_dir):
+ q = queue.Queue()
+ p = threading.Thread(target=run_server_thread, args=[dist_dir, q], daemon=True)
+ p.start()
+ address = q.get(timeout=50)
+ time.sleep(0.1) # wait to make sure server is started
+ yield address
+ p.terminate()
+
+
+class NodeDriver:
+ import subprocess
+
+ def __init__(self, hostname, port):
+ self.process = subprocess.Popen(
+ [shutil.which("script"), "-c", f'"{shutil.which("node")}"'],
Review Comment:
Do we need to quote `shutil.which("node")`? I think that `Popen()` handles arguments that include spaces automatically.
```suggestion
[shutil.which("script"), "-c", shutil.which("node")],
```
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
+ if('results' in evt.data){
+ if(window.python_done_callback){
+ let callback=window.python_done_callback;
+ window.python_done_callback=undefined;
+ callback({result:evt.data.results});
+ }
+ }
+ if('print' in evt.data){
+ evt.data.print.forEach((x)=>{window.python_logs.push(x)});
+ }
+ }
+ window.pyworker = new Worker("worker.js");
+ window.pyworker.onmessage=capturelogs;
+ </script>
+ </head>
+ <body></body>
+ </html>
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "text/html")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+ elif self.path.endswith("/worker.js"):
+ body = b"""
+ importScripts("./pyodide.js");
+ onmessage = async function (e) {
+ const data = e.data;
+ if(!self.pyodide){
+ self.pyodide = await loadPyodide()
+ }
+ function do_print(arg){
+ let databytes = Array.from(arg)
+ self.postMessage({print:databytes})
+ return databytes.length
Review Comment:
```suggestion
let databytes = Array.from(arg);
self.postMessage({print:databytes});
return databytes.length;
```
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
+ if('results' in evt.data){
+ if(window.python_done_callback){
+ let callback=window.python_done_callback;
+ window.python_done_callback=undefined;
+ callback({result:evt.data.results});
+ }
+ }
+ if('print' in evt.data){
+ evt.data.print.forEach((x)=>{window.python_logs.push(x)});
+ }
+ }
+ window.pyworker = new Worker("worker.js");
+ window.pyworker.onmessage=capturelogs;
+ </script>
+ </head>
+ <body></body>
+ </html>
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "text/html")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+ elif self.path.endswith("/worker.js"):
+ body = b"""
+ importScripts("./pyodide.js");
+ onmessage = async function (e) {
+ const data = e.data;
+ if(!self.pyodide){
+ self.pyodide = await loadPyodide()
+ }
+ function do_print(arg){
+ let databytes = Array.from(arg)
+ self.postMessage({print:databytes})
+ return databytes.length
+ }
+ self.pyodide.setStdout({write:do_print,isatty:data.isatty});
+ self.pyodide.setStderr({write:do_print,isatty:data.isatty});
+
+ await self.pyodide.loadPackagesFromImports(data.python);
+ let results = await self.pyodide.runPythonAsync(data.python);
+ self.postMessage({results});
+ console.log('FINISHED_WEBWORKER')
+ }
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "application/javascript")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+
+ else:
+ return super().do_GET()
+
+ def end_headers(self):
+ # Enable Cross-Origin Resource Sharing (CORS)
+ self.send_header("Access-Control-Allow-Origin", "*")
+ super().end_headers()
+
+
+def run_server_thread(dist_dir, q):
+ global _SERVER_ADDRESS
+ os.chdir(dist_dir)
+ server = http.server.HTTPServer(("", 0), TemplateOverrider)
+ q.put(server.server_address)
+ print(f"Starting server for {dist_dir} at: {server.server_address}")
+ server.serve_forever()
+
+
+@contextlib.contextmanager
+def launch_server(dist_dir):
+ q = queue.Queue()
+ p = threading.Thread(target=run_server_thread, args=[dist_dir, q], daemon=True)
+ p.start()
+ address = q.get(timeout=50)
+ time.sleep(0.1) # wait to make sure server is started
+ yield address
+ p.terminate()
+
+
+class NodeDriver:
+ import subprocess
+
+ def __init__(self, hostname, port):
+ self.process = subprocess.Popen(
+ [shutil.which("script"), "-c", f'"{shutil.which("node")}"'],
+ stdin=subprocess.PIPE,
+ shell=False,
+ bufsize=0,
+ )
+ print(self.process)
+ time.sleep(0.1) # wait for node to start
+ self.hostname = hostname
+ self.port = port
+ self.last_ret_code = None
+
+ def load_pyodide(self, dist_dir):
+ self.execute_js(
+ f"""
+ const {{ loadPyodide }} = require('{dist_dir}/pyodide.js')
+ let pyodide = await loadPyodide()
Review Comment:
```suggestion
const {{ loadPyodide }} = require('{dist_dir}/pyodide.js');
let pyodide = await loadPyodide();
```
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
+ if('results' in evt.data){
+ if(window.python_done_callback){
+ let callback=window.python_done_callback;
+ window.python_done_callback=undefined;
+ callback({result:evt.data.results});
+ }
+ }
+ if('print' in evt.data){
+ evt.data.print.forEach((x)=>{window.python_logs.push(x)});
+ }
+ }
+ window.pyworker = new Worker("worker.js");
+ window.pyworker.onmessage=capturelogs;
+ </script>
+ </head>
+ <body></body>
+ </html>
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "text/html")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+ elif self.path.endswith("/worker.js"):
+ body = b"""
+ importScripts("./pyodide.js");
+ onmessage = async function (e) {
+ const data = e.data;
+ if(!self.pyodide){
+ self.pyodide = await loadPyodide()
+ }
+ function do_print(arg){
+ let databytes = Array.from(arg)
+ self.postMessage({print:databytes})
+ return databytes.length
+ }
+ self.pyodide.setStdout({write:do_print,isatty:data.isatty});
+ self.pyodide.setStderr({write:do_print,isatty:data.isatty});
+
+ await self.pyodide.loadPackagesFromImports(data.python);
+ let results = await self.pyodide.runPythonAsync(data.python);
+ self.postMessage({results});
+ console.log('FINISHED_WEBWORKER')
+ }
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "application/javascript")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+
+ else:
+ return super().do_GET()
+
+ def end_headers(self):
+ # Enable Cross-Origin Resource Sharing (CORS)
+ self.send_header("Access-Control-Allow-Origin", "*")
+ super().end_headers()
+
+
+def run_server_thread(dist_dir, q):
+ global _SERVER_ADDRESS
+ os.chdir(dist_dir)
+ server = http.server.HTTPServer(("", 0), TemplateOverrider)
+ q.put(server.server_address)
+ print(f"Starting server for {dist_dir} at: {server.server_address}")
+ server.serve_forever()
+
+
+@contextlib.contextmanager
+def launch_server(dist_dir):
+ q = queue.Queue()
+ p = threading.Thread(target=run_server_thread, args=[dist_dir, q], daemon=True)
+ p.start()
+ address = q.get(timeout=50)
+ time.sleep(0.1) # wait to make sure server is started
+ yield address
+ p.terminate()
+
+
+class NodeDriver:
+ import subprocess
+
+ def __init__(self, hostname, port):
+ self.process = subprocess.Popen(
+ [shutil.which("script"), "-c", f'"{shutil.which("node")}"'],
+ stdin=subprocess.PIPE,
+ shell=False,
+ bufsize=0,
+ )
+ print(self.process)
+ time.sleep(0.1) # wait for node to start
+ self.hostname = hostname
+ self.port = port
+ self.last_ret_code = None
+
+ def load_pyodide(self, dist_dir):
+ self.execute_js(
+ f"""
+ const {{ loadPyodide }} = require('{dist_dir}/pyodide.js')
+ let pyodide = await loadPyodide()
+ """
+ )
+
+ def clear_logs(self):
+ pass # we don't handle logs for node
+
+ def write_stdin(self, buffer):
+ # because we use unbuffered IO for
+ # stdout, stdin.write is also unbuffered
+ # so might under-run on writes
+ while len(buffer) > 0 and self.process.poll() is None:
+ written = self.process.stdin.write(buffer)
+ if written == len(buffer):
+ break
+ elif written == 0:
+ # full buffer - wait
+ time.sleep(0.01)
+ else:
+ buffer = buffer[written:]
+
+ def execute_js(self, code, wait_for_terminate=True):
+ self.write_stdin((code + "\n").encode("utf-8"))
+
+ def load_arrow(self):
+ self.execute_js(f"await pyodide.loadPackage('{PYARROW_WHEEL_PATH}')")
+
+ def execute_python(self, code, wait_for_terminate=True):
+ js_code = f"""
+ python = `{code}`;
+ await pyodide.loadPackagesFromImports(python);
+ python_output = await pyodide.runPythonAsync(python);
+ """
+ self.last_ret_code = self.execute_js(js_code, wait_for_terminate)
+ return self.last_ret_code
+
+ def wait_for_done(self):
+ # in node we just let it run above
+ # then send EOF and join process
+ self.write_stdin(b"process.exit(python_output)\n")
+ return self.process.wait()
+
+
+class BrowserDriver:
+ def __init__(self, hostname, port, driver):
+ self.driver = driver
+ self.driver.get(f"http://{hostname}:{port}/test.html")
+ self.driver.set_script_timeout(100)
+
+ def load_pyodide(self, dist_dir):
+ pass
+
+ def load_arrow(self):
+ self.execute_python(
+ f"import pyodide_js as pjs\n"
+ f"await pjs.loadPackage('{PYARROW_WHEEL_PATH.name}')\n"
+ )
+
+ def execute_python(self, code, wait_for_terminate=True):
+ if wait_for_terminate:
+ self.driver.execute_async_script(
+ f"""
+ let callback=arguments[arguments.length-1];
+ python = `{code}`;
+ window.python_done_callback=callback
+ window.pyworker.postMessage(
+ {{python,isatty:{'true' if sys.stdout.isatty() else 'false'}}})
+ """
+ )
+ else:
+ self.driver.execute_script(
+ f"""
+ let python = `{code}`;
+ window.python_done_callback= (x) => {{window.python_script_done=x;}};
+ window.pyworker.postMessage(
+ {{python,isatty:{'true' if sys.stdout.isatty() else 'false'}}});
+ """
+ )
+
+ def clear_logs(self):
+ self.driver.execute_script("window.python_logs = []")
Review Comment:
```suggestion
self.driver.execute_script("window.python_logs = [];")
```
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
+ if('results' in evt.data){
+ if(window.python_done_callback){
+ let callback=window.python_done_callback;
+ window.python_done_callback=undefined;
+ callback({result:evt.data.results});
+ }
+ }
+ if('print' in evt.data){
+ evt.data.print.forEach((x)=>{window.python_logs.push(x)});
+ }
+ }
+ window.pyworker = new Worker("worker.js");
+ window.pyworker.onmessage=capturelogs;
+ </script>
+ </head>
+ <body></body>
+ </html>
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "text/html")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+ elif self.path.endswith("/worker.js"):
+ body = b"""
+ importScripts("./pyodide.js");
+ onmessage = async function (e) {
+ const data = e.data;
+ if(!self.pyodide){
+ self.pyodide = await loadPyodide()
+ }
+ function do_print(arg){
+ let databytes = Array.from(arg)
+ self.postMessage({print:databytes})
+ return databytes.length
+ }
+ self.pyodide.setStdout({write:do_print,isatty:data.isatty});
+ self.pyodide.setStderr({write:do_print,isatty:data.isatty});
+
+ await self.pyodide.loadPackagesFromImports(data.python);
+ let results = await self.pyodide.runPythonAsync(data.python);
+ self.postMessage({results});
+ console.log('FINISHED_WEBWORKER')
+ }
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "application/javascript")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+
+ else:
+ return super().do_GET()
+
+ def end_headers(self):
+ # Enable Cross-Origin Resource Sharing (CORS)
+ self.send_header("Access-Control-Allow-Origin", "*")
+ super().end_headers()
+
+
+def run_server_thread(dist_dir, q):
+ global _SERVER_ADDRESS
+ os.chdir(dist_dir)
+ server = http.server.HTTPServer(("", 0), TemplateOverrider)
+ q.put(server.server_address)
+ print(f"Starting server for {dist_dir} at: {server.server_address}")
+ server.serve_forever()
+
+
+@contextlib.contextmanager
+def launch_server(dist_dir):
+ q = queue.Queue()
+ p = threading.Thread(target=run_server_thread, args=[dist_dir, q], daemon=True)
+ p.start()
+ address = q.get(timeout=50)
+ time.sleep(0.1) # wait to make sure server is started
+ yield address
+ p.terminate()
+
+
+class NodeDriver:
+ import subprocess
+
+ def __init__(self, hostname, port):
+ self.process = subprocess.Popen(
+ [shutil.which("script"), "-c", f'"{shutil.which("node")}"'],
+ stdin=subprocess.PIPE,
+ shell=False,
+ bufsize=0,
+ )
+ print(self.process)
+ time.sleep(0.1) # wait for node to start
+ self.hostname = hostname
+ self.port = port
+ self.last_ret_code = None
+
+ def load_pyodide(self, dist_dir):
+ self.execute_js(
+ f"""
+ const {{ loadPyodide }} = require('{dist_dir}/pyodide.js')
+ let pyodide = await loadPyodide()
+ """
+ )
+
+ def clear_logs(self):
+ pass # we don't handle logs for node
+
+ def write_stdin(self, buffer):
+ # because we use unbuffered IO for
+ # stdout, stdin.write is also unbuffered
+ # so might under-run on writes
+ while len(buffer) > 0 and self.process.poll() is None:
+ written = self.process.stdin.write(buffer)
+ if written == len(buffer):
+ break
+ elif written == 0:
+ # full buffer - wait
+ time.sleep(0.01)
+ else:
+ buffer = buffer[written:]
+
+ def execute_js(self, code, wait_for_terminate=True):
+ self.write_stdin((code + "\n").encode("utf-8"))
+
+ def load_arrow(self):
+ self.execute_js(f"await pyodide.loadPackage('{PYARROW_WHEEL_PATH}')")
Review Comment:
```suggestion
self.execute_js(f"await pyodide.loadPackage('{PYARROW_WHEEL_PATH}');")
```
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
+ if('results' in evt.data){
+ if(window.python_done_callback){
+ let callback=window.python_done_callback;
+ window.python_done_callback=undefined;
+ callback({result:evt.data.results});
+ }
+ }
+ if('print' in evt.data){
+ evt.data.print.forEach((x)=>{window.python_logs.push(x)});
+ }
+ }
+ window.pyworker = new Worker("worker.js");
+ window.pyworker.onmessage=capturelogs;
+ </script>
+ </head>
+ <body></body>
+ </html>
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "text/html")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+ elif self.path.endswith("/worker.js"):
+ body = b"""
+ importScripts("./pyodide.js");
+ onmessage = async function (e) {
+ const data = e.data;
+ if(!self.pyodide){
+ self.pyodide = await loadPyodide()
+ }
+ function do_print(arg){
+ let databytes = Array.from(arg)
+ self.postMessage({print:databytes})
+ return databytes.length
+ }
+ self.pyodide.setStdout({write:do_print,isatty:data.isatty});
+ self.pyodide.setStderr({write:do_print,isatty:data.isatty});
+
+ await self.pyodide.loadPackagesFromImports(data.python);
+ let results = await self.pyodide.runPythonAsync(data.python);
+ self.postMessage({results});
+ console.log('FINISHED_WEBWORKER')
+ }
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "application/javascript")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+
+ else:
+ return super().do_GET()
+
+ def end_headers(self):
+ # Enable Cross-Origin Resource Sharing (CORS)
+ self.send_header("Access-Control-Allow-Origin", "*")
+ super().end_headers()
+
+
+def run_server_thread(dist_dir, q):
+ global _SERVER_ADDRESS
+ os.chdir(dist_dir)
+ server = http.server.HTTPServer(("", 0), TemplateOverrider)
+ q.put(server.server_address)
+ print(f"Starting server for {dist_dir} at: {server.server_address}")
+ server.serve_forever()
+
+
+@contextlib.contextmanager
+def launch_server(dist_dir):
+ q = queue.Queue()
+ p = threading.Thread(target=run_server_thread, args=[dist_dir, q], daemon=True)
+ p.start()
+ address = q.get(timeout=50)
+ time.sleep(0.1) # wait to make sure server is started
+ yield address
+ p.terminate()
+
+
+class NodeDriver:
+ import subprocess
+
+ def __init__(self, hostname, port):
+ self.process = subprocess.Popen(
+ [shutil.which("script"), "-c", f'"{shutil.which("node")}"'],
+ stdin=subprocess.PIPE,
+ shell=False,
+ bufsize=0,
+ )
+ print(self.process)
+ time.sleep(0.1) # wait for node to start
+ self.hostname = hostname
+ self.port = port
+ self.last_ret_code = None
+
+ def load_pyodide(self, dist_dir):
+ self.execute_js(
+ f"""
+ const {{ loadPyodide }} = require('{dist_dir}/pyodide.js')
+ let pyodide = await loadPyodide()
+ """
+ )
+
+ def clear_logs(self):
+ pass # we don't handle logs for node
+
+ def write_stdin(self, buffer):
+ # because we use unbuffered IO for
+ # stdout, stdin.write is also unbuffered
+ # so might under-run on writes
+ while len(buffer) > 0 and self.process.poll() is None:
+ written = self.process.stdin.write(buffer)
+ if written == len(buffer):
+ break
+ elif written == 0:
+ # full buffer - wait
+ time.sleep(0.01)
+ else:
+ buffer = buffer[written:]
+
+ def execute_js(self, code, wait_for_terminate=True):
+ self.write_stdin((code + "\n").encode("utf-8"))
+
+ def load_arrow(self):
+ self.execute_js(f"await pyodide.loadPackage('{PYARROW_WHEEL_PATH}')")
+
+ def execute_python(self, code, wait_for_terminate=True):
+ js_code = f"""
+ python = `{code}`;
+ await pyodide.loadPackagesFromImports(python);
+ python_output = await pyodide.runPythonAsync(python);
+ """
+ self.last_ret_code = self.execute_js(js_code, wait_for_terminate)
+ return self.last_ret_code
+
+ def wait_for_done(self):
+ # in node we just let it run above
+ # then send EOF and join process
+ self.write_stdin(b"process.exit(python_output)\n")
+ return self.process.wait()
+
+
+class BrowserDriver:
+ def __init__(self, hostname, port, driver):
+ self.driver = driver
+ self.driver.get(f"http://{hostname}:{port}/test.html")
+ self.driver.set_script_timeout(100)
+
+ def load_pyodide(self, dist_dir):
+ pass
+
+ def load_arrow(self):
+ self.execute_python(
+ f"import pyodide_js as pjs\n"
+ f"await pjs.loadPackage('{PYARROW_WHEEL_PATH.name}')\n"
+ )
+
+ def execute_python(self, code, wait_for_terminate=True):
+ if wait_for_terminate:
+ self.driver.execute_async_script(
+ f"""
+ let callback=arguments[arguments.length-1];
Review Comment:
```suggestion
let callback = arguments[arguments.length-1];
```
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
+ if('results' in evt.data){
+ if(window.python_done_callback){
+ let callback=window.python_done_callback;
+ window.python_done_callback=undefined;
+ callback({result:evt.data.results});
+ }
+ }
+ if('print' in evt.data){
+ evt.data.print.forEach((x)=>{window.python_logs.push(x)});
+ }
+ }
+ window.pyworker = new Worker("worker.js");
+ window.pyworker.onmessage=capturelogs;
+ </script>
+ </head>
+ <body></body>
+ </html>
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "text/html")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+ elif self.path.endswith("/worker.js"):
+ body = b"""
+ importScripts("./pyodide.js");
+ onmessage = async function (e) {
+ const data = e.data;
+ if(!self.pyodide){
+ self.pyodide = await loadPyodide()
+ }
+ function do_print(arg){
+ let databytes = Array.from(arg)
+ self.postMessage({print:databytes})
+ return databytes.length
+ }
+ self.pyodide.setStdout({write:do_print,isatty:data.isatty});
+ self.pyodide.setStderr({write:do_print,isatty:data.isatty});
+
+ await self.pyodide.loadPackagesFromImports(data.python);
+ let results = await self.pyodide.runPythonAsync(data.python);
+ self.postMessage({results});
+ console.log('FINISHED_WEBWORKER')
+ }
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "application/javascript")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+
+ else:
+ return super().do_GET()
+
+ def end_headers(self):
+ # Enable Cross-Origin Resource Sharing (CORS)
+ self.send_header("Access-Control-Allow-Origin", "*")
+ super().end_headers()
+
+
+def run_server_thread(dist_dir, q):
+ global _SERVER_ADDRESS
+ os.chdir(dist_dir)
+ server = http.server.HTTPServer(("", 0), TemplateOverrider)
+ q.put(server.server_address)
+ print(f"Starting server for {dist_dir} at: {server.server_address}")
+ server.serve_forever()
+
+
+@contextlib.contextmanager
+def launch_server(dist_dir):
+ q = queue.Queue()
+ p = threading.Thread(target=run_server_thread, args=[dist_dir, q], daemon=True)
+ p.start()
+ address = q.get(timeout=50)
+ time.sleep(0.1) # wait to make sure server is started
+ yield address
+ p.terminate()
+
+
+class NodeDriver:
+ import subprocess
+
+ def __init__(self, hostname, port):
+ self.process = subprocess.Popen(
+ [shutil.which("script"), "-c", f'"{shutil.which("node")}"'],
+ stdin=subprocess.PIPE,
+ shell=False,
+ bufsize=0,
+ )
+ print(self.process)
+ time.sleep(0.1) # wait for node to start
+ self.hostname = hostname
+ self.port = port
+ self.last_ret_code = None
+
+ def load_pyodide(self, dist_dir):
+ self.execute_js(
+ f"""
+ const {{ loadPyodide }} = require('{dist_dir}/pyodide.js')
+ let pyodide = await loadPyodide()
+ """
+ )
+
+ def clear_logs(self):
+ pass # we don't handle logs for node
+
+ def write_stdin(self, buffer):
+ # because we use unbuffered IO for
+ # stdout, stdin.write is also unbuffered
+ # so might under-run on writes
+ while len(buffer) > 0 and self.process.poll() is None:
+ written = self.process.stdin.write(buffer)
+ if written == len(buffer):
+ break
+ elif written == 0:
+ # full buffer - wait
+ time.sleep(0.01)
+ else:
+ buffer = buffer[written:]
+
+ def execute_js(self, code, wait_for_terminate=True):
+ self.write_stdin((code + "\n").encode("utf-8"))
+
+ def load_arrow(self):
+ self.execute_js(f"await pyodide.loadPackage('{PYARROW_WHEEL_PATH}')")
+
+ def execute_python(self, code, wait_for_terminate=True):
+ js_code = f"""
+ python = `{code}`;
+ await pyodide.loadPackagesFromImports(python);
+ python_output = await pyodide.runPythonAsync(python);
+ """
+ self.last_ret_code = self.execute_js(js_code, wait_for_terminate)
+ return self.last_ret_code
+
+ def wait_for_done(self):
+ # in node we just let it run above
+ # then send EOF and join process
+ self.write_stdin(b"process.exit(python_output)\n")
+ return self.process.wait()
+
+
+class BrowserDriver:
+ def __init__(self, hostname, port, driver):
+ self.driver = driver
+ self.driver.get(f"http://{hostname}:{port}/test.html")
+ self.driver.set_script_timeout(100)
+
+ def load_pyodide(self, dist_dir):
+ pass
+
+ def load_arrow(self):
+ self.execute_python(
+ f"import pyodide_js as pjs\n"
+ f"await pjs.loadPackage('{PYARROW_WHEEL_PATH.name}')\n"
+ )
+
+ def execute_python(self, code, wait_for_terminate=True):
+ if wait_for_terminate:
+ self.driver.execute_async_script(
+ f"""
+ let callback=arguments[arguments.length-1];
+ python = `{code}`;
+ window.python_done_callback=callback
+ window.pyworker.postMessage(
+ {{python,isatty:{'true' if sys.stdout.isatty() else 'false'}}})
+ """
+ )
+ else:
+ self.driver.execute_script(
+ f"""
+ let python = `{code}`;
+ window.python_done_callback= (x) => {{window.python_script_done=x;}};
+ window.pyworker.postMessage(
+ {{python,isatty:{'true' if sys.stdout.isatty() else 'false'}}});
+ """
+ )
+
+ def clear_logs(self):
+ self.driver.execute_script("window.python_logs = []")
+
+ def wait_for_done(self):
+ while True:
+ # poll for console.log messages from our webworker
+ # which are the output of pytest
+ lines = self.driver.execute_script(
+ "let temp = window.python_logs;window.python_logs=[];return temp;"
+ )
+ if len(lines) > 0:
+ sys.stdout.buffer.write(bytes(lines))
+ done = self.driver.execute_script("return window.python_script_done")
+ if done is not None:
+ value = done["result"]
+ self.driver.execute_script("delete window.python_script_done")
Review Comment:
```suggestion
self.driver.execute_script("delete window.python_script_done;")
```
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
+ if('results' in evt.data){
+ if(window.python_done_callback){
+ let callback=window.python_done_callback;
+ window.python_done_callback=undefined;
+ callback({result:evt.data.results});
+ }
+ }
+ if('print' in evt.data){
+ evt.data.print.forEach((x)=>{window.python_logs.push(x)});
+ }
+ }
+ window.pyworker = new Worker("worker.js");
+ window.pyworker.onmessage=capturelogs;
+ </script>
+ </head>
+ <body></body>
+ </html>
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "text/html")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+ elif self.path.endswith("/worker.js"):
+ body = b"""
+ importScripts("./pyodide.js");
+ onmessage = async function (e) {
+ const data = e.data;
+ if(!self.pyodide){
+ self.pyodide = await loadPyodide()
+ }
+ function do_print(arg){
+ let databytes = Array.from(arg)
+ self.postMessage({print:databytes})
+ return databytes.length
+ }
+ self.pyodide.setStdout({write:do_print,isatty:data.isatty});
+ self.pyodide.setStderr({write:do_print,isatty:data.isatty});
+
+ await self.pyodide.loadPackagesFromImports(data.python);
+ let results = await self.pyodide.runPythonAsync(data.python);
+ self.postMessage({results});
+ console.log('FINISHED_WEBWORKER')
+ }
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "application/javascript")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+
+ else:
+ return super().do_GET()
+
+ def end_headers(self):
+ # Enable Cross-Origin Resource Sharing (CORS)
+ self.send_header("Access-Control-Allow-Origin", "*")
+ super().end_headers()
+
+
+def run_server_thread(dist_dir, q):
+ global _SERVER_ADDRESS
+ os.chdir(dist_dir)
+ server = http.server.HTTPServer(("", 0), TemplateOverrider)
+ q.put(server.server_address)
+ print(f"Starting server for {dist_dir} at: {server.server_address}")
+ server.serve_forever()
+
+
+@contextlib.contextmanager
+def launch_server(dist_dir):
+ q = queue.Queue()
+ p = threading.Thread(target=run_server_thread, args=[dist_dir, q], daemon=True)
+ p.start()
+ address = q.get(timeout=50)
+ time.sleep(0.1) # wait to make sure server is started
+ yield address
+ p.terminate()
+
+
+class NodeDriver:
+ import subprocess
+
+ def __init__(self, hostname, port):
+ self.process = subprocess.Popen(
+ [shutil.which("script"), "-c", f'"{shutil.which("node")}"'],
+ stdin=subprocess.PIPE,
+ shell=False,
+ bufsize=0,
+ )
+ print(self.process)
+ time.sleep(0.1) # wait for node to start
+ self.hostname = hostname
+ self.port = port
+ self.last_ret_code = None
+
+ def load_pyodide(self, dist_dir):
+ self.execute_js(
+ f"""
+ const {{ loadPyodide }} = require('{dist_dir}/pyodide.js')
+ let pyodide = await loadPyodide()
+ """
+ )
+
+ def clear_logs(self):
+ pass # we don't handle logs for node
+
+ def write_stdin(self, buffer):
+ # because we use unbuffered IO for
+ # stdout, stdin.write is also unbuffered
+ # so might under-run on writes
+ while len(buffer) > 0 and self.process.poll() is None:
+ written = self.process.stdin.write(buffer)
+ if written == len(buffer):
+ break
+ elif written == 0:
+ # full buffer - wait
+ time.sleep(0.01)
+ else:
+ buffer = buffer[written:]
+
+ def execute_js(self, code, wait_for_terminate=True):
+ self.write_stdin((code + "\n").encode("utf-8"))
+
+ def load_arrow(self):
+ self.execute_js(f"await pyodide.loadPackage('{PYARROW_WHEEL_PATH}')")
+
+ def execute_python(self, code, wait_for_terminate=True):
+ js_code = f"""
+ python = `{code}`;
+ await pyodide.loadPackagesFromImports(python);
+ python_output = await pyodide.runPythonAsync(python);
+ """
+ self.last_ret_code = self.execute_js(js_code, wait_for_terminate)
+ return self.last_ret_code
+
+ def wait_for_done(self):
+ # in node we just let it run above
+ # then send EOF and join process
+ self.write_stdin(b"process.exit(python_output)\n")
+ return self.process.wait()
+
+
+class BrowserDriver:
+ def __init__(self, hostname, port, driver):
+ self.driver = driver
+ self.driver.get(f"http://{hostname}:{port}/test.html")
+ self.driver.set_script_timeout(100)
+
+ def load_pyodide(self, dist_dir):
+ pass
+
+ def load_arrow(self):
+ self.execute_python(
+ f"import pyodide_js as pjs\n"
+ f"await pjs.loadPackage('{PYARROW_WHEEL_PATH.name}')\n"
+ )
+
+ def execute_python(self, code, wait_for_terminate=True):
+ if wait_for_terminate:
+ self.driver.execute_async_script(
+ f"""
+ let callback=arguments[arguments.length-1];
+ python = `{code}`;
+ window.python_done_callback=callback
+ window.pyworker.postMessage(
+ {{python,isatty:{'true' if sys.stdout.isatty() else 'false'}}})
Review Comment:
```suggestion
window.python_done_callback = callback;
window.pyworker.postMessage(
{{python, isatty: {'true' if sys.stdout.isatty() else 'false'}}});
```
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
+ if('results' in evt.data){
+ if(window.python_done_callback){
+ let callback=window.python_done_callback;
+ window.python_done_callback=undefined;
+ callback({result:evt.data.results});
+ }
+ }
+ if('print' in evt.data){
+ evt.data.print.forEach((x)=>{window.python_logs.push(x)});
+ }
+ }
+ window.pyworker = new Worker("worker.js");
+ window.pyworker.onmessage=capturelogs;
+ </script>
+ </head>
+ <body></body>
+ </html>
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "text/html")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+ elif self.path.endswith("/worker.js"):
+ body = b"""
+ importScripts("./pyodide.js");
+ onmessage = async function (e) {
+ const data = e.data;
+ if(!self.pyodide){
+ self.pyodide = await loadPyodide()
+ }
+ function do_print(arg){
+ let databytes = Array.from(arg)
+ self.postMessage({print:databytes})
+ return databytes.length
+ }
+ self.pyodide.setStdout({write:do_print,isatty:data.isatty});
+ self.pyodide.setStderr({write:do_print,isatty:data.isatty});
+
+ await self.pyodide.loadPackagesFromImports(data.python);
+ let results = await self.pyodide.runPythonAsync(data.python);
+ self.postMessage({results});
+ console.log('FINISHED_WEBWORKER')
+ }
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "application/javascript")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+
+ else:
+ return super().do_GET()
+
+ def end_headers(self):
+ # Enable Cross-Origin Resource Sharing (CORS)
+ self.send_header("Access-Control-Allow-Origin", "*")
+ super().end_headers()
+
+
+def run_server_thread(dist_dir, q):
+ global _SERVER_ADDRESS
+ os.chdir(dist_dir)
+ server = http.server.HTTPServer(("", 0), TemplateOverrider)
+ q.put(server.server_address)
+ print(f"Starting server for {dist_dir} at: {server.server_address}")
+ server.serve_forever()
+
+
+@contextlib.contextmanager
+def launch_server(dist_dir):
+ q = queue.Queue()
+ p = threading.Thread(target=run_server_thread, args=[dist_dir, q], daemon=True)
+ p.start()
+ address = q.get(timeout=50)
+ time.sleep(0.1) # wait to make sure server is started
+ yield address
+ p.terminate()
+
+
+class NodeDriver:
+ import subprocess
+
+ def __init__(self, hostname, port):
+ self.process = subprocess.Popen(
+ [shutil.which("script"), "-c", f'"{shutil.which("node")}"'],
+ stdin=subprocess.PIPE,
+ shell=False,
+ bufsize=0,
+ )
+ print(self.process)
+ time.sleep(0.1) # wait for node to start
+ self.hostname = hostname
+ self.port = port
+ self.last_ret_code = None
+
+ def load_pyodide(self, dist_dir):
+ self.execute_js(
+ f"""
+ const {{ loadPyodide }} = require('{dist_dir}/pyodide.js')
+ let pyodide = await loadPyodide()
+ """
+ )
+
+ def clear_logs(self):
+ pass # we don't handle logs for node
+
+ def write_stdin(self, buffer):
+ # because we use unbuffered IO for
+ # stdout, stdin.write is also unbuffered
+ # so might under-run on writes
+ while len(buffer) > 0 and self.process.poll() is None:
+ written = self.process.stdin.write(buffer)
+ if written == len(buffer):
+ break
+ elif written == 0:
+ # full buffer - wait
+ time.sleep(0.01)
+ else:
+ buffer = buffer[written:]
+
+ def execute_js(self, code, wait_for_terminate=True):
+ self.write_stdin((code + "\n").encode("utf-8"))
+
+ def load_arrow(self):
+ self.execute_js(f"await pyodide.loadPackage('{PYARROW_WHEEL_PATH}')")
+
+ def execute_python(self, code, wait_for_terminate=True):
+ js_code = f"""
+ python = `{code}`;
+ await pyodide.loadPackagesFromImports(python);
+ python_output = await pyodide.runPythonAsync(python);
+ """
+ self.last_ret_code = self.execute_js(js_code, wait_for_terminate)
+ return self.last_ret_code
+
+ def wait_for_done(self):
+ # in node we just let it run above
+ # then send EOF and join process
+ self.write_stdin(b"process.exit(python_output)\n")
+ return self.process.wait()
+
+
+class BrowserDriver:
+ def __init__(self, hostname, port, driver):
+ self.driver = driver
+ self.driver.get(f"http://{hostname}:{port}/test.html")
+ self.driver.set_script_timeout(100)
+
+ def load_pyodide(self, dist_dir):
+ pass
+
+ def load_arrow(self):
+ self.execute_python(
+ f"import pyodide_js as pjs\n"
+ f"await pjs.loadPackage('{PYARROW_WHEEL_PATH.name}')\n"
+ )
+
+ def execute_python(self, code, wait_for_terminate=True):
+ if wait_for_terminate:
+ self.driver.execute_async_script(
+ f"""
+ let callback=arguments[arguments.length-1];
+ python = `{code}`;
+ window.python_done_callback=callback
+ window.pyworker.postMessage(
+ {{python,isatty:{'true' if sys.stdout.isatty() else 'false'}}})
+ """
+ )
+ else:
+ self.driver.execute_script(
+ f"""
+ let python = `{code}`;
+ window.python_done_callback= (x) => {{window.python_script_done=x;}};
+ window.pyworker.postMessage(
+ {{python,isatty:{'true' if sys.stdout.isatty() else 'false'}}});
+ """
+ )
+
+ def clear_logs(self):
+ self.driver.execute_script("window.python_logs = []")
+
+ def wait_for_done(self):
+ while True:
+ # poll for console.log messages from our webworker
+ # which are the output of pytest
+ lines = self.driver.execute_script(
+ "let temp = window.python_logs;window.python_logs=[];return temp;"
+ )
+ if len(lines) > 0:
+ sys.stdout.buffer.write(bytes(lines))
+ done = self.driver.execute_script("return window.python_script_done")
+ if done is not None:
+ value = done["result"]
+ self.driver.execute_script("delete window.python_script_done")
+ return value
+ time.sleep(0.1)
+
+
+class ChromeDriver(BrowserDriver):
+ def __init__(self, hostname, port):
+ from selenium.webdriver.chrome.options import Options
+
+ options = Options()
+ options.add_argument("--headless")
+ options.add_argument("--no-sandbox")
+ super().__init__(hostname, port, webdriver.Chrome(options=options))
+
+
+class FirefoxDriver(BrowserDriver):
+ def __init__(self, hostname, port):
+ from selenium.webdriver.firefox.options import Options
+
+ options = Options()
+ options.add_argument("--headless")
+
+ super().__init__(hostname, port, webdriver.Firefox(options=options))
+
+
+def _load_pyarrow_in_runner(driver, wheel_name):
+ driver.load_arrow()
+ driver.execute_python(
+ """import sys
+import micropip
+if "pyarrow" not in sys.modules:
+ await micropip.install("hypothesis")
+ import pyodide_js as pjs
+ await pjs.loadPackage("numpy")
+ await pjs.loadPackage("pandas")
+ import pytest
+ import pandas # import pandas after pyarrow package load for pandas/pyarrow
+ # functions to work
+import pyarrow
+ """,
+ wait_for_terminate=True,
+ )
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument(
+ "-d",
+ "--dist-dir",
+ type=str,
+ help="Pyodide distribution directory",
+ default="./pyodide",
+)
+parser.add_argument("wheel", type=str, help="Wheel to run tests from")
+parser.add_argument(
+ "-t", "--test-submodule", help="Submodule that tests live in", default="test"
+)
+parser.add_argument(
+ "-r",
+ "--runtime",
+ type=str,
+ choices=["chrome", "node", "firefox"],
+ help="Runtime to run tests in ",
Review Comment:
```suggestion
help="Runtime to run tests in",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1657089378
##########
ci/docker/conda-python-emscripten.dockerfile:
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+ARG repo
+ARG arch
+ARG python="3.12"
+FROM ${repo}:${arch}-conda-python-${python}
+
+ARG selenium_version="4.15.2"
+ARG pyodide_version="0.26.0"
+ARG emscripten_version="3.1.58"
+ARG chrome_version="latest"
+ARG required_python_min="(3,12)"
+# fail if python version < 3.12
+RUN echo "check PYTHON>=${required_python_min}" && python -c "import sys;sys.exit(0 if sys.version_info>=${required_python_min} else 1)"
+
+# install selenium and pyodide-build and recent python
+
+# needs to be a login shell so ~/.profile is read
+SHELL ["/bin/bash", "--login", "-c", "-o", "pipefail"]
+
+RUN python -m pip install --no-cache-dir selenium==${selenium_version} &&\
+ python -m pip install --no-cache-dir --upgrade --pre pyodide-build==${pyodide_version}
+
+COPY ci/scripts/install_emscripten.sh /arrow/ci/scripts/
+RUN bash /arrow/ci/scripts/install_emscripten.sh ~ "${emscripten_version}"
+
+# make sure zlib is cached in the EMSDK folder
+RUN source ~/emsdk/emsdk_env.sh && embuilder --pic build zlib
+
+# install pyodide dist directory to /pyodide
+WORKDIR /
+RUN pyodide_dist_url="https://github.com/pyodide/pyodide/releases/download/${pyodide_version}/pyodide-${pyodide_version}.tar.bz2" && \
+ wget --progress=dot:giga "${pyodide_dist_url}" -O- |tar -xj
+
+# install chrome for testing browser based runner
+# zip needed for chrome, libpthread stubs installs pthread module to cmake, build-essential makes
+# sure that unix make is available in isolated python environments
+
+# Install basic build stuff, don't pin versions, hence ignore lint
+# hadolint ignore=DL3008
+RUN apt-get update && apt-get install --no-install-recommends -y -q unzip zip libpthread-stubs0-dev build-essential && \
+ apt-get clean && \
+ rm -rf /var/lib/apt/lists/*
+
+COPY ci/scripts/install_chromedriver.sh /arrow/ci/scripts/
+RUN /arrow/ci/scripts/install_chromedriver.sh "${chrome_version}"
Review Comment:
nope, comment out above was a mistake sorry
##########
ci/docker/conda-python-emscripten.dockerfile:
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+ARG repo
+ARG arch
+ARG python="3.12"
+FROM ${repo}:${arch}-conda-python-${python}
+
+ARG selenium_version="4.15.2"
+ARG pyodide_version="0.26.0"
+ARG emscripten_version="3.1.58"
+ARG chrome_version="latest"
+ARG required_python_min="(3,12)"
+# fail if python version < 3.12
+RUN echo "check PYTHON>=${required_python_min}" && python -c "import sys;sys.exit(0 if sys.version_info>=${required_python_min} else 1)"
+
+# install selenium and pyodide-build and recent python
+
+# needs to be a login shell so ~/.profile is read
+SHELL ["/bin/bash", "--login", "-c", "-o", "pipefail"]
+
+RUN python -m pip install --no-cache-dir selenium==${selenium_version} &&\
Review Comment:
as above, sorry
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2210752674
@github-actions crossbow submit test-conda-python-emscripten
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou merged PR #37822:
URL: https://github.com/apache/arrow/pull/37822
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1666766790
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,347 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import os
+import queue
+import shutil
+import subprocess
+import sys
+import time
+import threading
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
+ if('results' in evt.data){
+ if(window.python_done_callback){
+ let callback=window.python_done_callback;
+ window.python_done_callback=undefined;
+ callback({result:evt.data.results});
+ }
+ }
+ if('print' in evt.data){
+ evt.data.print.forEach((x)=>{window.python_logs.push(x)});
+ }
+ }
+ window.pyworker = new Worker("worker.js");
+ window.pyworker.onmessage=capturelogs;
+ </script>
+ </head>
+ <body></body>
+ </html>
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "text/html")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+ elif self.path.endswith("/worker.js"):
+ body = b"""
+ importScripts("./pyodide.js");
+ onmessage = async function (e) {
+ const data = e.data;
+ if(!self.pyodide){
+ self.pyodide = await loadPyodide()
+ }
+ function do_print(arg){
+ let databytes = Array.from(arg)
+ self.postMessage({print:databytes})
+ return databytes.length
+ }
+ self.pyodide.setStdout({write:do_print,isatty:data.isatty});
+ self.pyodide.setStderr({write:do_print,isatty:data.isatty});
+
+ await self.pyodide.loadPackagesFromImports(data.python);
+ let results = await self.pyodide.runPythonAsync(data.python);
+ self.postMessage({results});
+ console.log('FINISHED_WEBWORKER')
+ }
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "application/javascript")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+
+ else:
+ return super().do_GET()
+
+ def end_headers(self):
+ # Enable Cross-Origin Resource Sharing (CORS)
+ self.send_header("Access-Control-Allow-Origin", "*")
+ super().end_headers()
+
+
+def run_server_thread(dist_dir, q):
+ global _SERVER_ADDRESS
+ os.chdir(dist_dir)
+ server = http.server.HTTPServer(("", 0), TemplateOverrider)
+ q.put(server.server_address)
+ print(f"Starting server for {dist_dir} at: {server.server_address}")
+ server.serve_forever()
+
+
+@contextlib.contextmanager
+def launch_server(dist_dir):
+ q = queue.Queue()
+ p = threading.Thread(target=run_server_thread, args=[dist_dir, q], daemon=True)
+ p.start()
+ address = q.get(timeout=50)
+ time.sleep(0.1) # wait to make sure server is started
+ yield address
+ p.terminate()
+
+
+class NodeDriver:
+ import subprocess
+
+ def __init__(self, hostname, port):
+ self.process = subprocess.Popen(
+ [shutil.which("script"), "-c", f'"{shutil.which("node")}"'],
+ stdin=subprocess.PIPE,
+ shell=False,
+ bufsize=0,
+ )
+ print(self.process)
+ time.sleep(0.1) # wait for node to start
+ self.hostname = hostname
+ self.port = port
+ self.last_ret_code = None
+
+ def load_pyodide(self, dist_dir):
+ self.execute_js(
+ f"""
+ const {{ loadPyodide }} = require('{dist_dir}/pyodide.js')
+ let pyodide = await loadPyodide()
+ """
+ )
+
+ def clear_logs(self):
+ pass # we don't handle logs for node
+
+ def write_stdin(self, buffer):
+ # because we use unbuffered IO for
+ # stdout, stdin.write is also unbuffered
+ # so might under-run on writes
+ while len(buffer) > 0 and self.process.poll() is None:
+ written = self.process.stdin.write(buffer)
+ if written == len(buffer):
+ break
+ elif written == 0:
+ # full buffer - wait
+ time.sleep(0.01)
+ else:
+ buffer = buffer[written:]
+
+ def execute_js(self, code, wait_for_terminate=True):
+ self.write_stdin((code + "\n").encode("utf-8"))
+
+ def load_arrow(self):
+ self.execute_js(f"await pyodide.loadPackage('{PYARROW_WHEEL_PATH}')")
+
+ def execute_python(self, code, wait_for_terminate=True):
+ js_code = f"""
+ python = `{code}`;
+ await pyodide.loadPackagesFromImports(python);
+ python_output = await pyodide.runPythonAsync(python);
+ """
+ self.last_ret_code = self.execute_js(js_code, wait_for_terminate)
+ return self.last_ret_code
+
+ def wait_for_done(self):
+ # in node we just let it run above
+ # then send EOF and join process
+ self.write_stdin(b"process.exit(python_output)\n")
+ return self.process.wait()
+
+
+class BrowserDriver:
+ def __init__(self, hostname, port, driver):
+ self.driver = driver
+ self.driver.get(f"http://{hostname}:{port}/test.html")
+ self.driver.set_script_timeout(100)
+
+ def load_pyodide(self, dist_dir):
+ pass
+
+ def load_arrow(self):
+ self.execute_python(
+ f"import pyodide_js as pjs\n"
+ f"await pjs.loadPackage('{PYARROW_WHEEL_PATH.name}')\n"
+ )
+
+ def execute_python(self, code, wait_for_terminate=True):
+ if wait_for_terminate:
+ self.driver.execute_async_script(
+ f"""
+ let callback=arguments[arguments.length-1];
+ python = `{code}`;
+ window.python_done_callback=callback
+ window.pyworker.postMessage(
+ {{python,isatty:{'true' if sys.stdout.isatty() else 'false'}}})
+ """
+ )
+ else:
+ self.driver.execute_script(
+ f"""
+ let python = `{code}`;
+ window.python_done_callback= (x) => {{window.python_script_done=x;}};
+ window.pyworker.postMessage(
+ {{python,isatty:{'true' if sys.stdout.isatty() else 'false'}}});
+ """
+ )
+
+ def clear_logs(self):
+ self.driver.execute_script("window.python_logs = []")
+
+ def wait_for_done(self):
+ while True:
+ # poll for console.log messages from our webworker
+ # which are the output of pytest
+ lines = self.driver.execute_script(
+ "let temp = window.python_logs;window.python_logs=[];return temp;"
+ )
+ if len(lines) > 0:
+ sys.stdout.buffer.write(bytes(lines))
+ done = self.driver.execute_script("return window.python_script_done")
+ if done is not None:
+ value = done["result"]
+ self.driver.execute_script("delete window.python_script_done")
+ return value
+ time.sleep(0.1)
+
+
+class ChromeDriver(BrowserDriver):
+ def __init__(self, hostname, port):
+ from selenium.webdriver.chrome.options import Options
+
+ options = Options()
+ options.add_argument("--headless")
+ options.add_argument("--no-sandbox")
+ super().__init__(hostname, port, webdriver.Chrome(options=options))
+
+
+class FirefoxDriver(BrowserDriver):
+ def __init__(self, hostname, port):
+ from selenium.webdriver.firefox.options import Options
+
+ options = Options()
+ options.add_argument("--headless")
+
+ super().__init__(hostname, port, webdriver.Firefox(options=options))
+
+
+def _load_pyarrow_in_runner(driver, wheel_name):
+ driver.load_arrow()
+ driver.execute_python(
+ """import sys
+import micropip
+if "pyarrow" not in sys.modules:
+ await micropip.install("hypothesis")
+ import pyodide_js as pjs
+ await pjs.loadPackage("numpy")
+ await pjs.loadPackage("pandas")
+ import pytest
+ import pandas # import pandas after pyarrow package load for pandas/pyarrow
+ # functions to work
+import pyarrow
+ """,
+ wait_for_terminate=True,
+ )
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument(
+ "-d",
+ "--dist-dir",
+ type=str,
+ help="Pyodide distribution directory",
+ default="./pyodide",
+)
+parser.add_argument("wheel", type=str, help="Wheel to run tests from")
+parser.add_argument(
+ "-t", "--test-submodule", help="Submodule that tests live in", default="test"
+)
+parser.add_argument(
+ "-r",
+ "--runtime",
+ type=str,
+ choices=["chrome", "node", "firefox"],
+ help="Runtime to run tests in ",
+ default="chrome",
+)
+args = parser.parse_args()
+
+PYARROW_WHEEL_PATH = Path(args.wheel).resolve()
+
+dist_dir = Path(os.getcwd(), args.dist_dir).resolve()
+print(f"dist dir={dist_dir}")
+with launch_server(dist_dir) as (hostname, port):
+ if args.runtime == "chrome":
+ driver = ChromeDriver(hostname, port)
+ elif args.runtime == "node":
+ driver = NodeDriver(hostname, port)
+ elif args.runtime == "firefox":
+ driver = FirefoxDriver(hostname, port)
+
+ print("Load pyodide in browser")
+ driver.load_pyodide(dist_dir)
+ print("Done\n")
Review Comment:
@joemarshall I see the `print("Done")` is removed entirely now. Is that what you intended?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-23221: [Python] Add support for Pyodide [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1622047559
##########
ci/docker/ubuntu-22.04-cpp-emscripten.dockerfile:
##########
Review Comment:
Can we use `ci/docker/linux-apt-python-3.dockerfile` like `ubuntu-python` service in `docker-compose.yml` does?
##########
ci/scripts/emscripten_python_build.sh:
##########
Review Comment:
Could you rename this to `python_build_emscripten.sh`?
We use `${language}_XXX.sh` naming convention.
##########
ci/scripts/emscripten_python_build.sh:
##########
@@ -0,0 +1,27 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -ex
+
+source ~/emsdk/emsdk_env.sh
+
+source_dir=${1}/python
+cd ${source_dir}
+rm -rf dist
+pyodide build
Review Comment:
Could you use build directory instead of changing source directory?
It creates files owned by `root` on host.
See also: https://github.com/apache/arrow/issues/41429
`python_build.sh` copies sources for now: https://github.com/apache/arrow/blob/255dbf990c3d3e5fb1270a2a11efe0af2be195ab/ci/scripts/python_build.sh#L81-L93
##########
python/CMakeLists.txt:
##########
@@ -650,28 +671,42 @@ endif()
# Acero
if(PYARROW_BUILD_ACERO)
- if(PYARROW_BUNDLE_ARROW_CPP)
- bundle_arrow_lib(${ARROW_ACERO_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
- if(MSVC)
- bundle_arrow_import_lib(${ARROW_ACERO_IMPORT_LIB})
+ list(APPEND CYTHON_EXTENSIONS _acero)
+
+ if(ARROW_BUILD_SHARED)
+
+ if(PYARROW_BUNDLE_ARROW_CPP)
+ bundle_arrow_lib(${ARROW_ACERO_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
+ if(MSVC)
+ bundle_arrow_import_lib(${ARROW_ACERO_IMPORT_LIB})
+ endif()
endif()
- endif()
- set(ACERO_LINK_LIBS ArrowAcero::arrow_acero_shared)
- list(APPEND CYTHON_EXTENSIONS _acero)
+ set(ACERO_LINK_LIBS ArrowAcero::arrow_acero_shared)
+ else()
+ # ACERO is statically linked into libarrow_python already
+ set(ACERO_LINK_LIBS)
+ endif()
endif()
# Dataset
if(PYARROW_BUILD_DATASET)
- if(PYARROW_BUNDLE_ARROW_CPP)
- bundle_arrow_lib(${ARROW_DATASET_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
- if(MSVC)
- bundle_arrow_import_lib(${ARROW_DATASET_IMPORT_LIB})
+ if(ARROW_BUILD_SHARED)
+
Review Comment:
```suggestion
```
##########
python/setup.py:
##########
@@ -40,6 +40,14 @@
# Check if we're running 64-bit Python
is_64_bit = sys.maxsize > 2**32
+is_emscripten = False
+if (
+ sysconfig.get_config_var("SOABI")
+ and sysconfig.get_config_var("SOABI").find("emscripten") != -1
+):
+ is_emscripten = True
Review Comment:
```suggestion
is_emscripten = (
sysconfig.get_config_var("SOABI")
and sysconfig.get_config_var("SOABI").find("emscripten") != -1
)
```
##########
python/CMakeLists.txt:
##########
@@ -143,6 +146,22 @@ if(NOT DEFINED ARROW_RUNTIME_SIMD_LEVEL)
endif()
include(SetupCxxFlags)
+if($ENV{PYODIDE})
+ # these variables are needed for building pyarrow on Emscripten
Review Comment:
```suggestion
# These variables are needed for building PyArrow on Emscripten.
```
##########
python/CMakeLists.txt:
##########
@@ -143,6 +146,22 @@ if(NOT DEFINED ARROW_RUNTIME_SIMD_LEVEL)
endif()
include(SetupCxxFlags)
+if($ENV{PYODIDE})
+ # these variables are needed for building pyarrow on Emscripten
+ # if they aren't set, cmake cross compiling fails for python
+ # modules (at least under pyodide it does)
+ set(Python3_INCLUDE_DIR $ENV{PYTHONINCLUDE})
+ set(Python3_LIBRARY $ENV{CPYTHONLIB})
+ set(Python3_NumPy_INCLUDE_DIR $ENV{NUMPY_LIB}/core/include)
+ set(Python3_EXECUTABLE)
+ set(ENV{_PYTHON_SYSCONFIGDATA_NAME} $ENV{SYSCONFIG_NAME})
+ # we set the c and cxx compiler manually to bypass pywasmcross
+ # which is pyodide's way of messing with C++ build parameters.
+ set(CMAKE_C_COMPILER emcc)
+ set(CMAKE_CXX_COMPILER em++)
+
Review Comment:
```suggestion
```
##########
ci/scripts/emscripten_python_test.sh:
##########
Review Comment:
Could you rename this to `python_test_emscripten.sh`?
##########
python/CMakeLists.txt:
##########
@@ -68,6 +68,9 @@ if(POLICY CMP0095)
cmake_policy(SET CMP0095 NEW)
endif()
+# this option is used to auto-set defaults for pyarrow build
+option(DUMP_ARROW_ARGUMENTS "Dump the arrow arguments then quit" OFF)
+
Review Comment:
Can we remove this?
##########
ci/scripts/cpp_build.sh:
##########
@@ -105,8 +105,6 @@ if [ "${ARROW_EMSCRIPTEN:-OFF}" = "ON" ]; then
-DCMAKE_C_FLAGS="${CFLAGS:-}" \
-DCMAKE_CXX_FLAGS="${CXXFLAGS:-}" \
-DCMAKE_CXX_STANDARD="${CMAKE_CXX_STANDARD:-17}" \
- -DCMAKE_INSTALL_LIBDIR=${CMAKE_INSTALL_LIBDIR:-lib} \
- -DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX:-${ARROW_HOME}} \
Review Comment:
Why do we need to remove them?
##########
python/CMakeLists.txt:
##########
@@ -408,13 +427,15 @@ if(NOT PYARROW_CPP_LINK_LIBS)
endif()
add_library(arrow_python SHARED ${PYARROW_CPP_SRCS})
+
Review Comment:
Could you revert needless changes?
```suggestion
```
##########
python/scripts/run_emscripten_tests.py:
##########
@@ -0,0 +1,330 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import argparse
+import contextlib
+import http.server
+import multiprocessing
+import os
+import shutil
+import subprocess
+import sys
+import time
+
+from pathlib import Path
+from io import BytesIO
+
+from selenium import webdriver
+
+
+class TemplateOverrider(http.server.SimpleHTTPRequestHandler):
+ def log_request(self, code="-", size="-"):
+ # don't log successful requests
+ return
+
+ def do_GET(self) -> bytes | None:
+ if self.path.endswith(PYARROW_WHEEL_PATH.name):
+ self.send_response(200)
+ self.send_header("Content-type", "application/x-zip")
+ self.end_headers()
+ self.copyfile(PYARROW_WHEEL_PATH.open(mode="rb"), self.wfile)
+ if self.path.endswith("/test.html"):
+ body = b"""
+ <!doctype html>
+ <html>
+ <head>
+ <script>
+ window.python_done_callback=undefined;
+ window.python_logs=[];
+ function capturelogs(evt)
+ {
+ if('results' in evt.data){
+ if(window.python_done_callback){
+ let callback=window.python_done_callback;
+ window.python_done_callback=undefined;
+ callback({result:evt.data.results});
+ }
+ }
+ if('print' in evt.data){
+ evt.data.print.forEach((x)=>{window.python_logs.push(x)});
+ }
+ }
+ window.pyworker = new Worker("worker.js");
+ window.pyworker.onmessage=capturelogs;
+ </script>
+ </head>
+ <body></body>
+ </html>
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "text/html")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+ elif self.path.endswith("/worker.js"):
+ body = b"""
+ importScripts("./pyodide.js");
+ onmessage = async function (e) {
+ const data = e.data;
+ if(!self.pyodide){
+ self.pyodide = await loadPyodide()
+ }
+ function do_print(arg){
+ let databytes = Array.from(arg)
+ self.postMessage({print:databytes})
+ return databytes.length
+ }
+ self.pyodide.setStdout({write:do_print,isatty:data.isatty});
+ self.pyodide.setStderr({write:do_print,isatty:data.isatty});
+
+ await self.pyodide.loadPackagesFromImports(data.python);
+ let results = await self.pyodide.runPythonAsync(data.python);
+ self.postMessage({results});
+ console.log('FINISHED_WEBWORKER')
+ }
+ """
+ self.send_response(200)
+ self.send_header("Content-type", "application/javascript")
+ self.send_header("Content-length", len(body))
+ self.end_headers()
+ self.copyfile(BytesIO(body), self.wfile)
+
+ else:
+ return super().do_GET()
+
+ def end_headers(self):
+ # Enable Cross-Origin Resource Sharing (CORS)
+ self.send_header("Access-Control-Allow-Origin", "*")
+ super().end_headers()
+
+
+def run_server_thread(dist_dir, q):
+ global _SERVER_ADDRESS
+ os.chdir(dist_dir)
+ server = http.server.HTTPServer(("", 0), TemplateOverrider)
+ q.put(server.server_address)
+ print(f"Starting server for {dist_dir} at: {server.server_address}")
+ server.serve_forever()
+
+
+@contextlib.contextmanager
+def launch_server(dist_dir):
+ q = multiprocessing.Queue()
+ p = multiprocessing.Process(target=run_server_thread, args=[dist_dir, q])
+ p.start()
+ address = q.get(timeout=50)
+ yield address
+ p.terminate()
+
+
+class NodeDriver:
+ import subprocess
+
+ def __init__(self, hostname, port):
+ self.process = subprocess.Popen(
+ [shutil.which("node"), "-i"], stdin=subprocess.PIPE, bufsize=0
+ )
+ print(self.process)
+ self.hostname = hostname
+ self.port = port
+ self.last_ret_code = None
+
+ def load_pyodide(self, dist_dir):
+ self.execute_js(
+ f"""
+ const {{ loadPyodide }} = require('{dist_dir}/pyodide.js')
+ let pyodide = await loadPyodide()
+ """
+ )
+
+ def clear_logs(self):
+ pass # we don't handle logs for node
+
+ def execute_js(self, code, wait_for_terminate=True):
+ self.process.stdin.write((code + "\n").encode("utf-8"))
+
+ def load_arrow(self):
+ self.execute_js(f"await pyodide.loadPackage('{PYARROW_WHEEL_PATH}')")
+
+ def execute_python(self, code, wait_for_terminate=True):
+ js_code = f"""
+ python = `{code}`;
+ await pyodide.loadPackagesFromImports(python);
+ await pyodide.runPythonAsync(python);
+ """
+ self.last_ret_code = self.execute_js(js_code, wait_for_terminate)
+ return self.last_ret_code
+
+ def wait_for_done(self):
+ # in node we just let it run above
+ # then send EOF
+ self.process.stdin.write(b".exit\n")
+ return self.last_ret_code
+
+
+class BrowserDriver:
+ def __init__(self, hostname, port, driver):
+ self.driver = driver
+ self.driver.get(f"http://{hostname}:{port}/test.html")
+ self.driver.set_script_timeout(100)
+
+ def load_pyodide(self, dist_dir):
+ pass
+
+ def load_arrow(self):
+ self.execute_python(
+ f"import pyodide_js as pjs\n"
+ f"await pjs.loadPackage('{PYARROW_WHEEL_PATH.name}')\n"
+ )
+
+ def execute_python(self, code, wait_for_terminate=True):
+ if wait_for_terminate:
+ self.driver.execute_async_script(
+ f"""
+ let callback=arguments[arguments.length-1];
+ python = `{code}`;
+ window.python_done_callback=callback
+ window.pyworker.postMessage(
+ {{python,isatty:{'true' if sys.stdout.isatty() else 'false'}}})
+ """
+ )
+ else:
+ self.driver.execute_script(
+ f"""
+ let python = `{code}`;
+ window.python_done_callback= (x) => {{window.python_script_done=x}}
+ window.pyworker.postMessage(
+ {{python,isatty:{'true' if sys.stdout.isatty() else 'false'}}});
+ """
+ )
+
+ def clear_logs(self):
+ self.driver.execute_script("window.python_logs = []")
+
+ def wait_for_done(self):
+ while True:
+ # poll for console.log messages from our webworker
+ # which are the output of pytest
+ lines = self.driver.execute_script(
+ "let temp = window.python_logs;window.python_logs=[];return temp;"
+ )
+ if len(lines) > 0:
+ sys.stdout.buffer.write(bytes(lines))
+ done = self.driver.execute_script("return window.python_script_done")
+ if done is not None:
+ value = done["result"]
+ self.driver.execute_script("delete window.python_script_done")
+ return value
+ time.sleep(0.1)
+
+
+class ChromeDriver(BrowserDriver):
+ def __init__(self, hostname, port):
+ from selenium.webdriver.chrome.options import Options
+
+ options = Options()
+ options.add_argument("--headless")
+ options.add_argument("--no-sandbox")
+ super().__init__(hostname, port, webdriver.Chrome(options=options))
+
+
+class FirefoxDriver(BrowserDriver):
+ def __init__(self, hostname, port):
+ from selenium.webdriver.firefox.options import Options
+
+ options = Options()
+ options.add_argument("--headless")
+
+ super().__init__(hostname, port, webdriver.Firefox(options=options))
+
+
+def _load_pyarrow_in_runner(driver, wheel_name):
+ driver.load_arrow()
+ driver.execute_python(
+ """import sys
+import micropip
+if "pyarrow" not in sys.modules:
+ try:
+ import tzdata
+ except:
+ await micropip.install("tzdata")
+ await micropip.install("hypothesis")
+ import pyodide_js as pjs
+ await pjs.loadPackage("numpy")
+ await pjs.loadPackage("pandas")
+ import pytest
+ import pandas # import pandas after pyarrow package load for pandas/pyarrow
+ #functions to work
Review Comment:
```suggestion
# functions to work
```
##########
python/CMakeLists.txt:
##########
@@ -650,28 +671,42 @@ endif()
# Acero
if(PYARROW_BUILD_ACERO)
- if(PYARROW_BUNDLE_ARROW_CPP)
- bundle_arrow_lib(${ARROW_ACERO_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
- if(MSVC)
- bundle_arrow_import_lib(${ARROW_ACERO_IMPORT_LIB})
+ list(APPEND CYTHON_EXTENSIONS _acero)
+
+ if(ARROW_BUILD_SHARED)
+
+ if(PYARROW_BUNDLE_ARROW_CPP)
+ bundle_arrow_lib(${ARROW_ACERO_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
+ if(MSVC)
+ bundle_arrow_import_lib(${ARROW_ACERO_IMPORT_LIB})
+ endif()
endif()
- endif()
- set(ACERO_LINK_LIBS ArrowAcero::arrow_acero_shared)
- list(APPEND CYTHON_EXTENSIONS _acero)
+ set(ACERO_LINK_LIBS ArrowAcero::arrow_acero_shared)
+ else()
+ # ACERO is statically linked into libarrow_python already
+ set(ACERO_LINK_LIBS)
+ endif()
endif()
# Dataset
if(PYARROW_BUILD_DATASET)
- if(PYARROW_BUNDLE_ARROW_CPP)
- bundle_arrow_lib(${ARROW_DATASET_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
- if(MSVC)
- bundle_arrow_import_lib(${ARROW_DATASET_IMPORT_LIB})
+ if(ARROW_BUILD_SHARED)
+
+ if(PYARROW_BUNDLE_ARROW_CPP)
+ bundle_arrow_lib(${ARROW_DATASET_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
+ if(MSVC)
+ bundle_arrow_import_lib(${ARROW_DATASET_IMPORT_LIB})
+ endif()
endif()
- endif()
- set(DATASET_LINK_LIBS ArrowDataset::arrow_dataset_shared)
+ set(DATASET_LINK_LIBS ArrowDataset::arrow_dataset_shared)
+ else()
+ # dataset is statically linked into libarrow_python already
+ set(DATASET_LINK_LIBS)
+ endif()
list(APPEND CYTHON_EXTENSIONS _dataset)
+
Review Comment:
```suggestion
```
##########
python/CMakeLists.txt:
##########
@@ -650,28 +671,42 @@ endif()
# Acero
if(PYARROW_BUILD_ACERO)
- if(PYARROW_BUNDLE_ARROW_CPP)
- bundle_arrow_lib(${ARROW_ACERO_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
- if(MSVC)
- bundle_arrow_import_lib(${ARROW_ACERO_IMPORT_LIB})
+ list(APPEND CYTHON_EXTENSIONS _acero)
+
+ if(ARROW_BUILD_SHARED)
+
+ if(PYARROW_BUNDLE_ARROW_CPP)
+ bundle_arrow_lib(${ARROW_ACERO_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
+ if(MSVC)
+ bundle_arrow_import_lib(${ARROW_ACERO_IMPORT_LIB})
+ endif()
endif()
- endif()
- set(ACERO_LINK_LIBS ArrowAcero::arrow_acero_shared)
- list(APPEND CYTHON_EXTENSIONS _acero)
+ set(ACERO_LINK_LIBS ArrowAcero::arrow_acero_shared)
+ else()
+ # ACERO is statically linked into libarrow_python already
Review Comment:
```suggestion
# Acero is statically linked into libarrow_python already
```
##########
python/CMakeLists.txt:
##########
@@ -745,14 +782,22 @@ if(PYARROW_BUILD_SUBSTRAIT)
message(FATAL_ERROR "You must build Arrow C++ with ARROW_SUBSTRAIT=ON")
endif()
find_package(ArrowSubstrait REQUIRED)
- if(PYARROW_BUNDLE_ARROW_CPP)
- bundle_arrow_lib(${ARROW_SUBSTRAIT_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
- if(MSVC)
- bundle_arrow_import_lib(${ARROW_SUBSTRAIT_IMPORT_LIB})
+
+ if(ARROW_BUILD_SHARED)
+ if(PYARROW_BUNDLE_ARROW_CPP)
+ bundle_arrow_lib(${ARROW_SUBSTRAIT_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
+ if(MSVC)
+ bundle_arrow_import_lib(${ARROW_SUBSTRAIT_IMPORT_LIB})
+ endif()
endif()
+ set(SUBSTRAIT_LINK_LIBS ArrowSubstrait::arrow_substrait_shared)
+ else()
+ # only link to the output of substrait, not to libarrow etc.
+ # if we link to the target directly we end up with multiple copies
Review Comment:
```suggestion
# Only link to the output of substrait, not to libarrow etc.
# If we link to the target directly we end up with multiple copies
```
##########
python/CMakeLists.txt:
##########
@@ -650,28 +671,42 @@ endif()
# Acero
if(PYARROW_BUILD_ACERO)
- if(PYARROW_BUNDLE_ARROW_CPP)
- bundle_arrow_lib(${ARROW_ACERO_SHARED_LIB} SO_VERSION ${ARROW_SO_VERSION})
- if(MSVC)
- bundle_arrow_import_lib(${ARROW_ACERO_IMPORT_LIB})
+ list(APPEND CYTHON_EXTENSIONS _acero)
+
+ if(ARROW_BUILD_SHARED)
+
Review Comment:
```suggestion
```
##########
python/CMakeLists.txt:
##########
@@ -143,6 +146,22 @@ if(NOT DEFINED ARROW_RUNTIME_SIMD_LEVEL)
endif()
include(SetupCxxFlags)
+if($ENV{PYODIDE})
+ # these variables are needed for building pyarrow on Emscripten
+ # if they aren't set, cmake cross compiling fails for python
+ # modules (at least under pyodide it does)
Review Comment:
```suggestion
# If they aren't set, CMake cross compiling fails for Python
# modules (at least under Pyodide it does).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1633277451
##########
python/scripts/run_emscripten_tests.py:
##########
Review Comment:
Yeah, it's because numpy uses the pytest-pyodide plugin, with some code I wrote. What that does is start one interpreter, then call `pytest.main` for each individual test. However that only works if the library being called is *extremely* happy with being reloaded, or the test setup/teardown fixtures make sure everything is unloaded nicely. The only way I could make arrow tests work is by doing a single call to `pytest.main` for all tests and then streaming the results out.
It's a side effect of how pytest is designed - it is only really designed to be called once per interpreter startup. So you can do things in session level fixtures that change e.g. C++ library state (e.g. user defined functions in arrow), and not worry about whether the changes can be undone, because you know that everything will be started from scratch before pytest is run again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2168287510
> I made it embed the timezone data in the python wheel rather than downloading it using the tzdata library
I started writing my comment at https://github.com/apache/arrow/pull/37822#discussion_r1639997184 an hour ago, but got interrupted and only finished it now before seeing your update. So you clearly did succeed to include it, which I thought might not be possible for non-Windows platforms.
I am a bit wary of including that data (for Windows we decided to not do that). But apart from that, the reason this works is because you actually copy the data to the location in "/usr/share/zoneinfo" (where tz.h expects it to be), right? (I was thinking about the windows case where we can point to a custom download location, which we don't support on other platforms).
Are there situations that this already exists? Or should that generally not be the case for emscripten (node or browser) environments? And if it would already exist, are we sure it has the same format?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2168289503
@github-actions crossbow submit test-conda-python-emscripten
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #37822:
URL: https://github.com/apache/arrow/pull/37822#discussion_r1628582687
##########
docker-compose.yml:
##########
@@ -679,6 +680,45 @@ services:
UBUNTU:
command: *cpp-command
+ ubuntu-cpp-emscripten-python:
Review Comment:
If we want to use Ubuntu based image for this, we should use `ubuntu-python-emscripten`.
If we need to use conda, we should use `conda-python-emscripten`.
##########
docker-compose.yml:
##########
@@ -679,6 +680,45 @@ services:
UBUNTU:
command: *cpp-command
+ ubuntu-cpp-emscripten-python:
+ # Usage:
+ # docker-compose build ubuntu-cpp-emscripten-python
+ # docker-compose run --rm ubuntu-cpp-emscripten-python
+ # Parameters:
+ # ARCH: amd64, arm64v8, ...
+ # UBUNTU: 22.04
+ image: ${REPO}:${ARCH}-ubuntu-22.04-cpp-emscripten
+ build:
+ context: .
+ dockerfile: ci/docker/ubuntu-22.04-cpp-emscripten.dockerfile
Review Comment:
If we want to use Ubuntu based image for this, we should reuse `ci/docker/linux-apt-python-3.dockerfile` or create `ci/docker/linux-apt-python-3-emscripten.dockerfile` based on `linux-apt-python-3.dockerfile`.
If we need to use conda, we should create `ci/docker/conda-python-emscripten.dockerfile` based on `ci/docker/conda-python.dockerfile`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41910: [Python] Add support for Pyodide [arrow]
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on PR #37822:
URL: https://github.com/apache/arrow/pull/37822#issuecomment-2184683074
@jorisvandenbossche
I think I worked out what the right thing to do is.
1. Don't package timezone data here at all.
2. In tests check if zoneinfo path exists and skip time tests accordingly. (I.e. via test marker)
3. On pyodide I will add a package to install Unix style timezone data as a dependency of arrow.
If that upstream bug is fixed then it could move back to using python tzdata package if installed, but until then I think this is the best approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org