You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "MarcSkovMadsen (via GitHub)" <gi...@apache.org> on 2023/04/10 06:02:16 UTC
[GitHub] [arrow] MarcSkovMadsen opened a new issue, #34996: Make pyarrow importable in Jupyterlite
MarcSkovMadsen opened a new issue, #34996:
URL: https://github.com/apache/arrow/issues/34996
### Describe the enhancement requested
I'm trying to use pyarrow with [Panel](https://panel.holoviz.org) in [Panelite](https://panelite.holoviz.org/lab/index.html). Panelite is a custom build of Jupyterlite that works with Panel.
The issue is that `piplite` cannot install `pyarrow`. It would be great if pyarrow also work with pyodide and in Jupyterlite/ Panelite.
Thanks.
-----
```python
import piplite
await piplite.install(['panel', 'xlsxwriter', 'pyarrow'])
```
```bash
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[16], line 2
1 import piplite
----> 2 await piplite.install(['panel', 'xlsxwriter', 'pyarrow'])
File /lib/python3.10/asyncio/futures.py:284, in Future.__await__(self)
282 if not self.done():
283 self._asyncio_future_blocking = True
--> 284 yield self # This tells Task to wait for completion.
285 if not self.done():
286 raise RuntimeError("await wasn't used with future")
File /lib/python3.10/asyncio/tasks.py:304, in Task.__wakeup(self, future)
302 def __wakeup(self, future):
303 try:
--> 304 future.result()
305 except BaseException as exc:
306 # This may also be a cancellation.
307 self.__step(exc)
File /lib/python3.10/asyncio/futures.py:201, in Future.result(self)
199 self.__log_traceback = False
200 if self._exception is not None:
--> 201 raise self._exception
202 return self._result
File /lib/python3.10/asyncio/tasks.py:234, in Task.__step(***failed resolving arguments***)
232 result = coro.send(None)
233 else:
--> 234 result = coro.throw(exc)
235 except StopIteration as exc:
236 if self._must_cancel:
237 # Task is cancelled right before coro stops.
File /lib/python3.10/site-packages/piplite/piplite.py:102, in _install(requirements, keep_going, deps, credentials, pre)
100 """Invoke micripip.install with a patch to get data from local indexes"""
101 with patch("micropip._micropip._get_pypi_json", _get_pypi_json):
--> 102 return await _micropip.install(
103 requirements=requirements,
104 keep_going=keep_going,
105 deps=deps,
106 credentials=credentials,
107 pre=pre,
108 )
File /lib/python3.10/site-packages/micropip/_micropip.py:573, in install(requirements, keep_going, deps, credentials, pre)
563 wheel_base = Path(getsitepackages()[0])
565 transaction = Transaction(
566 ctx=ctx,
567 ctx_extras=[],
(...)
571 fetch_kwargs=fetch_kwargs,
572 )
--> 573 await transaction.gather_requirements(requirements)
575 if transaction.failed:
576 failed_requirements = ", ".join([f"'{req}'" for req in transaction.failed])
File /lib/python3.10/site-packages/micropip/_micropip.py:333, in Transaction.gather_requirements(self, requirements)
330 for requirement in requirements:
331 requirement_promises.append(self.add_requirement(requirement))
--> 333 await gather(*requirement_promises)
File /lib/python3.10/asyncio/futures.py:284, in Future.__await__(self)
282 if not self.done():
283 self._asyncio_future_blocking = True
--> 284 yield self # This tells Task to wait for completion.
285 if not self.done():
286 raise RuntimeError("await wasn't used with future")
File /lib/python3.10/asyncio/tasks.py:304, in Task.__wakeup(self, future)
302 def __wakeup(self, future):
303 try:
--> 304 future.result()
305 except BaseException as exc:
306 # This may also be a cancellation.
307 self.__step(exc)
File /lib/python3.10/asyncio/futures.py:201, in Future.result(self)
199 self.__log_traceback = False
200 if self._exception is not None:
--> 201 raise self._exception
202 return self._result
File /lib/python3.10/asyncio/tasks.py:232, in Task.__step(***failed resolving arguments***)
228 try:
229 if exc is None:
230 # We use the `send` method directly, because coroutines
231 # don't have `__iter__` and `__next__` methods.
--> 232 result = coro.send(None)
233 else:
234 result = coro.throw(exc)
File /lib/python3.10/site-packages/micropip/_micropip.py:340, in Transaction.add_requirement(self, req)
337 return await self.add_requirement_inner(req)
339 if not urlparse(req).path.endswith(".whl"):
--> 340 return await self.add_requirement_inner(Requirement(req))
342 # custom download location
343 wheel = WheelInfo.from_url(req)
File /lib/python3.10/site-packages/micropip/_micropip.py:435, in Transaction.add_requirement_inner(self, req)
432 metadata = await _get_pypi_json(req.name, self.fetch_kwargs)
434 try:
--> 435 wheel = find_wheel(metadata, req)
436 except ValueError:
437 self.failed.append(req)
File /lib/python3.10/site-packages/micropip/_micropip.py:303, in find_wheel(metadata, req)
300 if best_wheel is not None:
301 return wheel
--> 303 raise ValueError(
304 f"Can't find a pure Python 3 wheel for '{req}'.\n"
305 f"See: {FAQ_URLS['cant_find_wheel']}\n"
306 "You can use `micropip.install(..., keep_going=True)`"
307 "to get a list of all packages with missing wheels."
308 )
ValueError: Can't find a pure Python 3 wheel for 'pyarrow'.
See: https://pyodide.org/en/stable/usage/faq.html#micropip-can-t-find-a-pure-python-wheel
You can use `micropip.install(..., keep_going=True)`to get a list of all packages with missing wheels.
```
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #34996: Make pyarrow installable in Jupyterlite
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #34996:
URL: https://github.com/apache/arrow/issues/34996#issuecomment-1538774758
I'm not super familiar with pyodide. Is the only solution to have a pure python wheel? There have been other efforts to get arrow-c++ to be usable with emscripten. If this were done would the C++ then be usable with pyodide?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eliasdabbas commented on issue #34996: Make pyarrow installable in Jupyterlite
Posted by "eliasdabbas (via GitHub)" <gi...@apache.org>.
eliasdabbas commented on issue #34996:
URL: https://github.com/apache/arrow/issues/34996#issuecomment-1532822351
+1
`pyarrow` is becoming more crucial with pandas 2.0 as well. Would be great to have it supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] joemarshall commented on issue #34996: [Python] Make pyarrow installable in Jupyterlite
Posted by "joemarshall (via GitHub)" <gi...@apache.org>.
joemarshall commented on issue #34996:
URL: https://github.com/apache/arrow/issues/34996#issuecomment-1540062806
I am 90% there on arrow in pyodide. I have a running build of pyarrow here, but it needs a little work to be generally buildable by anyone. And a little more work so that it loads straight into pyodide nicely.
This PR:
https://github.com/apache/arrow/pull/35471
is the bulk of the work, because for pyodide / browsers we require a build of arrow cpp that works okay without threads.
Once that is merged, there are a few minor build file changes to add support and cmake presets for emscripten & pyodide - this isn't that hard because we've done a bunch of work in pyodide-build which simplifies stuff. Oh and that all needs adding to the github CI here or else people will break it over time.
Then finally, once there is a sensible build process for emscripten pyarrow, it will probably make sense to add a recipe to pyodide so that it is distributed with core pyodide. That is pretty straightforward (I've done a few of them before) - basically pretty much a matter of pointing it at the git repository tag and telling it the build command to make arrow-cpp and pyarrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] MarcSkovMadsen commented on issue #34996: Make pyarrow installable in Jupyterlite
Posted by "MarcSkovMadsen (via GitHub)" <gi...@apache.org>.
MarcSkovMadsen commented on issue #34996:
URL: https://github.com/apache/arrow/issues/34996#issuecomment-1510419286
Related to https://github.com/pyodide/pyodide/issues/2933
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org