You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by George <ge...@gmail.com> on 2025/05/14 08:22:05 UTC
Python based User defined function on Flink 1.19.1
Hi all
My Flink Jobmanager and Taskmanager is based on the community 1.19.1 on the
Java 17 image.
I've extended this by installing Python 3.11 and python3.11-venv.
created my venv environment, activated it,
when trying to pip install apache-flink. I first got an error saying
/opt/java/openjdk/include does not exist... so created it.
Next attempt ended in the below error.
Please advise.
G
```
venv) flink@jobmanager:/sql/prompush$ pip install apache-flink
Collecting apache-flink
Downloading apache-flink-2.0.0.tar.gz (1.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 1.9 MB/s eta
0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting python-dateutil<3,>=2.8.0
Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Collecting requests>=2.26.0
Using cached requests-2.32.3-py3-none-any.whl (64 kB)
Collecting pyarrow>=5.0.0
Downloading pyarrow-20.0.0-cp311-cp311-manylinux_2_28_aarch64.whl (40.7
MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.7/40.7 MB 415.8 kB/s eta
0:00:00
Collecting pytz>=2018.3
Using cached pytz-2025.2-py2.py3-none-any.whl (509 kB)
Collecting fastavro!=1.8.0,>=1.1.0
Using cached
fastavro-1.10.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
(3.3 MB)
Collecting pemja==0.4.1
Downloading pemja-0.4.1.tar.gz (50 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.7/50.7 KB 600.0 kB/s eta
0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 255
╰─> [1 lines of output]
jni.h should be in '/opt/java/openjdk/include/jni.h' but doesn't
exist. Please check you've installed the JDK properly.
[end of output]
note: This error originates from a subprocess, and is likely not a
problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 255
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem
with pip.
```
--
You have the obligation to inform one honestly of the risk, and as a person
you are committed to educate yourself to the total risk in any activity!
Once informed & totally aware of the risk,
every fool has the right to kill or injure themselves as they see fit!
Re: Python based User defined function on Flink 1.19.1
Posted by George <ge...@gmail.com>.
Hi Nikola
Correct on the assumption that I'm working on ARM, MBP M1.
So ye i can add the -headless to my image build.
The pip install it's in a total different phase though, don't install the
packages ... currently as part of the base image, but might need to move it.
let me have a look. will advise when done.
G
On Wed, May 14, 2025 at 11:18 AM Nikola Milutinovic <n....@levi9.com>
wrote:
> Hi George.
>
>
>
> We saw the same problem, running Apache Flink 1.19 and 1.20 images. The
> cause is that Flink image provides a JRE and you need JDK to build/install
> PyFlink. And, oddly enough, I think it was only on ARM64 images. Amd64 was
> OK, I think. So, Mac M1, M2, M3…
>
>
>
> Our Docker file for building our custom image has some extra steps, to
> install JDK, pip-install and then cleanup.
>
>
>
> RUN apt-get update -y \
>
> && apt-get upgrade -y \
>
> && apt-get install --no-install-recommends -y \
>
> …
>
> openjdk-17-jdk-headless \
>
> ENV JAVA_HOME=/usr/lib/jvm/java-17-openjdk-arm64
>
>
>
> After PIP install our requirement.txt, we do cleanup.
>
>
>
> # Cleanup JDK
>
> RUN apt-get remove -y openjdk-17-jdk-headless
>
> ENV JAVA_HOME=/opt/java/openjdk
>
>
>
> Hope it helps.
>
>
>
> Nix.
>
>
>
> *From: *George <ge...@gmail.com>
> *Date: *Wednesday, May 14, 2025 at 10:23 AM
> *To: *user@flink.apache.org <us...@flink.apache.org>
> *Subject: *Python based User defined function on Flink 1.19.1
>
> Hi all
>
>
>
> My Flink Jobmanager and Taskmanager is based on the community 1.19.1 on
> the Java 17 image.
>
>
>
> I've extended this by installing Python 3.11 and python3.11-venv.
>
>
>
> created my venv environment, activated it,
>
> when trying to pip install apache-flink. I first got an error saying
> /opt/java/openjdk/include does not exist... so created it.
>
> Next attempt ended in the below error.
>
>
>
> Please advise.
>
>
>
> G
>
>
>
> ```
>
> venv) flink@jobmanager:/sql/prompush$ pip install apache-flink
> Collecting apache-flink
> Downloading apache-flink-2.0.0.tar.gz (1.5 MB)
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 1.9 MB/s eta
> 0:00:00
> Installing build dependencies ... done
> Getting requirements to build wheel ... done
> Preparing metadata (pyproject.toml) ... done
> Collecting python-dateutil<3,>=2.8.0
> Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
> Collecting requests>=2.26.0
> Using cached requests-2.32.3-py3-none-any.whl (64 kB)
> Collecting pyarrow>=5.0.0
> Downloading pyarrow-20.0.0-cp311-cp311-manylinux_2_28_aarch64.whl (40.7
> MB)
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.7/40.7 MB 415.8 kB/s eta
> 0:00:00
> Collecting pytz>=2018.3
> Using cached pytz-2025.2-py2.py3-none-any.whl (509 kB)
> Collecting fastavro!=1.8.0,>=1.1.0
> Using cached
> fastavro-1.10.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
> (3.3 MB)
> Collecting pemja==0.4.1
> Downloading pemja-0.4.1.tar.gz (50 kB)
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.7/50.7 KB 600.0 kB/s eta
> 0:00:00
> Installing build dependencies ... done
> Getting requirements to build wheel ... error
> error: subprocess-exited-with-error
>
> × Getting requirements to build wheel did not run successfully.
> │ exit code: 255
> ╰─> [1 lines of output]
> jni.h should be in '/opt/java/openjdk/include/jni.h' but doesn't
> exist. Please check you've installed the JDK properly.
> [end of output]
>
> note: This error originates from a subprocess, and is likely not a
> problem with pip.
> error: subprocess-exited-with-error
>
> × Getting requirements to build wheel did not run successfully.
> │ exit code: 255
> ╰─> See above for output.
>
> note: This error originates from a subprocess, and is likely not a problem
> with pip.
>
> ```
>
>
>
> --
>
> You have the obligation to inform one honestly of the risk, and as a person
> you are committed to educate yourself to the total risk in any activity!
>
> Once informed & totally aware of the risk,
> every fool has the right to kill or injure themselves as they see fit!
>
--
You have the obligation to inform one honestly of the risk, and as a person
you are committed to educate yourself to the total risk in any activity!
Once informed & totally aware of the risk,
every fool has the right to kill or injure themselves as they see fit!
Re: Python based User defined function on Flink 1.19.1
Posted by George <ge...@gmail.com>.
Hi Nikola
awesome, thank you, it worked,got the basic example to work, so can now try
my solutions/code.
G
On Wed, May 14, 2025 at 3:52 PM George <ge...@gmail.com> wrote:
> Thanks.
>
> I would think this should rather be done up stream in the source image,
> for that matter I would have expected for this package to have been
> installed already as you can't really do allot onthe flink nodes with
> python without this package.
>
> G
>
> On Wed, May 14, 2025 at 3:33 PM Nikola Milutinovic <
> n.milutinovic@levi9.com> wrote:
>
>> Hmm, lemme see…
>>
>>
>>
>> Oh, yes, we also had to link Python bin:
>>
>>
>>
>> ldconfig /usr/lib
>>
>> ln -s /usr/bin/python3 /usr/bin/python
>>
>>
>>
>> Classic trickery.
>>
>>
>>
>> Nix,
>>
>>
>>
>> *From: *George <ge...@gmail.com>
>> *Date: *Wednesday, May 14, 2025 at 1:09 PM
>> *To: *Nikola Milutinovic <n....@levi9.com>, user@flink.apache.org
>> <us...@flink.apache.org>
>> *Subject: *Re: Python based User defined function on Flink 1.19.1
>>
>> Hi there
>>
>>
>>
>> Got it build :)
>>
>>
>>
>> I installed python3-pip in addition to the java - headless version, then
>> installed the package globally and then did the clean up.
>>
>>
>>
>> I am however getting the below now.
>>
>> it seems to be looking for python from the flink side and not python3
>>
>>
>>
>> ```
>>
>> flink@jobmanager:/sql/prompush$ /opt/flink/bin/flink run \
>> > -py /sql/prompush/main.py \
>> > -m jobmanager:8083
>> WARNING: Unknown module: jdk.compiler specified to --add-exports
>> WARNING: Unknown module: jdk.compiler specified to --add-exports
>> WARNING: Unknown module: jdk.compiler specified to --add-exports
>> WARNING: Unknown module: jdk.compiler specified to --add-exports
>> WARNING: Unknown module: jdk.compiler specified to --add-exports
>> org.apache.flink.client.program.ProgramAbortException:
>> java.io.IOException: Cannot run program "python": error=2, No such file or
>> directory
>> at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:134)
>> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown
>> Source)
>> at
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
>> Source)
>> at java.base/java.lang.reflect.Method.invoke(Unknown Source)
>> at
>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:356)
>> at
>> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:223)
>> at
>> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:113)
>> at
>> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:1026)
>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:247)
>> at
>> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1270)
>> at
>> org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$10(CliFrontend.java:1367)
>> at java.base/java.security.AccessController.doPrivileged(Unknown Source)
>> at java.base/javax.security.auth.Subject.doAs(Unknown Source)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>> at
>> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>> at
>> org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1367)
>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1335)
>> Caused by: java.io.IOException: Cannot run program "python": error=2, No
>> such file or directory
>> at java.base/java.lang.ProcessBuilder.start(Unknown Source)
>> at java.base/java.lang.ProcessBuilder.start(Unknown Source)
>> at
>> org.apache.flink.client.python.PythonEnvUtils.startPythonProcess(PythonEnvUtils.java:378)
>> at
>> org.apache.flink.client.python.PythonEnvUtils.launchPy4jPythonClient(PythonEnvUtils.java:492)
>> at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:92)
>> ... 17 more
>> Caused by: java.io.IOException: error=2, No such file or directory
>> at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
>> at java.base/java.lang.ProcessImpl.<init>(Unknown Source)
>> at java.base/java.lang.ProcessImpl.start(Unknown Source)
>> ... 22 more
>>
>> ```
>>
>>
>>
>> On Wed, May 14, 2025 at 11:50 AM George <ge...@gmail.com> wrote:
>>
>> Hi there
>>
>>
>>
>> Wonder if you can assist.
>>
>>
>>
>> First thinking of installing apache-flink into a dedicated environment,
>> see my docker file attached... this does result in the attached err.txt
>> stack though.
>>
>> Wonder if it does not simply make more sense to install apache-flink
>> globally, then everyone that works with python on this node would have
>> access to it.
>>
>>
>>
>> personally i would have expected this have formed part of this image
>> available...
>>
>>
>>
>> G
>>
>>
>>
>> On Wed, May 14, 2025 at 11:18 AM Nikola Milutinovic <
>> n.milutinovic@levi9.com> wrote:
>>
>> Hi George.
>>
>>
>>
>> We saw the same problem, running Apache Flink 1.19 and 1.20 images. The
>> cause is that Flink image provides a JRE and you need JDK to build/install
>> PyFlink. And, oddly enough, I think it was only on ARM64 images. Amd64 was
>> OK, I think. So, Mac M1, M2, M3…
>>
>>
>>
>> Our Docker file for building our custom image has some extra steps, to
>> install JDK, pip-install and then cleanup.
>>
>>
>>
>> RUN apt-get update -y \
>>
>> && apt-get upgrade -y \
>>
>> && apt-get install --no-install-recommends -y \
>>
>> …
>>
>> openjdk-17-jdk-headless \
>>
>> ENV JAVA_HOME=/usr/lib/jvm/java-17-openjdk-arm64
>>
>>
>>
>> After PIP install our requirement.txt, we do cleanup.
>>
>>
>>
>> # Cleanup JDK
>>
>> RUN apt-get remove -y openjdk-17-jdk-headless
>>
>> ENV JAVA_HOME=/opt/java/openjdk
>>
>>
>>
>> Hope it helps.
>>
>>
>>
>> Nix.
>>
>>
>>
>> *From: *George <ge...@gmail.com>
>> *Date: *Wednesday, May 14, 2025 at 10:23 AM
>> *To: *user@flink.apache.org <us...@flink.apache.org>
>> *Subject: *Python based User defined function on Flink 1.19.1
>>
>> Hi all
>>
>>
>>
>> My Flink Jobmanager and Taskmanager is based on the community 1.19.1 on
>> the Java 17 image.
>>
>>
>>
>> I've extended this by installing Python 3.11 and python3.11-venv.
>>
>>
>>
>> created my venv environment, activated it,
>>
>> when trying to pip install apache-flink. I first got an error saying
>> /opt/java/openjdk/include does not exist... so created it.
>>
>> Next attempt ended in the below error.
>>
>>
>>
>> Please advise.
>>
>>
>>
>> G
>>
>>
>>
>> ```
>>
>> venv) flink@jobmanager:/sql/prompush$ pip install apache-flink
>> Collecting apache-flink
>> Downloading apache-flink-2.0.0.tar.gz (1.5 MB)
>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 1.9 MB/s eta
>> 0:00:00
>> Installing build dependencies ... done
>> Getting requirements to build wheel ... done
>> Preparing metadata (pyproject.toml) ... done
>> Collecting python-dateutil<3,>=2.8.0
>> Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
>> Collecting requests>=2.26.0
>> Using cached requests-2.32.3-py3-none-any.whl (64 kB)
>> Collecting pyarrow>=5.0.0
>> Downloading pyarrow-20.0.0-cp311-cp311-manylinux_2_28_aarch64.whl (40.7
>> MB)
>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.7/40.7 MB 415.8 kB/s
>> eta 0:00:00
>> Collecting pytz>=2018.3
>> Using cached pytz-2025.2-py2.py3-none-any.whl (509 kB)
>> Collecting fastavro!=1.8.0,>=1.1.0
>> Using cached
>> fastavro-1.10.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
>> (3.3 MB)
>> Collecting pemja==0.4.1
>> Downloading pemja-0.4.1.tar.gz (50 kB)
>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.7/50.7 KB 600.0 kB/s
>> eta 0:00:00
>> Installing build dependencies ... done
>> Getting requirements to build wheel ... error
>> error: subprocess-exited-with-error
>>
>> × Getting requirements to build wheel did not run successfully.
>> │ exit code: 255
>> ╰─> [1 lines of output]
>> jni.h should be in '/opt/java/openjdk/include/jni.h' but doesn't
>> exist. Please check you've installed the JDK properly.
>> [end of output]
>>
>> note: This error originates from a subprocess, and is likely not a
>> problem with pip.
>> error: subprocess-exited-with-error
>>
>> × Getting requirements to build wheel did not run successfully.
>> │ exit code: 255
>> ╰─> See above for output.
>>
>> note: This error originates from a subprocess, and is likely not a
>> problem with pip.
>>
>> ```
>>
>>
>>
>> --
>>
>> You have the obligation to inform one honestly of the risk, and as a
>> person
>> you are committed to educate yourself to the total risk in any activity!
>>
>> Once informed & totally aware of the risk,
>> every fool has the right to kill or injure themselves as they see fit!
>>
>>
>>
>>
>> --
>>
>> You have the obligation to inform one honestly of the risk, and as a
>> person
>> you are committed to educate yourself to the total risk in any activity!
>>
>> Once informed & totally aware of the risk,
>> every fool has the right to kill or injure themselves as they see fit!
>>
>>
>>
>>
>> --
>>
>> You have the obligation to inform one honestly of the risk, and as a
>> person
>> you are committed to educate yourself to the total risk in any activity!
>>
>> Once informed & totally aware of the risk,
>> every fool has the right to kill or injure themselves as they see fit!
>>
>
>
> --
> You have the obligation to inform one honestly of the risk, and as a person
> you are committed to educate yourself to the total risk in any activity!
>
> Once informed & totally aware of the risk,
> every fool has the right to kill or injure themselves as they see fit!
>
--
You have the obligation to inform one honestly of the risk, and as a person
you are committed to educate yourself to the total risk in any activity!
Once informed & totally aware of the risk,
every fool has the right to kill or injure themselves as they see fit!
Re: Python based User defined function on Flink 1.19.1
Posted by George <ge...@gmail.com>.
Thanks.
I would think this should rather be done up stream in the source image, for
that matter I would have expected for this package to have been installed
already as you can't really do allot onthe flink nodes with python without
this package.
G
On Wed, May 14, 2025 at 3:33 PM Nikola Milutinovic <n....@levi9.com>
wrote:
> Hmm, lemme see…
>
>
>
> Oh, yes, we also had to link Python bin:
>
>
>
> ldconfig /usr/lib
>
> ln -s /usr/bin/python3 /usr/bin/python
>
>
>
> Classic trickery.
>
>
>
> Nix,
>
>
>
> *From: *George <ge...@gmail.com>
> *Date: *Wednesday, May 14, 2025 at 1:09 PM
> *To: *Nikola Milutinovic <n....@levi9.com>, user@flink.apache.org
> <us...@flink.apache.org>
> *Subject: *Re: Python based User defined function on Flink 1.19.1
>
> Hi there
>
>
>
> Got it build :)
>
>
>
> I installed python3-pip in addition to the java - headless version, then
> installed the package globally and then did the clean up.
>
>
>
> I am however getting the below now.
>
> it seems to be looking for python from the flink side and not python3
>
>
>
> ```
>
> flink@jobmanager:/sql/prompush$ /opt/flink/bin/flink run \
> > -py /sql/prompush/main.py \
> > -m jobmanager:8083
> WARNING: Unknown module: jdk.compiler specified to --add-exports
> WARNING: Unknown module: jdk.compiler specified to --add-exports
> WARNING: Unknown module: jdk.compiler specified to --add-exports
> WARNING: Unknown module: jdk.compiler specified to --add-exports
> WARNING: Unknown module: jdk.compiler specified to --add-exports
> org.apache.flink.client.program.ProgramAbortException:
> java.io.IOException: Cannot run program "python": error=2, No such file or
> directory
> at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:134)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown
> Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> Source)
> at java.base/java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:356)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:223)
> at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:113)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:1026)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:247)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1270)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$10(CliFrontend.java:1367)
> at java.base/java.security.AccessController.doPrivileged(Unknown Source)
> at java.base/javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at
> org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1367)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1335)
> Caused by: java.io.IOException: Cannot run program "python": error=2, No
> such file or directory
> at java.base/java.lang.ProcessBuilder.start(Unknown Source)
> at java.base/java.lang.ProcessBuilder.start(Unknown Source)
> at
> org.apache.flink.client.python.PythonEnvUtils.startPythonProcess(PythonEnvUtils.java:378)
> at
> org.apache.flink.client.python.PythonEnvUtils.launchPy4jPythonClient(PythonEnvUtils.java:492)
> at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:92)
> ... 17 more
> Caused by: java.io.IOException: error=2, No such file or directory
> at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
> at java.base/java.lang.ProcessImpl.<init>(Unknown Source)
> at java.base/java.lang.ProcessImpl.start(Unknown Source)
> ... 22 more
>
> ```
>
>
>
> On Wed, May 14, 2025 at 11:50 AM George <ge...@gmail.com> wrote:
>
> Hi there
>
>
>
> Wonder if you can assist.
>
>
>
> First thinking of installing apache-flink into a dedicated environment,
> see my docker file attached... this does result in the attached err.txt
> stack though.
>
> Wonder if it does not simply make more sense to install apache-flink
> globally, then everyone that works with python on this node would have
> access to it.
>
>
>
> personally i would have expected this have formed part of this image
> available...
>
>
>
> G
>
>
>
> On Wed, May 14, 2025 at 11:18 AM Nikola Milutinovic <
> n.milutinovic@levi9.com> wrote:
>
> Hi George.
>
>
>
> We saw the same problem, running Apache Flink 1.19 and 1.20 images. The
> cause is that Flink image provides a JRE and you need JDK to build/install
> PyFlink. And, oddly enough, I think it was only on ARM64 images. Amd64 was
> OK, I think. So, Mac M1, M2, M3…
>
>
>
> Our Docker file for building our custom image has some extra steps, to
> install JDK, pip-install and then cleanup.
>
>
>
> RUN apt-get update -y \
>
> && apt-get upgrade -y \
>
> && apt-get install --no-install-recommends -y \
>
> …
>
> openjdk-17-jdk-headless \
>
> ENV JAVA_HOME=/usr/lib/jvm/java-17-openjdk-arm64
>
>
>
> After PIP install our requirement.txt, we do cleanup.
>
>
>
> # Cleanup JDK
>
> RUN apt-get remove -y openjdk-17-jdk-headless
>
> ENV JAVA_HOME=/opt/java/openjdk
>
>
>
> Hope it helps.
>
>
>
> Nix.
>
>
>
> *From: *George <ge...@gmail.com>
> *Date: *Wednesday, May 14, 2025 at 10:23 AM
> *To: *user@flink.apache.org <us...@flink.apache.org>
> *Subject: *Python based User defined function on Flink 1.19.1
>
> Hi all
>
>
>
> My Flink Jobmanager and Taskmanager is based on the community 1.19.1 on
> the Java 17 image.
>
>
>
> I've extended this by installing Python 3.11 and python3.11-venv.
>
>
>
> created my venv environment, activated it,
>
> when trying to pip install apache-flink. I first got an error saying
> /opt/java/openjdk/include does not exist... so created it.
>
> Next attempt ended in the below error.
>
>
>
> Please advise.
>
>
>
> G
>
>
>
> ```
>
> venv) flink@jobmanager:/sql/prompush$ pip install apache-flink
> Collecting apache-flink
> Downloading apache-flink-2.0.0.tar.gz (1.5 MB)
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 1.9 MB/s eta
> 0:00:00
> Installing build dependencies ... done
> Getting requirements to build wheel ... done
> Preparing metadata (pyproject.toml) ... done
> Collecting python-dateutil<3,>=2.8.0
> Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
> Collecting requests>=2.26.0
> Using cached requests-2.32.3-py3-none-any.whl (64 kB)
> Collecting pyarrow>=5.0.0
> Downloading pyarrow-20.0.0-cp311-cp311-manylinux_2_28_aarch64.whl (40.7
> MB)
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.7/40.7 MB 415.8 kB/s eta
> 0:00:00
> Collecting pytz>=2018.3
> Using cached pytz-2025.2-py2.py3-none-any.whl (509 kB)
> Collecting fastavro!=1.8.0,>=1.1.0
> Using cached
> fastavro-1.10.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
> (3.3 MB)
> Collecting pemja==0.4.1
> Downloading pemja-0.4.1.tar.gz (50 kB)
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.7/50.7 KB 600.0 kB/s eta
> 0:00:00
> Installing build dependencies ... done
> Getting requirements to build wheel ... error
> error: subprocess-exited-with-error
>
> × Getting requirements to build wheel did not run successfully.
> │ exit code: 255
> ╰─> [1 lines of output]
> jni.h should be in '/opt/java/openjdk/include/jni.h' but doesn't
> exist. Please check you've installed the JDK properly.
> [end of output]
>
> note: This error originates from a subprocess, and is likely not a
> problem with pip.
> error: subprocess-exited-with-error
>
> × Getting requirements to build wheel did not run successfully.
> │ exit code: 255
> ╰─> See above for output.
>
> note: This error originates from a subprocess, and is likely not a problem
> with pip.
>
> ```
>
>
>
> --
>
> You have the obligation to inform one honestly of the risk, and as a person
> you are committed to educate yourself to the total risk in any activity!
>
> Once informed & totally aware of the risk,
> every fool has the right to kill or injure themselves as they see fit!
>
>
>
>
> --
>
> You have the obligation to inform one honestly of the risk, and as a person
> you are committed to educate yourself to the total risk in any activity!
>
> Once informed & totally aware of the risk,
> every fool has the right to kill or injure themselves as they see fit!
>
>
>
>
> --
>
> You have the obligation to inform one honestly of the risk, and as a person
> you are committed to educate yourself to the total risk in any activity!
>
> Once informed & totally aware of the risk,
> every fool has the right to kill or injure themselves as they see fit!
>
--
You have the obligation to inform one honestly of the risk, and as a person
you are committed to educate yourself to the total risk in any activity!
Once informed & totally aware of the risk,
every fool has the right to kill or injure themselves as they see fit!
Re: Python based User defined function on Flink 1.19.1
Posted by Nikola Milutinovic <n....@levi9.com>.
Hmm, lemme see…
Oh, yes, we also had to link Python bin:
ldconfig /usr/lib
ln -s /usr/bin/python3 /usr/bin/python
Classic trickery.
Nix,
From: George <ge...@gmail.com>
Date: Wednesday, May 14, 2025 at 1:09 PM
To: Nikola Milutinovic <n....@levi9.com>, user@flink.apache.org <us...@flink.apache.org>
Subject: Re: Python based User defined function on Flink 1.19.1
Hi there
Got it build :)
I installed python3-pip in addition to the java - headless version, then installed the package globally and then did the clean up.
I am however getting the below now.
it seems to be looking for python from the flink side and not python3
```
flink@jobmanager:/sql/prompush$ /opt/flink/bin/flink run \
> -py /sql/prompush/main.py \
> -m jobmanager:8083
WARNING: Unknown module: jdk.compiler specified to --add-exports
WARNING: Unknown module: jdk.compiler specified to --add-exports
WARNING: Unknown module: jdk.compiler specified to --add-exports
WARNING: Unknown module: jdk.compiler specified to --add-exports
WARNING: Unknown module: jdk.compiler specified to --add-exports
org.apache.flink.client.program.ProgramAbortException: java.io.IOException: Cannot run program "python": error=2, No such file or directory
at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:134)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:356)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:223)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:113)
at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:1026)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:247)
at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1270)
at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$10(CliFrontend.java:1367)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1367)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1335)
Caused by: java.io.IOException: Cannot run program "python": error=2, No such file or directory
at java.base/java.lang.ProcessBuilder.start(Unknown Source)
at java.base/java.lang.ProcessBuilder.start(Unknown Source)
at org.apache.flink.client.python.PythonEnvUtils.startPythonProcess(PythonEnvUtils.java:378)
at org.apache.flink.client.python.PythonEnvUtils.launchPy4jPythonClient(PythonEnvUtils.java:492)
at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:92)
... 17 more
Caused by: java.io.IOException: error=2, No such file or directory
at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
at java.base/java.lang.ProcessImpl.<init>(Unknown Source)
at java.base/java.lang.ProcessImpl.start(Unknown Source)
... 22 more
```
On Wed, May 14, 2025 at 11:50 AM George <ge...@gmail.com>> wrote:
Hi there
Wonder if you can assist.
First thinking of installing apache-flink into a dedicated environment, see my docker file attached... this does result in the attached err.txt stack though.
Wonder if it does not simply make more sense to install apache-flink globally, then everyone that works with python on this node would have access to it.
personally i would have expected this have formed part of this image available...
G
On Wed, May 14, 2025 at 11:18 AM Nikola Milutinovic <n....@levi9.com>> wrote:
Hi George.
We saw the same problem, running Apache Flink 1.19 and 1.20 images. The cause is that Flink image provides a JRE and you need JDK to build/install PyFlink. And, oddly enough, I think it was only on ARM64 images. Amd64 was OK, I think. So, Mac M1, M2, M3…
Our Docker file for building our custom image has some extra steps, to install JDK, pip-install and then cleanup.
RUN apt-get update -y \
&& apt-get upgrade -y \
&& apt-get install --no-install-recommends -y \
…
openjdk-17-jdk-headless \
ENV JAVA_HOME=/usr/lib/jvm/java-17-openjdk-arm64
After PIP install our requirement.txt, we do cleanup.
# Cleanup JDK
RUN apt-get remove -y openjdk-17-jdk-headless
ENV JAVA_HOME=/opt/java/openjdk
Hope it helps.
Nix.
From: George <ge...@gmail.com>>
Date: Wednesday, May 14, 2025 at 10:23 AM
To: user@flink.apache.org<ma...@flink.apache.org> <us...@flink.apache.org>>
Subject: Python based User defined function on Flink 1.19.1
Hi all
My Flink Jobmanager and Taskmanager is based on the community 1.19.1 on the Java 17 image.
I've extended this by installing Python 3.11 and python3.11-venv.
created my venv environment, activated it,
when trying to pip install apache-flink. I first got an error saying /opt/java/openjdk/include does not exist... so created it.
Next attempt ended in the below error.
Please advise.
G
```
venv) flink@jobmanager:/sql/prompush$ pip install apache-flink
Collecting apache-flink
Downloading apache-flink-2.0.0.tar.gz (1.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 1.9 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting python-dateutil<3,>=2.8.0
Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Collecting requests>=2.26.0
Using cached requests-2.32.3-py3-none-any.whl (64 kB)
Collecting pyarrow>=5.0.0
Downloading pyarrow-20.0.0-cp311-cp311-manylinux_2_28_aarch64.whl (40.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.7/40.7 MB 415.8 kB/s eta 0:00:00
Collecting pytz>=2018.3
Using cached pytz-2025.2-py2.py3-none-any.whl (509 kB)
Collecting fastavro!=1.8.0,>=1.1.0
Using cached fastavro-1.10.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.3 MB)
Collecting pemja==0.4.1
Downloading pemja-0.4.1.tar.gz (50 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.7/50.7 KB 600.0 kB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 255
╰─> [1 lines of output]
jni.h should be in '/opt/java/openjdk/include/jni.h' but doesn't exist. Please check you've installed the JDK properly.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 255
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
```
--
You have the obligation to inform one honestly of the risk, and as a person
you are committed to educate yourself to the total risk in any activity!
Once informed & totally aware of the risk,
every fool has the right to kill or injure themselves as they see fit!
--
You have the obligation to inform one honestly of the risk, and as a person
you are committed to educate yourself to the total risk in any activity!
Once informed & totally aware of the risk,
every fool has the right to kill or injure themselves as they see fit!
--
You have the obligation to inform one honestly of the risk, and as a person
you are committed to educate yourself to the total risk in any activity!
Once informed & totally aware of the risk,
every fool has the right to kill or injure themselves as they see fit!
Re: Python based User defined function on Flink 1.19.1
Posted by George <ge...@gmail.com>.
Hi there
Got it build :)
I installed python3-pip in addition to the java - headless version, then
installed the package globally and then did the clean up.
I am however getting the below now.
it seems to be looking for python from the flink side and not python3
```
flink@jobmanager:/sql/prompush$ /opt/flink/bin/flink run \
> -py /sql/prompush/main.py \
> -m jobmanager:8083
WARNING: Unknown module: jdk.compiler specified to --add-exports
WARNING: Unknown module: jdk.compiler specified to --add-exports
WARNING: Unknown module: jdk.compiler specified to --add-exports
WARNING: Unknown module: jdk.compiler specified to --add-exports
WARNING: Unknown module: jdk.compiler specified to --add-exports
org.apache.flink.client.program.ProgramAbortException: java.io.IOException:
Cannot run program "python": error=2, No such file or directory
at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:134)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:356)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:223)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:113)
at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:1026)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:247)
at
org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1270)
at
org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$10(CliFrontend.java:1367)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
at
org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at
org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1367)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1335)
Caused by: java.io.IOException: Cannot run program "python": error=2, No
such file or directory
at java.base/java.lang.ProcessBuilder.start(Unknown Source)
at java.base/java.lang.ProcessBuilder.start(Unknown Source)
at
org.apache.flink.client.python.PythonEnvUtils.startPythonProcess(PythonEnvUtils.java:378)
at
org.apache.flink.client.python.PythonEnvUtils.launchPy4jPythonClient(PythonEnvUtils.java:492)
at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:92)
... 17 more
Caused by: java.io.IOException: error=2, No such file or directory
at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
at java.base/java.lang.ProcessImpl.<init>(Unknown Source)
at java.base/java.lang.ProcessImpl.start(Unknown Source)
... 22 more
```
On Wed, May 14, 2025 at 11:50 AM George <ge...@gmail.com> wrote:
> Hi there
>
> Wonder if you can assist.
>
> First thinking of installing apache-flink into a dedicated environment,
> see my docker file attached... this does result in the attached err.txt
> stack though.
> Wonder if it does not simply make more sense to install apache-flink
> globally, then everyone that works with python on this node would have
> access to it.
>
> personally i would have expected this have formed part of this image
> available...
>
> G
>
> On Wed, May 14, 2025 at 11:18 AM Nikola Milutinovic <
> n.milutinovic@levi9.com> wrote:
>
>> Hi George.
>>
>>
>>
>> We saw the same problem, running Apache Flink 1.19 and 1.20 images. The
>> cause is that Flink image provides a JRE and you need JDK to build/install
>> PyFlink. And, oddly enough, I think it was only on ARM64 images. Amd64 was
>> OK, I think. So, Mac M1, M2, M3…
>>
>>
>>
>> Our Docker file for building our custom image has some extra steps, to
>> install JDK, pip-install and then cleanup.
>>
>>
>>
>> RUN apt-get update -y \
>>
>> && apt-get upgrade -y \
>>
>> && apt-get install --no-install-recommends -y \
>>
>> …
>>
>> openjdk-17-jdk-headless \
>>
>> ENV JAVA_HOME=/usr/lib/jvm/java-17-openjdk-arm64
>>
>>
>>
>> After PIP install our requirement.txt, we do cleanup.
>>
>>
>>
>> # Cleanup JDK
>>
>> RUN apt-get remove -y openjdk-17-jdk-headless
>>
>> ENV JAVA_HOME=/opt/java/openjdk
>>
>>
>>
>> Hope it helps.
>>
>>
>>
>> Nix.
>>
>>
>>
>> *From: *George <ge...@gmail.com>
>> *Date: *Wednesday, May 14, 2025 at 10:23 AM
>> *To: *user@flink.apache.org <us...@flink.apache.org>
>> *Subject: *Python based User defined function on Flink 1.19.1
>>
>> Hi all
>>
>>
>>
>> My Flink Jobmanager and Taskmanager is based on the community 1.19.1 on
>> the Java 17 image.
>>
>>
>>
>> I've extended this by installing Python 3.11 and python3.11-venv.
>>
>>
>>
>> created my venv environment, activated it,
>>
>> when trying to pip install apache-flink. I first got an error saying
>> /opt/java/openjdk/include does not exist... so created it.
>>
>> Next attempt ended in the below error.
>>
>>
>>
>> Please advise.
>>
>>
>>
>> G
>>
>>
>>
>> ```
>>
>> venv) flink@jobmanager:/sql/prompush$ pip install apache-flink
>> Collecting apache-flink
>> Downloading apache-flink-2.0.0.tar.gz (1.5 MB)
>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 1.9 MB/s eta
>> 0:00:00
>> Installing build dependencies ... done
>> Getting requirements to build wheel ... done
>> Preparing metadata (pyproject.toml) ... done
>> Collecting python-dateutil<3,>=2.8.0
>> Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
>> Collecting requests>=2.26.0
>> Using cached requests-2.32.3-py3-none-any.whl (64 kB)
>> Collecting pyarrow>=5.0.0
>> Downloading pyarrow-20.0.0-cp311-cp311-manylinux_2_28_aarch64.whl (40.7
>> MB)
>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.7/40.7 MB 415.8 kB/s
>> eta 0:00:00
>> Collecting pytz>=2018.3
>> Using cached pytz-2025.2-py2.py3-none-any.whl (509 kB)
>> Collecting fastavro!=1.8.0,>=1.1.0
>> Using cached
>> fastavro-1.10.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
>> (3.3 MB)
>> Collecting pemja==0.4.1
>> Downloading pemja-0.4.1.tar.gz (50 kB)
>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.7/50.7 KB 600.0 kB/s
>> eta 0:00:00
>> Installing build dependencies ... done
>> Getting requirements to build wheel ... error
>> error: subprocess-exited-with-error
>>
>> × Getting requirements to build wheel did not run successfully.
>> │ exit code: 255
>> ╰─> [1 lines of output]
>> jni.h should be in '/opt/java/openjdk/include/jni.h' but doesn't
>> exist. Please check you've installed the JDK properly.
>> [end of output]
>>
>> note: This error originates from a subprocess, and is likely not a
>> problem with pip.
>> error: subprocess-exited-with-error
>>
>> × Getting requirements to build wheel did not run successfully.
>> │ exit code: 255
>> ╰─> See above for output.
>>
>> note: This error originates from a subprocess, and is likely not a
>> problem with pip.
>>
>> ```
>>
>>
>>
>> --
>>
>> You have the obligation to inform one honestly of the risk, and as a
>> person
>> you are committed to educate yourself to the total risk in any activity!
>>
>> Once informed & totally aware of the risk,
>> every fool has the right to kill or injure themselves as they see fit!
>>
>
>
> --
> You have the obligation to inform one honestly of the risk, and as a person
> you are committed to educate yourself to the total risk in any activity!
>
> Once informed & totally aware of the risk,
> every fool has the right to kill or injure themselves as they see fit!
>
--
You have the obligation to inform one honestly of the risk, and as a person
you are committed to educate yourself to the total risk in any activity!
Once informed & totally aware of the risk,
every fool has the right to kill or injure themselves as they see fit!
Re: Python based User defined function on Flink 1.19.1
Posted by Nikola Milutinovic <n....@levi9.com>.
Hi George.
We saw the same problem, running Apache Flink 1.19 and 1.20 images. The cause is that Flink image provides a JRE and you need JDK to build/install PyFlink. And, oddly enough, I think it was only on ARM64 images. Amd64 was OK, I think. So, Mac M1, M2, M3…
Our Docker file for building our custom image has some extra steps, to install JDK, pip-install and then cleanup.
RUN apt-get update -y \
&& apt-get upgrade -y \
&& apt-get install --no-install-recommends -y \
…
openjdk-17-jdk-headless \
ENV JAVA_HOME=/usr/lib/jvm/java-17-openjdk-arm64
After PIP install our requirement.txt, we do cleanup.
# Cleanup JDK
RUN apt-get remove -y openjdk-17-jdk-headless
ENV JAVA_HOME=/opt/java/openjdk
Hope it helps.
Nix.
From: George <ge...@gmail.com>
Date: Wednesday, May 14, 2025 at 10:23 AM
To: user@flink.apache.org <us...@flink.apache.org>
Subject: Python based User defined function on Flink 1.19.1
Hi all
My Flink Jobmanager and Taskmanager is based on the community 1.19.1 on the Java 17 image.
I've extended this by installing Python 3.11 and python3.11-venv.
created my venv environment, activated it,
when trying to pip install apache-flink. I first got an error saying /opt/java/openjdk/include does not exist... so created it.
Next attempt ended in the below error.
Please advise.
G
```
venv) flink@jobmanager:/sql/prompush$ pip install apache-flink
Collecting apache-flink
Downloading apache-flink-2.0.0.tar.gz (1.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 1.9 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting python-dateutil<3,>=2.8.0
Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Collecting requests>=2.26.0
Using cached requests-2.32.3-py3-none-any.whl (64 kB)
Collecting pyarrow>=5.0.0
Downloading pyarrow-20.0.0-cp311-cp311-manylinux_2_28_aarch64.whl (40.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.7/40.7 MB 415.8 kB/s eta 0:00:00
Collecting pytz>=2018.3
Using cached pytz-2025.2-py2.py3-none-any.whl (509 kB)
Collecting fastavro!=1.8.0,>=1.1.0
Using cached fastavro-1.10.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.3 MB)
Collecting pemja==0.4.1
Downloading pemja-0.4.1.tar.gz (50 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.7/50.7 KB 600.0 kB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 255
╰─> [1 lines of output]
jni.h should be in '/opt/java/openjdk/include/jni.h' but doesn't exist. Please check you've installed the JDK properly.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 255
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
```
--
You have the obligation to inform one honestly of the risk, and as a person
you are committed to educate yourself to the total risk in any activity!
Once informed & totally aware of the risk,
every fool has the right to kill or injure themselves as they see fit!