You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/05/05 10:44:31 UTC
[GitHub] [airflow] mfumagalli68 opened a new issue #8716: Support on packaged dags
mfumagalli68 opened a new issue #8716:
URL: https://github.com/apache/airflow/issues/8716
I'm trying to use `apache airlfow` with packaged dags.
I've written my code as a python package and my code depends on other libraries such as numpy, scipy etc.
This is `setup.py` of my custom python package:
```python
from setuptools import setup, find_packages
from pathlib import Path
from typing import List
import distutils.text_file
def parse_requirements(filename: str) -> List[str]:
"""Return requirements from requirements file."""
# Ref: https://stackoverflow.com/a/42033122/
return distutils.text_file.TextFile(filename=str(Path(__file__).with_name(filename))).readlines()
setup(name='classify_business',
version='0.1',
python_requires=">=3.6",
description='desc',
url='https://urlgitlab/datascience/classifybusiness',
author='Marco fumagalli',
author_email='marco.fumagalli@mycompany.com',
packages = find_packages(),
license='MIT',
install_requires=
parse_requirements('requirements.txt'),
zip_safe=False,
include_package_data=True)
```
requirements.txt contains packages ( vertica_python, pandas, numpy etc) along with their version needed for my code.
I wrote a litte shell script based on the one provied in the doc for creating packaged dags:
```bash
set -eu -o pipefail
if [ $# == 0 ]; then
echo "First param should be /srv/user_name/virtualenvs/name_virtual_env"
echo "Second param should be name of temp_directory"
echo "Third param directory should be git url"
echo "Fourth param should be dag zip name, i.e dag_zip.zip to be copied into AIRFLOW__CORE__DAGS__FOLDER"
echo "Fifth param should be package name, i.e classify_business"
fi
venv_path=${1}
dir_tmp=${2}
git_url=${3}
dag_zip=${4}
pkg_name=${5}
python3 -m venv $venv_path
source $venv_path/bin/activate
mkdir $dir_tmp
cd $dir_tmp
python3 -m pip install --prefix=$PWD git+$git_url
zip -r $dag_zip *
cp $dag_zip $AIRFLOW__CORE__DAGS_FOLDER
rm -r $dir_tmp
```
The shell will install my package along with dependencies directly from gitlab, zip and then move to the dags folder.
This is the content of the folder tmp_dir before being zipped.
```
bin
lib
lib64
predict_dag.py
train_dag.py
```
Airflow doesn't seem to be able to import package installed in lib or lib64.
I'm getting this error
> ModuleNotFoundError: No module named 'vertica_python'
I even tried to move my custom package outside of lib:
```
bin
my_custom_package
lib
lib64
predict_dag.py
train_dag.py
```
But still getting same error.
One of the problem I think relies on how to use `pip` to install package in a specific location.
Airflow example use `--install-option="--install-lib=/path/"` but it's unsupported:
> Location-changing options found in --install-option: ['--install-lib']
> from command line. This configuration may cause unexpected behavior
> and is unsupported. pip 20.2 will remove support for this
> functionality. A possible replacement is using pip-level options like
> --user, --prefix, --root, and --target. You can find discussion regarding this at https://github.com/pypa/pip/issues/7309.
Using `--prefix` leads to a structure like above, with module not found error.
Using `--target` leads to every package installed in the directory specified.
In this case I have a pandas related error
> C extension: No module named 'pandas._libs.tslibs.conversion' not built
I guess that it's related to dynamic libraries that should be available at a system level?
I really don't know how to do that.
Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mfumagalli68 edited a comment on issue #8716: Support on packaged dags
Posted by GitBox <gi...@apache.org>.
mfumagalli68 edited a comment on issue #8716:
URL: https://github.com/apache/airflow/issues/8716#issuecomment-624008547
Ok.
But what should I do about `--install-option="--install-lib=/path/"` ?
And what's the best practice for dealing dependency problems?
What I've seen so far:
- `PythonVirtualEnvOperator`. Problem: create and destory virtualenv on the fly. Not feasible for a project with lots of dependencies.
- Use `BashOperator`: write a shell which execute a python script with virtualenv interpreter. Seems a workaround.
- `DockerOperator`: Is this the only option we have?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mfumagalli68 edited a comment on issue #8716: Support on packaged dags
Posted by GitBox <gi...@apache.org>.
mfumagalli68 edited a comment on issue #8716:
URL: https://github.com/apache/airflow/issues/8716#issuecomment-624008547
Ok.
But what should I do about `--install-option="--install-lib=/path/"` ?
And what's the best practice for dealing with dependency problems?
What I've seen so far:
- `PythonVirtualEnvOperator`. Problem: create and destory virtualenv on the fly. Not feasible for a project with lots of dependencies.
- Use `BashOperator`: write a shell which execute a python script with virtualenv interpreter. Seems a workaround.
- `DockerOperator`: Is this the only option we have?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #8716: Support on packaged dags
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #8716:
URL: https://github.com/apache/airflow/issues/8716#issuecomment-623986153
> packaged dags cannot contain dynamic libraries (eg. libz.so) these need to be available on the system if a module needs those. In other words only pure python modules can be packaged.
https://airflow.readthedocs.io/en/latest/concepts.html?highlight=packaged%20dags#packaged-dags
Packaged DAGs is only a partial solution to the dependency problem because it only allows you to load simple Python libraries. All complex math libraries are not supported.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #8716: Support on packaged dags
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #8716:
URL: https://github.com/apache/airflow/issues/8716#issuecomment-623986153
> packaged dags cannot contain dynamic libraries (eg. libz.so) these need to be available on the system if a module needs those. In other words only pure python modules can be packaged.
https://airflow.readthedocs.io/en/latest/concepts.html?highlight=packaged%20dags#packaged-dags
Packaged DAGs is only a partial solution to the dependency problem because it only allows you to load Python libraries. All complex math libraries are not supported.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mfumagalli68 edited a comment on issue #8716: Support on packaged dags
Posted by GitBox <gi...@apache.org>.
mfumagalli68 edited a comment on issue #8716:
URL: https://github.com/apache/airflow/issues/8716#issuecomment-624008547
Ok so what's the best practice for dealing dependency problems?
What I've seen so far:
- `PythonVirtualEnvOperator`. Problem: create and destory virtualenv on the fly. Not feasible for a project with lots of dependencies.
- Use `BashOperator`: write a shell which execute a python script with virtualenv interpreter. Seems a workaround.
Morover I cannot pass complex kwargs to the operator. For example `params={'A':{'B':[1,2,3]}}` won't work.
- `DockerOperator`: Is this the only option we have?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mfumagalli68 edited a comment on issue #8716: Support on packaged dags
Posted by GitBox <gi...@apache.org>.
mfumagalli68 edited a comment on issue #8716:
URL: https://github.com/apache/airflow/issues/8716#issuecomment-624008547
Ok.
But what should I do about `--install-option="--install-lib=/path/"` ?
And what's the best practice for dealing dependency problems?
What I've seen so far:
- `PythonVirtualEnvOperator`. Problem: create and destory virtualenv on the fly. Not feasible for a project with lots of dependencies.
- Use `BashOperator`: write a shell which execute a python script with virtualenv interpreter. Seems a workaround.
Morover I cannot pass complex kwargs to the operator. For example `params={'A':{'B':[1,2,3]}}` won't work.
- `DockerOperator`: Is this the only option we have?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #8716: Support on packaged dags
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #8716:
URL: https://github.com/apache/airflow/issues/8716
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #8716: Support on packaged dags
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #8716:
URL: https://github.com/apache/airflow/issues/8716#issuecomment-623982065
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mfumagalli68 commented on issue #8716: Support on packaged dags
Posted by GitBox <gi...@apache.org>.
mfumagalli68 commented on issue #8716:
URL: https://github.com/apache/airflow/issues/8716#issuecomment-624008547
Ok so what's the best practice for dealing dependency problems?
What I've seen so far:
- PythonVirtualEnvOperator. Problem: create and destory virtualenv on the fly. Not feasible for a project with lots of dependencies.
- Use BashOpeator: write a shell which execute a python script with virtualenv interpreter. Seems a workaround.
Morover I cannot pass complex kwargs to the operator. For example `params={'A':{'B':[1,2,3]}}` won't work.
- DockerOperator: Is this the only option we have?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org