[D] Is airflow the right tool to define and track workflows on file level [airflow]

["mahadi (via GitHub)" <gi...@apache.org>]

GitHub user mahadi created a discussion: Is airflow the right tool to define and track workflows on file level

I am searching for a framework (preferably in python) which supports handling the following workflow:

For a specific shared location (e.g. Sharepoint or S3) a user shall be able to create an instance of a "pipeline" - calling it P_i:

1. Data files are incoming on the shared location in irregular intervals, typically not faster than 1 minute. (But still it is possible that the user copies multiple files at once to the shared location)
2. Every data file needs to traverse a pipeline consisting of several steps like checking the validity of the file contents, performing some preliminary data analysis and applying meta data to the files.
3. The whole process is meant to be iterative: Adding new and eligible files shall traverse the pipeline and if the files are valid, they shall be added to a final entity.
4. Every data file shall have its own status which shows if the data file is eligible for further processing (airflow would offer a great UI for it so the user could easily see which files show issues).
5. In case a single file shows an issue the user needs to take manual actions (e.g. interactive classification). Is it possible to provide a link in case a part of the preprocessing fails, so the user can be forwarded to a page where the issue could be fixed?
6. An overall status of P_i is required: The user should see if the final entity is ok (which means that all single files are ok as well).
7. Finally, every user should be able to create multiple independent workflows: E.g. for every shared location a separate pipeline P_n should exist. Every shared location typically holds ~1000 files.

As far as i understand/imagine, in airflow a DAG is not meant to be applied on multiple single files. 
And if it is, i am wondering if it would cause potential performance issues.

So my question is: Is airflow the right framework to handle tasks like described above?

GitHub link: https://github.com/apache/airflow/discussions/47293

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org