Re: [PR] Load Example Plugins with Example DAGs [airflow]

["zachliu (via GitHub)" <gi...@apache.org>]

zachliu commented on PR #39999:
URL: https://github.com/apache/airflow/pull/39999#issuecomment-2338985124

   Sorry for commenting on a merged PR. Just wanna make sure this is a real issue before i create an issue :sweat_smile: 
   
   The PR does make sure the plugin is loaded when `load_examples = True`. And the DAG itself runs ok. No loading errors or anything. The UI -> Admin -> Plugins shows the 3 plugins from the `example_dags/plugins`
   
   However, both the `sync-perm` cli and the `plugins` cli don't agree with it. When executing `airflow sync-perm --include-dags -v`, it always errors out on this example dag:
   ```
   {serialized_dag.py:200} DEBUG - Deserializing DAG: example_workday_timetable
   {plugins_manager.py:351} DEBUG - Plugins are already loaded. Skipping.
   {plugins_manager.py:484} DEBUG - Initialize extra timetables plugins
   {cli_action_loggers.py:98} DEBUG - Calling callbacks: []
   Traceback (most recent call last):
     File "/usr/local/airflow/.local/bin/airflow", line 8, in <module>
       sys.exit(main())
                ^^^^^^
     File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/__main__.py", line 62, in main
       args.func(args)
     File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/cli/cli_config.py", line 49, in command
       return func(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^
     File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/utils/cli.py", line 115, in wrapper
       return f(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^
     File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function
       return func(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^
     File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/fab/auth_manager/cli_commands/sync_perm_command.py", line 39, in sync_perm
       appbuilder.sm.create_dag_specific_permissions()
     File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1072, in create_dag_specific_permissions
       dagbag.collect_dags_from_db()
     File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/dagbag.py", line 628, in collect_dags_from_db
       self.dags = SerializedDagModel.read_all_dags()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/utils/session.py", line 97, in wrapper
       return func(*args, session=session, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/serialized_dag.py", line 201, in read_all_dags
       dag = row.dag
             ^^^^^^^
     File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/serialized_dag.py", line 235, in dag
       return SerializedDAG.from_dict(data)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/serialization/serialized_objects.py", line 1776, in from_dict
       return cls.deserialize_dag(serialized_obj["dag"])
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/serialization/serialized_objects.py", line 1704, in deserialize_dag
       v = decode_timetable(v)
           ^^^^^^^^^^^^^^^^^^^
     File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/serialization/serialized_objects.py", line 329, in decode_timetable
       raise _TimetableNotRegistered(importable_string)
   airflow.serialization.serialized_objects._TimetableNotRegistered: Timetable class 'airflow.example_dags.plugins.workday.AfterWorkdayTimetable' is not registered or you have a top level database access that disrupted the session. Please check the airflow best practices documentation.
   ```
   and when executing `airflow plugins -o table -v`, the 3 plugins from `example_dags/plugins` are not in the output table.
   
   of course if I set `load_examples = False`, the `sync-perm` and `plugin` work fine without any errors
   
   the only way that I can run both this example and the cli commands is:
   1. copy the source code `airflow/example_dags/plugins/workday.py` to my own `airflow/plugins` (in my `airflow.cfg`, i have `plugins_folder = /usr/local/airflow/plugins`)
   2. re-create the example as one of my own dags but import from my own plugins: `from workday import AfterWorkdayTimetable`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org