[jira] [Closed] (AIRFLOW-2808) Plugin duplication checking is not working

["Xiaodong DENG (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xiaodong DENG closed AIRFLOW-2808.
----------------------------------
    Resolution: Invalid

Can't pass Kubernetes tests properly (even though seems not relating to this commit). Will need to check

> Plugin duplication checking is not working
> ------------------------------------------
>
>                 Key: AIRFLOW-2808
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2808
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: plugins
>            Reporter: Xiaodong DENG
>            Assignee: Xiaodong DENG
>            Priority: Major
>
> h2. *Background*
> A plugin duplication checking was designed in *plugins_manager.py* [https://github.com/apache/incubator-airflow/blob/master/airflow/plugins_manager.py#L93]  .
> Corresponding commit was [https://github.com/apache/incubator-airflow/commit/3f38dec9bf44717a275412d1fe155e8252e45ee5|https://github.com/apache/incubator-airflow/commit/3f38dec9bf44717a275412d1fe155e8252e45ee5.]    
> However, it turns out that this checking is not really working (reason: plugin method object name is formed using plugin file path + plugin file name + Plugin Class name. It will never be duplicated given there will not be two files with the same name in the same directory).
> h2. *Issue*
> In my production environment, there are two plugin files with the same name and operator names in the new _AirflowPlugin_ classes defined inside. However, they passed the check without any warning or exception.
> For example, I have a plugin *file_sensor_1.py* as below, 
> {code:java}
> from airflow.plugins_manager import AirflowPlugin
> from airflow.operators.sensors import BaseSensorOperator
> from airflow.utils.decorators import apply_defaults
> import os
> class local_file_sensor(BaseSensorOperator):
>     @apply_defaults
>     def __init__(self, file_path, *args, **kwargs):
>         super(local_file_sensor, self).__init__(*args, **kwargs)
>         self.file_path = file_path
>     def poke(self, context):
>         self.log.info('A-Poking: %s', self.file_path)
>         return os.path.exists(self.file_path)
> class AirflowLocalFileSensorPlugin(AirflowPlugin):
>     name = "local_file_sensor_plugin"
>     operators = [local_file_sensor]
> {code}
>  
> I copy & paste it into another plugin file *file_sensor_2.py*, and make the only change to change the log info from "_A-Poking_" to "_B-Poking_" (to help me check which one is picked).
> Only one plugin would be loaded eventually (because the earlier loaded one will be overwritten by the later loaded one [https://github.com/apache/incubator-airflow/blob/master/airflow/plugins_manager.py#L101] ). However, which one? We don't know. It's indeterminate. So far the file name seems to be the only factor affecting which one would be picked by Airflow.
> h2. *My proposal*
> Give WARNING to the users when they launch the Airflow. (Or should we give error msg and fail the launching?) 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)