[GitHub] [airflow] potiuk commented on a change in pull request #13371: Add clearer exception for read failures in macro plugin upgrade check

[GitBox <gi...@apache.org>]

potiuk commented on a change in pull request #13371:
URL: https://github.com/apache/airflow/pull/13371#discussion_r550270606



##########
File path: airflow/upgrade/rules/airflow_macro_plugin_removed.py
##########
@@ -39,9 +39,16 @@ def _check_file(self, file_path):
         problems = []
         class_name_to_check = self.MACRO_PLUGIN_CLASS.split(".")[-1]
         with open(file_path, "r") as file_pointer:
-            for line_number, line in enumerate(file_pointer, 1):
-                if class_name_to_check in line:
-                    problems.append(self._change_info(file_path, line_number))
+            try:
+                for line_number, line in enumerate(file_pointer, 1):
+                    if class_name_to_check in line:
+                        problems.append(self._change_info(file_path, line_number))
+            except UnicodeDecodeError:
+                raise Exception(
+                    "Unable to read file {}, it may need to be added to the .airflowignore".format(
+                        file_path

Review comment:
       I think it would also be great to add the original exception as a "chained" exception https://docs.python.org/3/tutorial/errors.html#exception-chaining this way we can see also the original stack trace.

##########
File path: airflow/upgrade/rules/airflow_macro_plugin_removed.py
##########
@@ -39,9 +39,16 @@ def _check_file(self, file_path):
         problems = []
         class_name_to_check = self.MACRO_PLUGIN_CLASS.split(".")[-1]
         with open(file_path, "r") as file_pointer:
-            for line_number, line in enumerate(file_pointer, 1):
-                if class_name_to_check in line:
-                    problems.append(self._change_info(file_path, line_number))
+            try:
+                for line_number, line in enumerate(file_pointer, 1):
+                    if class_name_to_check in line:
+                        problems.append(self._change_info(file_path, line_number))
+            except UnicodeDecodeError:
+                raise Exception(
+                    "Unable to read file {}, it may need to be added to the .airflowignore".format(

Review comment:
       I think it would be much better to fix the problem as well for non-default encodings.
   
   As of python 3.4 there is this function in the importlib:
   
   https://docs.python.org/3/library/importlib.html?highlight=importlib#importlib.util.decode_source
   
   And it will read the source code of python, respecting the encoding defined in the source code( the `# coding = ` comment). This way it will be more universal and handle the cases where non-utf8 encoding is used (which might be important in many countries). 
   
   I am not sure if similar approach is used in other rules - if so, the we can make a common approach and correct it in all those places. 
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org