You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "vandonr-amz (via GitHub)" <gi...@apache.org> on 2023/04/04 19:10:10 UTC

[GitHub] [airflow] vandonr-amz opened a new pull request, #30466: add more info to quicksight errors

vandonr-amz opened a new pull request, #30466:
URL: https://github.com/apache/airflow/pull/30466

   in the current state, error handling was not very helpful. Adding the exception and error info available in the response will help with debugging.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] o-nikolas merged pull request #30466: add more info to quicksight errors

Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas merged PR #30466:
URL: https://github.com/apache/airflow/pull/30466


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] o-nikolas commented on a diff in pull request #30466: add more info to quicksight errors

Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #30466:
URL: https://github.com/apache/airflow/pull/30466#discussion_r1157698045


##########
airflow/providers/amazon/aws/hooks/quicksight.py:
##########
@@ -113,10 +113,17 @@ def get_status(self, aws_account_id: str, data_set_id: str, ingestion_id: str) -
                 AwsAccountId=aws_account_id, DataSetId=data_set_id, IngestionId=ingestion_id
             )
             return describe_ingestion_response["Ingestion"]["IngestionStatus"]
-        except KeyError:
-            raise AirflowException("Could not get status of the Amazon QuickSight Ingestion")
-        except ClientError:
-            raise AirflowException("AWS request failed, check logs for more info")
+        except KeyError as e:
+            raise AirflowException(f"Could not get status of the Amazon QuickSight Ingestion: {e}")
+        except ClientError as e:
+            raise AirflowException(f"AWS request failed: {e}")
+
+    def get_error_info(self, aws_account_id: str, data_set_id: str, ingestion_id: str) -> dict | None:
+        """If the ingestion failed, returns the error info. Else, returns None."""
+        describe_ingestion_response = self.get_conn().describe_ingestion(
+            AwsAccountId=aws_account_id, DataSetId=data_set_id, IngestionId=ingestion_id
+        )
+        return describe_ingestion_response["Ingestion"].get("ErrorInfo", None)

Review Comment:
   > Well at least it makes it obvious that it's something we expect can happen here.
   
   Fair point
   
   >  I can change it and add a comment to convey the info instead.
   
   Agreed, I like that much better!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] o-nikolas commented on a diff in pull request #30466: add more info to quicksight errors

Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #30466:
URL: https://github.com/apache/airflow/pull/30466#discussion_r1157677141


##########
airflow/providers/amazon/aws/hooks/quicksight.py:
##########
@@ -113,10 +113,17 @@ def get_status(self, aws_account_id: str, data_set_id: str, ingestion_id: str) -
                 AwsAccountId=aws_account_id, DataSetId=data_set_id, IngestionId=ingestion_id
             )
             return describe_ingestion_response["Ingestion"]["IngestionStatus"]
-        except KeyError:
-            raise AirflowException("Could not get status of the Amazon QuickSight Ingestion")
-        except ClientError:
-            raise AirflowException("AWS request failed, check logs for more info")
+        except KeyError as e:
+            raise AirflowException(f"Could not get status of the Amazon QuickSight Ingestion: {e}")
+        except ClientError as e:
+            raise AirflowException(f"AWS request failed: {e}")
+
+    def get_error_info(self, aws_account_id: str, data_set_id: str, ingestion_id: str) -> dict | None:
+        """If the ingestion failed, returns the error info. Else, returns None."""

Review Comment:
   Do you mind reformatting this with `param <param_name>:` and `return:` tags? The rest of the module follows that quite closely 



##########
airflow/providers/amazon/aws/hooks/quicksight.py:
##########
@@ -113,10 +113,17 @@ def get_status(self, aws_account_id: str, data_set_id: str, ingestion_id: str) -
                 AwsAccountId=aws_account_id, DataSetId=data_set_id, IngestionId=ingestion_id
             )
             return describe_ingestion_response["Ingestion"]["IngestionStatus"]
-        except KeyError:
-            raise AirflowException("Could not get status of the Amazon QuickSight Ingestion")
-        except ClientError:
-            raise AirflowException("AWS request failed, check logs for more info")
+        except KeyError as e:
+            raise AirflowException(f"Could not get status of the Amazon QuickSight Ingestion: {e}")
+        except ClientError as e:
+            raise AirflowException(f"AWS request failed: {e}")
+
+    def get_error_info(self, aws_account_id: str, data_set_id: str, ingestion_id: str) -> dict | None:
+        """If the ingestion failed, returns the error info. Else, returns None."""
+        describe_ingestion_response = self.get_conn().describe_ingestion(
+            AwsAccountId=aws_account_id, DataSetId=data_set_id, IngestionId=ingestion_id
+        )
+        return describe_ingestion_response["Ingestion"].get("ErrorInfo", None)

Review Comment:
   Nit: the default of `.get(...)` is already None.



##########
airflow/providers/amazon/aws/hooks/quicksight.py:
##########
@@ -141,12 +148,14 @@ def wait_for_state(
         status = self.get_status(aws_account_id, data_set_id, ingestion_id)
         while status in self.NON_TERMINAL_STATES and status != target_state:
             self.log.info("Current status is %s", status)
-            time.sleep(check_interval)
-            sec += check_interval
             if status in self.FAILED_STATES:
-                raise AirflowException("The Amazon QuickSight Ingestion failed!")
+                info = self.get_error_info(aws_account_id, data_set_id, ingestion_id)
+                raise AirflowException(f"The Amazon QuickSight Ingestion failed. Error info: {info}")
             if status == "CANCELLED":
                 raise AirflowException("The Amazon QuickSight SPICE ingestion cancelled!")
+            # wait and try again
+            time.sleep(check_interval)
+            sec += check_interval

Review Comment:
   :+1: 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] vandonr-amz commented on a diff in pull request #30466: add more info to quicksight errors

Posted by "vandonr-amz (via GitHub)" <gi...@apache.org>.
vandonr-amz commented on code in PR #30466:
URL: https://github.com/apache/airflow/pull/30466#discussion_r1157695133


##########
airflow/providers/amazon/aws/hooks/quicksight.py:
##########
@@ -113,10 +113,17 @@ def get_status(self, aws_account_id: str, data_set_id: str, ingestion_id: str) -
                 AwsAccountId=aws_account_id, DataSetId=data_set_id, IngestionId=ingestion_id
             )
             return describe_ingestion_response["Ingestion"]["IngestionStatus"]
-        except KeyError:
-            raise AirflowException("Could not get status of the Amazon QuickSight Ingestion")
-        except ClientError:
-            raise AirflowException("AWS request failed, check logs for more info")
+        except KeyError as e:
+            raise AirflowException(f"Could not get status of the Amazon QuickSight Ingestion: {e}")
+        except ClientError as e:
+            raise AirflowException(f"AWS request failed: {e}")
+
+    def get_error_info(self, aws_account_id: str, data_set_id: str, ingestion_id: str) -> dict | None:
+        """If the ingestion failed, returns the error info. Else, returns None."""
+        describe_ingestion_response = self.get_conn().describe_ingestion(
+            AwsAccountId=aws_account_id, DataSetId=data_set_id, IngestionId=ingestion_id
+        )
+        return describe_ingestion_response["Ingestion"].get("ErrorInfo", None)

Review Comment:
   ah ok. Well at least it makes it obvious that it's something we expect can happen here. I can change it and add a comment to convey the info instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] vandonr-amz commented on a diff in pull request #30466: add more info to quicksight errors

Posted by "vandonr-amz (via GitHub)" <gi...@apache.org>.
vandonr-amz commented on code in PR #30466:
URL: https://github.com/apache/airflow/pull/30466#discussion_r1157695648


##########
airflow/providers/amazon/aws/hooks/quicksight.py:
##########
@@ -141,12 +148,14 @@ def wait_for_state(
         status = self.get_status(aws_account_id, data_set_id, ingestion_id)
         while status in self.NON_TERMINAL_STATES and status != target_state:
             self.log.info("Current status is %s", status)
-            time.sleep(check_interval)
-            sec += check_interval
             if status in self.FAILED_STATES:
-                raise AirflowException("The Amazon QuickSight Ingestion failed!")
+                info = self.get_error_info(aws_account_id, data_set_id, ingestion_id)
+                raise AirflowException(f"The Amazon QuickSight Ingestion failed. Error info: {info}")
             if status == "CANCELLED":
                 raise AirflowException("The Amazon QuickSight SPICE ingestion cancelled!")
+            # wait and try again
+            time.sleep(check_interval)
+            sec += check_interval

Review Comment:
   ah yes I forgot to mention that in the description, I'll edit it !



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org