[GitHub] [airflow] aneesh-joseph commented on a change in pull request #9044: Ensure Kerberos token is valid in SparkSubmitOperator before running `yarn kill`

[GitBox <gi...@apache.org>]

aneesh-joseph commented on a change in pull request #9044:
URL: https://github.com/apache/airflow/pull/9044#discussion_r441739654



##########
File path: tests/providers/apache/spark/hooks/test_spark_submit.py
##########
@@ -679,7 +680,46 @@ def test_yarn_process_on_kill(self, mock_popen):
         # Then
         self.assertIn(call(['yarn', 'application', '-kill',
                             'application_1486558679801_1820'],
-                           stderr=-1, stdout=-1),
+                      env=None, stderr=-1, stdout=-1),
+                      mock_popen.mock_calls)
+
+    @patch('airflow.providers.apache.spark.hooks.spark_submit.airflow_conf')
+    @patch('airflow.providers.apache.spark.hooks.spark_submit.renew_from_kt')
+    @patch('airflow.providers.apache.spark.hooks.spark_submit.subprocess.Popen')
+    def test_yarn_process_on_kill_with_keytab(self, mock_popen, mock_krb, mock_conf):
+        # Given
+        mock_popen.return_value.stdout = io.StringIO('stdout')
+        mock_popen.return_value.stderr = io.StringIO('stderr')
+        mock_popen.return_value.poll.return_value = None
+        mock_popen.return_value.wait.return_value = 0
+        log_lines = [
+            'SPARK_MAJOR_VERSION is set to 2, using Spark2',
+            'WARN NativeCodeLoader: Unable to load native-hadoop library for your ' +
+            'platform... using builtin-java classes where applicable',
+            'WARN DomainSocketFactory: The short-circuit local reads feature cannot ' +
+            'be used because libhadoop cannot be loaded.',
+            'INFO Client: Requesting a new application from cluster with 10 ' +
+            'NodeManagerapplication_1486558679801_1820s',
+            'INFO Client: Submitting application application_1486558679801_1820 ' +
+            'to ResourceManager'
+        ]
+
+        krb5_conf_loc = '/tmp/krb5'
+        mock_conf.get.return_value = krb5_conf_loc

Review comment:
       @ashb - is there a better way to do this?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org