[GitHub] abellina edited a comment on issue #23806: [SPARK-26895][CORE] prepareSubmitEnvironment should be called within doAs for proxy users

[GitBox <gi...@apache.org>]

abellina edited a comment on issue #23806: [SPARK-26895][CORE] prepareSubmitEnvironment should be called within doAs for proxy users
URL: https://github.com/apache/spark/pull/23806#issuecomment-465610619
 
 
   Hmm, python tests failed. Not sure if related though. The tests point at the `/home/jenkins/workspace/SparkPullRequestBuilder/python/unit-tests.log` file, but I can't get to that from jenkins UI it seems (are all prs writing to the same file?). Is this a known issue? Can we move it to the artifacts?
   
   From the error, it seems like the `shmutil.rmtree` function is trying to delete a directory, but there's likely another thread adding entries to a directory, so when it gets to `os.rmdir(path)` it blows up. I think the test should call `q.awaitTermination` after `q.stop`, before going on. I'll file a separate jira. 
   
   ```
   ERROR: test_query_manager_await_termination (pyspark.sql.tests.test_streaming.StreamingTests)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/sql/tests/test_streaming.py", line 259, in test_query_manager_await_termination
       shutil.rmtree(tmpPath)
     File "/home/anaconda/lib/python2.7/shutil.py", line 256, in rmtree
       onerror(os.rmdir, path, sys.exc_info())
     File "/home/anaconda/lib/python2.7/shutil.py", line 254, in rmtree
       os.rmdir(path)
   OSError: [Errno 39] Directory not empty: '/home/jenkins/workspace/SparkPullRequestBuilder/python/target/072153bd-f981-47be-bda2-e2b657a16f65/tmp4WGp7n'
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org