[GitHub] vishaalkapoor commented on a change in pull request #12068: [MXAPPS-805] Notebook execution failures in CI.

[GitBox <gi...@apache.org>]

vishaalkapoor commented on a change in pull request #12068: [MXAPPS-805] Notebook execution failures in CI.
URL: https://github.com/apache/incubator-mxnet/pull/12068#discussion_r208764832
 
 

 ##########
 File path: tests/utils/notebook_test/__init__.py
 ##########
 @@ -72,15 +79,21 @@ def run_notebook(notebook, notebook_dir, kernel=None, no_cache=False, temp_dir='
         os.makedirs(working_dir)
     try:
         notebook = nbformat.read(notebook_path + '.ipynb', as_version=IPYTHON_VERSION)
-        # Adding a small delay to allow time for sockets to be freed
-        # stop-gap measure to battle the 1000ms linger of socket hard coded
-        # in the kernel API code
-        time.sleep(1.1)
         if kernel is not None:
             eprocessor = ExecutePreprocessor(timeout=TIME_OUT, kernel_name=kernel)
         else:
             eprocessor = ExecutePreprocessor(timeout=TIME_OUT)
-        nb, _ = eprocessor.preprocess(notebook, {'metadata': {'path': working_dir}})
+
+        # There is a low (< 1%) chance that starting a notebook executor will fail due to the kernel
+        # taking to long to start, or a port collision, etc.
+        for i in range(RETRIES):
+            try:
+                nb, _ = eprocessor.preprocess(notebook, {'metadata': {'path': working_dir}})
+            except (RuntimeError, TimeoutError) as rte:
+                logging.info("Error starting preprocessor: {}. Attempt {}/{}".format(str(rte), i+1, RETRIES))
+                time.sleep(1)
+                continue
 
 Review comment:
   I've added a commit which scans for the error message we would like to retry on that encompasses the issues we're seeing. We won't need to scan for TimeoutErrors as before, just RuntimeErrors with one specific error message. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services