[GitHub] [tvm] guberti commented on issue #11117: [CI Problem] CI has logging errors even when successful

[GitBox <gi...@apache.org>]

guberti commented on issue #11117:
URL: https://github.com/apache/tvm/issues/11117#issuecomment-1136152256

   This issue is causing me quite a headache, as it is blocking me from merging #10921. I think I've figured it out though - it's caused by an interaction between `sphinx-gallery` and `absl` (imported by `tensorflow`). Here's what happens:
   
   1. `sphinx-gallery` wants to capture the output of code blocks as it runs them (this is one of its main features). To do this, [it uses a custom `LoggingTee` object](https://github.com/sphinx-gallery/sphinx-gallery/blob/master/sphinx_gallery/gen_rst.py#L60-L72) to replace `sys.stdout` and `sys.stderr`:
   
   ```python
   class _LoggingTee(object):
       """A tee object to redirect streams to the logger."""
      ...
   
       def set_std_and_reset_position(self):
           if not isinstance(sys.stdout, _LoggingTee):
               self.origs = (sys.stdout, sys.stderr)
           sys.stdout = sys.stderr = self
   ```
   
   2. `tensorflow` wants to have nice logging features, so it imports `logging` from [`absl`](https://abseil.io/). Hence, an `absl` `PythonHandler`, which is a subclass of Python's `StreamHandler`, is created.
   3. When a `StreamHandler` is created, it creates a pointer to `sys.stderr`. However, since `sphinx-gallery` replaced `sys.stderr` with `LoggingTee` earlier, a `LoggingTee` is instead inserted here:
   ```python
   class StreamHandler(Handler):
       def __init__(self, stream=None):
           """
           Initialize the handler.
   
           If stream is not specified, sys.stderr is used.
           """
           Handler.__init__(self)
           if stream is None:
               stream = sys.stderr
           self.stream = stream
   ```
   4. The error does not happen until `sphinx-gallery` finishes executing the code block. When this occurs, `sphinx-gallery` attempts to switch `stderr` and `stdout` back to their original values. However, it does not know that `StreamHandler` has a pointer to the `LoggingTee` object, and is unable to fix it. Hence, `stdout` and `stderr` go back to their original values, but `self.stream` in `StreamHandler` has the `LoggingTee`.
   ```python
       def restore_std(self):
           sys.stdout.flush()
           sys.stderr.flush()
           sys.stdout, sys.stderr = self.origs
   ```
   5. When it is time to close the log with `absl`, the `close` function is called:
   ```python
     def close(self):
       """Closes the stream to which we are writing."""
       self.acquire()
       try:
         self.flush()
         try:
           # Do not close the stream if it's sys.stderr|stdout. They may be
           # redirected or overridden to files, which should be managed by users
           # explicitly.
           user_managed = sys.stderr, sys.stdout, sys.__stderr__, sys.__stdout__
           if self.stream not in user_managed and (
               not hasattr(self.stream, 'isatty') or not self.stream.isatty()):
             self.stream.close()
         except ValueError:
           # A ValueError is thrown if we try to run isatty() on a closed file.
           pass
         super(PythonHandler, self).close()
       finally:
         self.release()
   ```
   We **should not call `close`** if we are writing to `stderr` or `stdout`, and `absl` knows this. Hence it checks whether `self.stream` is equal to either `sys.stderr` or `sys.stdout`. However, since `self.stream` is the `LoggingTee` object, it sees that it is not equal, and assumes the stream is a file that needs closing.
   6. `close()` is then called on `LoggingTee`, which does not have a `close()` method. This yields the `AttributeError` found above.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org