[GitHub] yangaws commented on a change in pull request #4516: [AIRFLOW-2524] Handle StopIteration in CloudWatch logs retrieval

[GitBox <gi...@apache.org>]

yangaws commented on a change in pull request #4516: [AIRFLOW-2524] Handle StopIteration in CloudWatch logs retrieval
URL: https://github.com/apache/airflow/pull/4516#discussion_r247632982
 
 

 ##########
 File path: airflow/contrib/hooks/sagemaker_hook.py
 ##########
 @@ -304,7 +304,15 @@ def multi_stream_iter(self, log_group, streams, positions=None):
         positions = positions or {s: Position(timestamp=0, skip=0) for s in streams}
         event_iters = [self.log_stream(log_group, s, positions[s].timestamp, positions[s].skip)
                        for s in streams]
-        events = [next(s) if s else None for s in event_iters]
+        events = []
+        for s in event_iters:
+            if not s:
+                events.append(None)
+                continue
+            try:
+                events.append(next(s))
+            except StopIteration:
+                events.append(None)
 
 Review comment:
   `
   >>> # tested in python 3.6.4
   ... 
   >>> def mkiter():
   ...     yield 1
   ... 
   >>> iter_list = [mkiter(), mkiter()]
   >>> value_list = [next(iter) for iter in iter_list]
   >>> value_list
   [1, 1]
   >>> value_list = [next(iter) for iter in iter_list]
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "<stdin>", line 1, in <listcomp>
   StopIteration
   `
   
   I think the StopIteration is thrown because I explicitly call next() here on an iterator which is at the end of iteration. So the for loop is iterating on a list of iterators instead of values in a single iterator.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services