You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/05/12 14:52:07 UTC

[GitHub] [airflow] FabianF92 opened a new issue #15803: Task Instance details can't be displayed when using special attribute json

FabianF92 opened a new issue #15803:
URL: https://github.com/apache/airflow/issues/15803


   
   **Apache Airflow version**: 2.0.1
   
   **Environment**: Azure App Service
   
   **What happened**:
   
   In UI, when viewing Instance Details of a task (Dag tree view -> task -> Instance Details) that contains special attribute `json` exception is thrown. This happens when the json attribute contains a dict with non-string values.
   ![image](https://user-images.githubusercontent.com/30059482/117994237-ae63ff80-b340-11eb-9449-64e4b3792abe.png)
   
   **What you expected to happen**:
   Task instance details are rendered including the special attribute json.
   
   Problem is caused by function `render(obj, lexer)` defined in `utils.py` (line 317, version 2.0.1):
   ```
       elif isinstance(obj, dict):
           for k, v in obj.items():
               out += Markup('<div>Dict item "{}"</div>').format(k)  # noqa
               out += Markup("<div>" + pygment_html_render(v, lexer) + "</div>")
       return out
   ```
   When value of one item is not a string but for example a dict itself, method `pygment_html_render(v, lexer)` can't decode data and throws error `AttributeError: 'dict' object has no attribute 'decode'`.
   
   **How to reproduce it**:
   Define a task with a dict as json attribute. The dict should contain at least one value that is another dict.
   For example, the `DatabricksSubmitRunOperator` takes `json` attribute to pass notebook parameters:
   ```
   json = {
             'new_cluster': {
               'spark_version': '2.1.0-db3-scala2.11',
               'num_workers': 2
             },
             'notebook_task': {
               'notebook_path': '/Users/airflow@example.com/PrepareData',
             },
           }
           notebook_run = DatabricksSubmitRunOperator(task_id='notebook_run', json=json)
   ```
   
   **Anything else we need to know**:
   
   Might be an idea to convert complete dict to json string and pass it to `pygments` JsonLexer instead of highlighting all items of the json.
   We wouldn't need all items displayed separately in UI as shown here:
   ![image](https://user-images.githubusercontent.com/30059482/117995232-74dfc400-b341-11eb-9750-2a26717bab78.png)
   
   
   **How often does this problem occur?**
   Every time we use json task attribute that contains non-string item values.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] griseau commented on issue #15803: Task Instance details can't be displayed when using special attribute json

Posted by GitBox <gi...@apache.org>.

griseau commented on issue #15803:
URL: https://github.com/apache/airflow/issues/15803#issuecomment-853892986


   Hey @Junnplus, 
   I'm using this image : 2.0.2-python3.8 and I also have the same problem...
   
   Thomas


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #15803: Task Instance details can't be displayed when using special attribute json

Posted by GitBox <gi...@apache.org>.

boring-cyborg[bot] commented on issue #15803:
URL: https://github.com/apache/airflow/issues/15803#issuecomment-839837154


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] griseau commented on issue #15803: Task Instance details can't be displayed when using special attribute json

Posted by GitBox <gi...@apache.org>.

griseau commented on issue #15803:
URL: https://github.com/apache/airflow/issues/15803#issuecomment-853892986


   Hey @Junnplus, 
   I'm using this image : 2.0.2-python3.8 and I also have the same problem...
   
   Thomas


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] FabianF92 commented on issue #15803: Task Instance details can't be displayed when using special attribute json

Posted by GitBox <gi...@apache.org>.

FabianF92 commented on issue #15803:
URL: https://github.com/apache/airflow/issues/15803#issuecomment-854863842

Hi @griseau, I didn't test with version 2.0.2 so far, but I already checked the code which causes this error and it is the same as in 2.0.1.
We accepted this bug for the moment in our environment.
A workaround is trying to get rid of the special attribute (in my example json attribute but can also be one of these https://github.com/apache/airflow/blob/71de801c18f8e8c302f08113b85f4ff1e98d4afe/airflow/www/utils.py#L354 ) in your operator. The Databricks Operator would also allow to set each element of the json directly inside the task definition. Then, you would not go into special attribute case for rendering.
As this workaround would make our DAGS longer, we decided to wait for this to be fixed.

My suggestion for the code fix would be to convert the dict to a json string and render the whole string instead of each json item, cause if you have many items this doesn't look really good (see my last screen shot in the post, I passed a json string just to see how GUI will display it - but then my task didn't run). Something like this should work:

```
elif isinstance(obj, dict):
out = Markup(pygment_html_render(json.dumps(obj), lexer))
return out
```
So, just 2 lines to change. We will move to production next week, I may open a PR here, then it might be included in on of the next releases!

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] griseau edited a comment on issue #15803: Task Instance details can't be displayed when using special attribute json

Posted by GitBox <gi...@apache.org>.

griseau edited a comment on issue #15803:
URL: https://github.com/apache/airflow/issues/15803#issuecomment-853892986


   Hey @Junnplus, 
   I'm using this image : 2.0.2-python3.8 and I also have the same problem...
   
   Thanks
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] griseau commented on issue #15803: Task Instance details can't be displayed when using special attribute json

Posted by GitBox <gi...@apache.org>.

griseau commented on issue #15803:
URL: https://github.com/apache/airflow/issues/15803#issuecomment-854822195


   @FabianF92 did you find any solution ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] FabianF92 commented on issue #15803: Task Instance details can't be displayed when using special attribute json

Posted by GitBox <gi...@apache.org>.

FabianF92 commented on issue #15803:
URL: https://github.com/apache/airflow/issues/15803#issuecomment-855976402


   @griseau I checked again: 
   This is fixed in PR https://github.com/apache/airflow/pull/14024 for json attribute.
   It is included in versions >= 2.1.0 as I can see, I testet with apache/airflow:2.1.0 and it works.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] griseau commented on issue #15803: Task Instance details can't be displayed when using special attribute json

Posted by GitBox <gi...@apache.org>.

griseau commented on issue #15803:
URL: https://github.com/apache/airflow/issues/15803#issuecomment-856011406


   @FabianF92 crystal clear ! 
   
   I'll test that everything is working fine with 2.1.x and think about migrating.
   Thanks for your feedback 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #15803: Task Instance details can't be displayed when using special attribute json

Posted by GitBox <gi...@apache.org>.

eladkal commented on issue #15803:
URL: https://github.com/apache/airflow/issues/15803#issuecomment-998784881


   fixed in #14024


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal closed issue #15803: Task Instance details can't be displayed when using special attribute json

Posted by GitBox <gi...@apache.org>.

eladkal closed issue #15803:
URL: https://github.com/apache/airflow/issues/15803


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] griseau edited a comment on issue #15803: Task Instance details can't be displayed when using special attribute json

Posted by GitBox <gi...@apache.org>.

griseau edited a comment on issue #15803:
URL: https://github.com/apache/airflow/issues/15803#issuecomment-853892986


   Hey @Junnplus, 
   I'm using this image : 2.0.2-python3.8 and I also have the same problem...
   
   Thanks
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] FabianF92 edited a comment on issue #15803: Task Instance details can't be displayed when using special attribute json

Posted by GitBox <gi...@apache.org>.

FabianF92 edited a comment on issue #15803:
URL: https://github.com/apache/airflow/issues/15803#issuecomment-854863842

```
elif isinstance(obj, dict):
out = Markup(pygment_html_render(json.dumps(obj), lexer))
return out
```
So, just 2 lines to change. We will move to production next week, so we would also like to get this working. I will open a PR the next days, then it might be included in one of the next releases!

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] Junnplus commented on issue #15803: Task Instance details can't be displayed when using special attribute json

Posted by GitBox <gi...@apache.org>.

Junnplus commented on issue #15803:
URL: https://github.com/apache/airflow/issues/15803#issuecomment-841218653


   It seems that 2.0.2 is working, Can you try it ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org