[jira] [Commented] (AIRFLOW-5270) JSONDecodeError in DockerOperator when force_pull

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16911722#comment-16911722 ] 

ASF GitHub Bot commented on AIRFLOW-5270:
-----------------------------------------

mattpetersen commented on pull request #5876: [AIRFLOW-5270] JSONDecodeError in DockerOperator when force_pull
URL: https://github.com/apache/airflow/pull/5876
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [X] My PR addresses the following [Airflow Jira]
   https://issues.apache.org/jira/browse/AIRFLOW-5270
   
   ### Description
   
   - [X] Here are some details about my PR, including screenshots of any UI changes
   The DockerAPI often returns multiple JSONs within the same line, joined by `\r\n`. If we replace `\r\n` with `\n`, then the response is decoded as expected; otherwise we get "JSONDecodeError"
   
   ### Tests
   
   - [X] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason
   
   Happy to add tests if necessary.
   
   ### Commits
   
   - [X] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines.
   
   ### Documentation
   
   - [X] In case of new functionality, my PR adds documentation that describes how to use it.
     - All the public functions and the classes in the PR contain docstrings that explain what it does
     - If you implement backwards incompatible changes, please leave a note in the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so we can assign it to a appropriate release
   
   ### Code Quality
   
   - [X] Passes `flake8`
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> JSONDecodeError in DockerOperator when force_pull
> -------------------------------------------------
>
>                 Key: AIRFLOW-5270
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5270
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: operators
>    Affects Versions: 1.10.3
>         Environment: MacOS Mojave
>            Reporter: Matthew Petersen
>            Assignee: Matthew Petersen
>            Priority: Major
>
> Lines of the Docker API response usually end with "\r\n". We .strip() them to correct for this, but sometimes two lines are joined together, creating one line that is not valid JSON. The exact error we see is
> ```
> json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 55)
> ```
> And can be reproduced with the following code
> ```
> pip install docker
> from docker import APIClient
> import json
> cli = APIClient(base_url='unix://var/run/docker.sock')
> for l in cli.pull('hello-world', stream=*True*):
>     output = json.loads(l.decode('utf-8').strip()
> ```
> which is exactly what this line of `DockerOperator` does which causes it to crash
> [https://github.com/apache/airflow/blob/46885ccdbc8618f295b8f986da95c1abbb85bb02/airflow/operators/docker_operator.py#L259]
> The solution is to replace "\r\n" with "\n" before json parsing



--
This message was sent by Atlassian Jira
(v8.3.2#803003)