You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/01 01:51:13 UTC
[GitHub] [airflow] m0bi5 opened a new issue #12728: Unable to parse string [failed
m0bi5 opened a new issue #12728:
URL: https://github.com/apache/airflow/issues/12728
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
<!--
IMPORTANT!!!
PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.
-->
**Apache Airflow version**: 2.0
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): 1.15
**Environment**: Kubernetes, using the KubernetesExecutor as well
- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release): Debian GNU/Linux 10 (buster)
- **Kernel** (e.g. `uname -a`): Linux airflow-7f86fbfdc-6s4l6 3.10.0-957.10.1.el7.x86_64
- **Others**: I am using the official Airflow Docker Image
**What happened**:
I was trying to execute a DAG but for some reason it works at certain times but there are times it throws an error. Below I have attached two execution logs (I am executing the DAG twice, one after the other), the first execution throws a `pendulum.parsing.exceptions.ParserError: Unable to parse string [failed]` error
<!-- (please include exact error messages if you can) -->
**What you expected to happen**:
<!-- What do you think went wrong? -->
**How to reproduce it**:
<!---
As minimally and precisely as possible. Keep in mind we do not have access to your cluster or dags.
If you are using kubernetes, please attempt to recreate the issue using minikube or kind.
## Install minikube/kind
- Minikube https://minikube.sigs.k8s.io/docs/start/
- Kind https://kind.sigs.k8s.io/docs/user/quick-start/
If this is a UI bug, please provide a screenshot of the bug or a link to a youtube video of the bug in action
You can include images using the .md style of
To record a screencast, mac users can use QuickTime and then create an unlisted youtube video with the resulting .mov file.
--->
**Anything else we need to know**:
<!--
How often does this problem occur? Once? Every time etc?
Any relevant logs to include? Put them here in side a detail tag:
<details><summary>x.log</summary> lots of stuff </details>
-->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bmfisher commented on issue #12728: DAG execution works some times but throws Unable to parse string [failed] error at other times
Posted by GitBox <gi...@apache.org>.
bmfisher commented on issue #12728:
URL: https://github.com/apache/airflow/issues/12728#issuecomment-776825605
@dimberman @m0bi5
I'm seeing the same error. I strongly suspect its due to this line https://github.com/apache/airflow/blob/master/airflow/kubernetes/pod_launcher.py#L168
Reading the log line splits on `' '` and the timestamps in my logs as well as yours are similar to `[2020-12-01 01:42:53,699]` containing a space. I've seen other k8s logs with the begging of the line looking like `2021-02-09T22:49:58.852Z log`. For now I'll be digging into k8s options looking for a way to change my timestamp format, but that could be a relatively simple PR. I'd also be willing to help out implementing a fix.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #12728: DAG execution works some times but throws Unable to parse string [failed] error at other times
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #12728:
URL: https://github.com/apache/airflow/issues/12728#issuecomment-774235142
@m0bi5 I'd say for the moment let's just hack it in and run the DAGs so we can see what the response from k8s looks like when this fails, once that's set we can get you set up with a dev environment using this guide https://github.com/apache/airflow/blob/master/CONTRIBUTORS_QUICK_START.rst
Once we track down this bug would love to help you get a commit in :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bmfisher commented on issue #12728: DAG execution works some times but throws Unable to parse string [failed] error at other times
Posted by GitBox <gi...@apache.org>.
bmfisher commented on issue #12728:
URL: https://github.com/apache/airflow/issues/12728#issuecomment-776924351
@dimberman I would be interested in helping out. Unfortunately I am much less confident that I found the actual bug than I was a couple hours ago. I did a little local testing with pendulum and realized that the string getting passed is definitely `'failed'` and not a partial timestamp like `'[2020-12-01'`. Along with seeing that `timestamps=True` is set properly, I think the issue must be a little bit farther upstream and just happens to bubble up here. There might be a race condition somewhere given the inconsistent nature around the bugs appearance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] m0bi5 edited a comment on issue #12728: DAG execution works some times but throws Unable to parse string [failed] error at other times
Posted by GitBox <gi...@apache.org>.
m0bi5 edited a comment on issue #12728:
URL: https://github.com/apache/airflow/issues/12728#issuecomment-774146639
@dimberman And then open a pull request for it? I would definitely need some guidance but I can definitely contribute
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] m0bi5 commented on issue #12728: DAG execution works some times but throws Unable to parse string [failed] error at other times
Posted by GitBox <gi...@apache.org>.
m0bi5 commented on issue #12728:
URL: https://github.com/apache/airflow/issues/12728#issuecomment-773591261
It happens quite randomly but it does happen quite often. For a quick fix, I bypassed the parsing of the string in `pod_launcher.py`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12728: KubePodOperator sometimes fails with `Unable to parse string [failed]`
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12728:
URL: https://github.com/apache/airflow/issues/12728#issuecomment-860646109
Today!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #12728: DAG execution works some times but throws Unable to parse string [failed] error at other times
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #12728:
URL: https://github.com/apache/airflow/issues/12728#issuecomment-776908881
@bmfisher would you be interesting making a PR for this? Would be glad to help you get set up/PR this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #12728: KubePodOperator sometimes fails with `Unable to parse string [failed]`
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #12728:
URL: https://github.com/apache/airflow/issues/12728#issuecomment-827587630
@bmfisher If you happen to still have those pod logs in your ES, could you go and search through them and find where the log line starts with "failed" or "Unable"? i.e. I'd love to be able to see the line the errors and a few lines (10?) either side of it.
Same to anyone who has this problem: if you are able to get the "raw" pod logs via `kubectl ... get logs <podname> --timestamps` this would be very helpful in debugging this issue. You probably need to set `is_delete_operator_pod=False` in the DAG to not have the pod be automatically failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #12728: DAG execution works some times but throws Unable to parse string [failed] error at other times
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #12728:
URL: https://github.com/apache/airflow/issues/12728#issuecomment-773588575
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb closed issue #12728: KubePodOperator sometimes fails with `Unable to parse string [failed]`
Posted by GitBox <gi...@apache.org>.
ashb closed issue #12728:
URL: https://github.com/apache/airflow/issues/12728
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bmfisher edited a comment on issue #12728: DAG execution works some times but throws Unable to parse string [failed] error at other times
Posted by GitBox <gi...@apache.org>.
bmfisher edited a comment on issue #12728:
URL: https://github.com/apache/airflow/issues/12728#issuecomment-776825605
@dimberman @m0bi5
I'm seeing the same error. I strongly suspect its due to this line https://github.com/apache/airflow/blob/master/airflow/kubernetes/pod_launcher.py#L168
Reading the log line splits on `' '` and the timestamps in my logs as well as yours are similar to `[2020-12-01 01:42:53,699]` containing a space. I've seen other k8s logs with the begging of the line looking like `2021-02-09T22:49:58.852Z log`. For now I'll be digging into k8s options looking for a way to change my timestamp format, but that could be a relatively simple PR. I'd also be willing to help out implementing a fix.
Edit: now seeing that https://github.com/apache/airflow/blob/master/airflow/kubernetes/pod_launcher.py#L132 asks to read logs with `timestamps=True` which should put the timestamps in the right format for splitting on a space.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #12728: DAG execution works some times but throws Unable to parse string [failed] error at other times
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #12728:
URL: https://github.com/apache/airflow/issues/12728#issuecomment-773650592
@m0bi5 is there any chance you can modify the pod_laucher to instead wrap this line in a try/catch and print the whole line if it catches?
```last_log_time = pendulum.parse(timestamp)```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] timgriffiths commented on issue #12728: KubePodOperator sometimes fails with `Unable to parse string [failed]`
Posted by GitBox <gi...@apache.org>.
timgriffiths commented on issue #12728:
URL: https://github.com/apache/airflow/issues/12728#issuecomment-854365350
@ashb @dimberman any idea when this will be merged, we seem to be hitting this with about 25% of our jobs it's rather interesting some times we hit it when the job finishes and other times it's on creation of the job, it's giving us a lot of false negatives of problems.
It's almost like if it tries to do a get log after k8s has terminated the container (ie between when you checked it's still there then you going to get the logs) or in that window when it's created but the api not ready to give you the logs yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] timgriffiths commented on issue #12728: KubePodOperator sometimes fails with `Unable to parse string [failed]`
Posted by GitBox <gi...@apache.org>.
timgriffiths commented on issue #12728:
URL: https://github.com/apache/airflow/issues/12728#issuecomment-854365350
@ashb @dimberman any idea when this will be merged, we seem to be hitting this with about 25% of our jobs it's rather interesting some times we hit it when the job finishes and other times it's on creation of the job, it's giving us a lot of false negatives of problems.
It's almost like if it tries to do a get log after k8s has terminated the container (ie between when you checked it's still there then you going to get the logs) or in that window when it's created but the api not ready to give you the logs yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bmfisher commented on issue #12728: KubePodOperator sometimes fails with `Unable to parse string [failed]`
Posted by GitBox <gi...@apache.org>.
bmfisher commented on issue #12728:
URL: https://github.com/apache/airflow/issues/12728#issuecomment-827703195
@ashb I was able to find a section of logs I kept in my notes. Its been a while since I was looking at this problem so I'm not too fresh on it and I don't recall exactly which pod this set of logs came from. But I recall thinking there was a race condition and my workaround with the `get_logs` flag set to false has continued to work for us.
```
[2021-02-09 21:53:11,356] {pod_launcher.py:176} INFO - Event: my-task-id-bd83adbc07b044b0a01058e44a3662eb had an event of type Pending
[2021-02-09 21:53:11,356] {pod_launcher.py:113} WARNING - Pod not yet started: my-task-id-bd83adbc07b044b0a01058e44a3662eb
[2021-02-09 21:53:12,367] {pod_launcher.py:176} INFO - Event: my-task-id-bd83adbc07b044b0a01058e44a3662eb had an event of type Pending
[2021-02-09 21:53:12,367] {pod_launcher.py:113} WARNING - Pod not yet started: my-task-id-bd83adbc07b044b0a01058e44a3662eb
[2021-02-09 21:53:13,376] {pod_launcher.py:176} INFO - Event: my-task-id-bd83adbc07b044b0a01058e44a3662eb had an event of type Succeeded
[2021-02-09 21:53:13,376] {pod_launcher.py:289} INFO - Event with job id my-task-id-bd83adbc07b044b0a01058e44a3662eb Succeeded
[2021-02-09 21:53:13,394] {pod_launcher.py:136} INFO - {"log_id": "my_dag_id-my_task_id-2021-02-09T21:49:01.955021+00:00-2", "asctime": "2021-02-09 21:53:11,796", "filename": "connection_workflow.py", "lineno": 179, "levelname": "INFO", "message": "Pika version 1.1.0 connecting to ('10.103.243.78', 5672)", "offset": 1612907591796887296}
[2021-02-09 21:53:13,394] {pod_launcher.py:136} INFO - {"log_id": "my_dag_id-my_task_id-2021-02-09T21:49:01.955021+00:00-2", "asctime": "2021-02-09 21:53:11,797", "filename": "io_services_utils.py", "lineno": 345, "levelname": "INFO", "message": "Socket connected: <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.244.1.146', 50972), raddr=('10.103.243.78', 5672)>", "offset": 1612907591797981696}
[2021-02-09 21:53:13,394] {pod_launcher.py:136} INFO - {"log_id": "my_dag_id-my_task_id-2021-02-09T21:49:01.955021+00:00-2", "asctime": "2021-02-09 21:53:11,800", "filename": "connection_workflow.py", "lineno": 428, "levelname": "INFO", "message": "Streaming transport linked up: (<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fedd02bf8e0>, _StreamingProtocolShim: <SelectConnection PROTOCOL transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fedd02bf8e0> params=<ConnectionParameters host=net-service port=5672 virtual_host=/ ssl=False>>).", "offset": 1612907591800135424}
[2021-02-09 21:53:13,395] {pod_launcher.py:136} INFO - {"log_id": "my_dag_id-my_task_id-2021-02-09T21:49:01.955021+00:00-2", "asctime": "2021-02-09 21:53:11,806", "filename": "connection_workflow.py", "lineno": 293, "levelname": "INFO", "message": "AMQPConnector - reporting success: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fedd02bf8e0> params=<ConnectionParameters host=net-service port=5672 virtual_host=/ ssl=False>>", "offset": 1612907591806535680}
[2021-02-09 21:53:13,395] {pod_launcher.py:136} INFO - {"log_id": "my_dag_id-my_task_id-2021-02-09T21:49:01.955021+00:00-2", "asctime": "2021-02-09 21:53:11,806", "filename": "connection_workflow.py", "lineno": 725, "levelname": "INFO", "message": "AMQPConnectionWorkflow - reporting success: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fedd02bf8e0> params=<ConnectionParameters host=net-service port=5672 virtual_host=/ ssl=False>>", "offset": 1612907591806787584}
[2021-02-09 21:53:13,395] {pod_launcher.py:136} INFO - {"log_id": "my_dag_id-my_task_id-2021-02-09T21:49:01.955021+00:00-2", "asctime": "2021-02-09 21:53:11,806", "filename": "blocking_connection.py", "lineno": 452, "levelname": "INFO", "message": "Connection workflow succeeded: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fedd02bf8e0> params=<ConnectionParameters host=net-service port=5672 virtual_host=/ ssl=False>>", "offset": 1612907591806982400}
[2021-02-09 21:53:13,395] {pod_launcher.py:136} INFO - {"log_id": "my_dag_id-my_task_id-2021-02-09T21:49:01.955021+00:00-2", "asctime": "2021-02-09 21:53:11,844", "filename": "pysolr.py", "lineno": 438, "levelname": "INFO", "message": "Finished 'http://solr-service:8983/solr/mycollection/select/?q=birth%3A19550101&rows=100&cursorMark=%2A&sort=__lastUpdatedOn+asc%2C+key+asc&wt=json' (get) with body '' in 0.038 seconds, with status 200", "offset": 1612907591844953600}
[2021-02-09 21:53:13,395] {pod_launcher.py:136} INFO - {"log_id": "my_dag_id-my_task_id-2021-02-09T21:49:01.955021+00:00-2", "asctime": "2021-02-09 21:53:11,848", "filename": "blocking_connection.py", "lineno": 1247, "levelname": "INFO", "message": "Created channel=1", "offset": 1612907591848534528}
[2021-02-09 21:53:13,396] {pod_launcher.py:136} INFO - {"log_id": "my_dag_id-my_task_id-2021-02-09T21:49:01.955021+00:00-2", "asctime": "2021-02-09 21:53:11,875", "filename": "pysolr.py", "lineno": 438, "levelname": "INFO", "message": "Finished 'http://solr-service:8983/solr/mycollection/select/?q=birth%3A19550101&rows=100&cursorMark=AoIHCNizRCDtFiw1YWVlNjhhMzEhYmM0MDk1ZGNAcmFk&sort=__lastUpdatedOn+asc%2C+key+asc&wt=json' (get) with body '' in 0.020 seconds, with status 200", "offset": 1612907591875407616}
[2021-02-09 21:53:13,396] {pod_launcher.py:136} INFO - {"log_id": "my_dag_id-my_task_id-2021-02-09T21:49:01.955021+00:00-2", "asctime": "2021-02-09 21:53:11,875", "filename": "channel.py", "lineno": 533, "levelname": "INFO", "message": "Closing channel (0): 'Normal shutdown' on <Channel number=1 OPEN conn=<SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fedd02bf8e0> params=<ConnectionParameters host=net-service port=5672 virtual_host=/ ssl=False>>>", "offset": 1612907591875733504}
[2021-02-09 21:53:13,396] {pod_launcher.py:136} INFO - {"log_id": "my_dag_id-my_task_id-2021-02-09T21:49:01.955021+00:00-2", "asctime": "2021-02-09 21:53:11,877", "filename": "channel.py", "lineno": 1119, "levelname": "INFO", "message": "Received <Channel.CloseOk> on <Channel number=1 CLOSING conn=<SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fedd02bf8e0> params=<ConnectionParameters host=net-service port=5672 virtual_host=/ ssl=False>>>", "offset": 1612907591877563392}
[2021-02-09 21:53:13,414] {taskinstance.py:1396} ERROR - Unable to parse string [failed]
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1086, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1260, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1300, in _execute_task
result = task_copy.execute(context=context)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 340, in execute
final_state, _, result = self.create_new_pod_for_operator(labels, launcher)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 486, in create_new_pod_for_operator
final_state, result = launcher.monitor_pod(pod=self.pod, get_logs=self.get_logs)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/kubernetes/pod_launcher.py", line 135, in monitor_pod
last_log_time = pendulum.parse(timestamp)
File "/home/airflow/.local/lib/python3.6/site-packages/pendulum/parser.py", line 29, in parse
return _parse(text, **options)
File "/home/airflow/.local/lib/python3.6/site-packages/pendulum/parser.py", line 45, in _parse
parsed = base_parse(text, **options)
File "/home/airflow/.local/lib/python3.6/site-packages/pendulum/parsing/__init__.py", line 74, in parse
return _normalize(_parse(text, **_options), **_options)
File "/home/airflow/.local/lib/python3.6/site-packages/pendulum/parsing/__init__.py", line 128, in _parse
raise ParserError("Unable to parse string [{}]".format(text))
pendulum.parsing.exceptions.ParserError: Unable to parse string [failed]
[2021-02-09 21:53:13,416] {taskinstance.py:1440} INFO - Marking task as UP_FOR_RETRY. dag_id=my_dag_id, task_id=my_task_id, execution_date=20210209T214901, start_date=20210209T215309, end_date=20210209T215313
[2021-02-09 21:53:13,482] {local_task_job.py:118} INFO - Task exited with return code 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] m0bi5 commented on issue #12728: DAG execution works some times but throws Unable to parse string [failed] error at other times
Posted by GitBox <gi...@apache.org>.
m0bi5 commented on issue #12728:
URL: https://github.com/apache/airflow/issues/12728#issuecomment-776876347
@bmfisher Please take this issue up. I am currently busy with a lot of other projects and am unable to work on this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] m0bi5 commented on issue #12728: DAG execution works some times but throws Unable to parse string [failed] error at other times
Posted by GitBox <gi...@apache.org>.
m0bi5 commented on issue #12728:
URL: https://github.com/apache/airflow/issues/12728#issuecomment-773591261
It happens quite randomly but it does happen quite often. For a quick fix, I bypassed the parsing of the string in `pod_launcher.py`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #12728: KubePodOperator sometimes fails with `Unable to parse string [failed]`
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #12728:
URL: https://github.com/apache/airflow/issues/12728#issuecomment-831420589
> @ashb it appears that the issue might be when we have log lines that come from errors (e.g. in this case the word was "failed" and in another case the word was "Unable"), this breaks the timestamp parsing because hte first word is not a timestamp.
That shouldn't be possible - the time stamp is added by Kube itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #12728: KubePodOperator sometimes fails with `Unable to parse string [failed]`
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #12728:
URL: https://github.com/apache/airflow/issues/12728#issuecomment-831330496
@bmfisher thanks for those logs - unfortunately I need the pod logs to look at what the problematic line is, the airflow task logs just tell us where in the Airflow code the error is occurring, but I need to see the input line that is causing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #12728: KubePodOperator sometimes fails with `Unable to parse string [failed]`
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #12728:
URL: https://github.com/apache/airflow/issues/12728#issuecomment-860645013
@timgriffiths Just merged now, so this should be included in the June providers release which @potiuk Is going to start preparing soon (this week I think?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #12728: KubePodOperator sometimes fails with `Unable to parse string [failed]`
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #12728:
URL: https://github.com/apache/airflow/issues/12728#issuecomment-831414223
https://github.com/apache/airflow/pull/15638 I believe this should solve the issue. Will add unit tests soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] m0bi5 commented on issue #12728: DAG execution works some times but throws Unable to parse string [failed] error at other times
Posted by GitBox <gi...@apache.org>.
m0bi5 commented on issue #12728:
URL: https://github.com/apache/airflow/issues/12728#issuecomment-774146639
And then open a pull request for it? I would definitely need some guidance but I can definitely contribute
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] m0bi5 edited a comment on issue #12728: DAG execution works some times but throws Unable to parse string [failed] error at other times
Posted by GitBox <gi...@apache.org>.
m0bi5 edited a comment on issue #12728:
URL: https://github.com/apache/airflow/issues/12728#issuecomment-774146639
@dimberman And then open a pull request for it? I would need some guidance but I can definitely contribute
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #12728: DAG execution works some times but throws Unable to parse string [failed] error at other times
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #12728:
URL: https://github.com/apache/airflow/issues/12728#issuecomment-773588575
Hi @m0bi5
Does this issue happen consistently?
Reading the log it seems like this might be caused by k8s sending a message that doesn't cleanly parse into a timestamp. I'm wondering if this is us not properly handling a k8s error.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #12728: KubePodOperator sometimes fails with `Unable to parse string [failed]`
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #12728:
URL: https://github.com/apache/airflow/issues/12728#issuecomment-831396814
@ashb it appears that the issue might be when we have log lines that come from errors (e.g. in this case the word was "failed" and in another case the word was "Unable"), this breaks the timestamp parsing because hte first word is not a timestamp.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bmfisher commented on issue #12728: DAG execution works some times but throws Unable to parse string [failed] error at other times
Posted by GitBox <gi...@apache.org>.
bmfisher commented on issue #12728:
URL: https://github.com/apache/airflow/issues/12728#issuecomment-777774826
@m0bi5 I have narrowed the race condition to within the `if get_logs:` code path in `monitor_pod`. https://github.com/apache/airflow/blob/master/airflow/kubernetes/pod_launcher.py#L128
I was able to set `get_logs=False` in my `KubernetesPodOperator` and run without experiencing these issues. Luckily, I have a remote logging setup with Filebeat and Elasticsearch so I can get by without the `get_logs` flag.
@dimberman I'm still not 100% sure what is causing the issue, but I can try to get a development environment setup and test some possible fixes. I'll start with the quick start linked above and let you know if I have any issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org