You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/05/21 14:42:12 UTC
[GitHub] [airflow] egS27 opened a new issue #15990: Logging bug in a long runs
egS27 opened a new issue #15990:
URL: https://github.com/apache/airflow/issues/15990
**Apache Airflow version**:
2.0.2
**Environment**:
Kubernetes v1.18.3+cdb0358
Openshift 4.5.37_1536
**What happened**:
We are running our python code in kubernetes operators(airflow.contrib.operators.kubernetes_pod_operator).
During long runs(>10h) the airflow with the logs turned on(`get_logs=True` in k8s operator field) behaves absolutely normally, and then throws an unexpected error.
If we set `get_logs=False` - we have success dag run, otherwise, we have the same error every time.
Logs:
> [2021-05-18 13:54:10,199] {taskinstance.py:1482} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/response.py", line 696, in _update_chunk_length
self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/response.py", line 436, in _error_catcher
yield
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/response.py", line 763, in read_chunked
self._update_chunk_length()
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/response.py", line 700, in _update_chunk_length
raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(0 bytes read)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1138, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1341, in _execute_task
result = task_copy.execute(context=context)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 366, in execute
final_state, _, result = self.create_new_pod_for_operator(labels, launcher)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 513, in create_new_pod_for_operator
final_state, result = launcher.monitor_pod(pod=self.pod, get_logs=self.get_logs)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/utils/pod_launcher.py", line 145, in monitor_pod
for line in logs:
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/response.py", line 807, in __iter__
for chunk in self.stream(decode_content=True):
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/response.py", line 571, in stream
for line in self.read_chunked(amt, decode_content=decode_content):
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/response.py", line 792, in read_chunked
self._original_response.close()
File "/usr/local/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/response.py", line 454, in _error_catcher
raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
[2021-05-18 13:54:10,204] {taskinstance.py:1532} INFO - Marking task as FAILED. dag_id=pipline, task_id=task7, execution_date=20210518T132920, start_date=20210518T133244, end_date=20210518T135410
[2021-05-18 13:54:10,280] {local_task_job.py:146} INFO - Task exited with return code 1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15990: Logging bug in a long runs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15990:
URL: https://github.com/apache/airflow/issues/15990#issuecomment-1081675293
> @dstandish was the original issue resolved? In this case, please, provide corresponded note in the changelog. We are still using custom logging. In another case, please, open the issue again
@egS27 - as mentioned above the original report is for 1.10 which is EOL for 10 months already and there is no expectation on any support there, including finding specific commits that fix the problem from the community. However our code and issues are fully open, and you can search for relevant issus fixing the problem. Using EOL version means exactly this - you are on your own and you should expect no support from the community for issues there.
If you need support for - let me remind you the FREE software you have - from people who mostly support users like you in your free time, there are a number of companies that provide paid support for Airflow. You can easilly google them. If your expectation is some kind of SLA (and especially for an EOL version), then your best way to approach is to hire such a company.
If you migrate to a supported version (Airlfow 2 generally is "live" and supported) and find similar issue (it won't be likely the same as Airflow 2 codebase and 1.10 codebase differ quite drastically) - feel free to open one, and there is a chance someone in their free time will help you (but again, if you expect some SLA, paying for support is a good option).
I just write it so that you have the right expectations set.
Feel free to search for relevant issue. We are not going to open the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #15990: Logging bug in a long runs
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #15990:
URL: https://github.com/apache/airflow/issues/15990#issuecomment-1081364324
> @dstandish was the original issue resolved? In this case, please, provide corresponded note in the changelog. We are still using custom logging. In another case, please, open the issue again.
Jarek provided the [resolving PR](https://github.com/apache/airflow/pull/17649) [above](https://github.com/apache/airflow/issues/15990#issuecomment-1081015823). You can open the changelog and search for `17649` to see the changelog note. It was fixed quite a while ago (in 2.0.3) it seems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15990: Logging bug in a long runs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15990:
URL: https://github.com/apache/airflow/issues/15990#issuecomment-1081016935
@trucnguyenlam -> just upgrade to latest cncf.kubernetes provider.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] egS27 edited a comment on issue #15990: Logging bug in a long runs
Posted by GitBox <gi...@apache.org>.
egS27 edited a comment on issue #15990:
URL: https://github.com/apache/airflow/issues/15990#issuecomment-1081328848
@dstandish was the original issue resolved? In this case, please, provide corresponded note in the changelog. We are still using custom logging. In another case, please, open the issue again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] egS27 commented on issue #15990: Logging bug in a long runs
Posted by GitBox <gi...@apache.org>.
egS27 commented on issue #15990:
URL: https://github.com/apache/airflow/issues/15990#issuecomment-846955288
@jedcunningham
`True` is set for `enable_tcp_keepalive`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish closed issue #15990: Logging bug in a long runs
Posted by GitBox <gi...@apache.org>.
dstandish closed issue #15990:
URL: https://github.com/apache/airflow/issues/15990
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] egS27 commented on issue #15990: Logging bug in a long runs
Posted by GitBox <gi...@apache.org>.
egS27 commented on issue #15990:
URL: https://github.com/apache/airflow/issues/15990#issuecomment-846783060
@jedcunningham Thanks for replying, I will provide updates soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] egS27 commented on issue #15990: Logging bug in a long runs
Posted by GitBox <gi...@apache.org>.
egS27 commented on issue #15990:
URL: https://github.com/apache/airflow/issues/15990#issuecomment-1081766675
@dstandish Jarek mentioned that this pr was for resolving not the original issue. (**different** story which appears this issue was resolved in PR 17649). This is why I am asking about the original issue, in particular, maybe don't get the idea correctly...
@potiuk I got your point. Yes, we identified a bug in the stale version, but we use a fresh one and don't miss updates. If the bug was identified, I think we can expect that it was fixed in the next versions, and I am asking about a mention of that fix. I am not expecting any support for 1.10. I am asking about the note in the changelog(with pr attached), with which I can go and make sure that issue was fixed.
Also, I cannot find this pr in the airflow changelog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #15990: Logging bug in a long runs
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #15990:
URL: https://github.com/apache/airflow/issues/15990#issuecomment-846312161
@egS27, can you confirm if you have `enable_tcp_keepalive` set to True? If it isn't, can you try it with it enabled?
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#enable-tcp-keepalive
(Reason I ask is the old default was False, so instances that were updated sometimes still have the old default in airflow.cfg)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #15990: Logging bug in a long runs
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #15990:
URL: https://github.com/apache/airflow/issues/15990#issuecomment-846000749
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15990: Logging bug in a long runs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15990:
URL: https://github.com/apache/airflow/issues/15990#issuecomment-1081684477
Again - just to set the expectations, becasue apparently your expectations were different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15990: Logging bug in a long runs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15990:
URL: https://github.com/apache/airflow/issues/15990#issuecomment-1081788489
@sg27 Because you are looking in a wrong place. This is a kubernetes provider fix, not airflow. https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #15990: Logging bug in a long runs
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15990:
URL: https://github.com/apache/airflow/issues/15990#issuecomment-1081015823
Original issue was for an old airlfow version (EOL already)
The issue raised by @trucnguyenlam was a different story which appears this issue was resolved in PR https://github.com/apache/airflow/pull/17649 that PR was released in airflow 2.0.3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] egS27 edited a comment on issue #15990: Logging bug in a long runs
Posted by GitBox <gi...@apache.org>.
egS27 edited a comment on issue #15990:
URL: https://github.com/apache/airflow/issues/15990#issuecomment-846955288
@jedcunningham
`True` was set for `enable_tcp_keepalive`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] trucnguyenlam commented on issue #15990: Logging bug in a long runs
Posted by GitBox <gi...@apache.org>.
trucnguyenlam commented on issue #15990:
URL: https://github.com/apache/airflow/issues/15990#issuecomment-881289283
I can confirm this happens on airflow 2.1.1, eks 1.19, we use s3 as a logging backend
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15990: Logging bug in a long runs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15990:
URL: https://github.com/apache/airflow/issues/15990#issuecomment-1081015823
it appears this issue was resolved in PR https://github.com/apache/airflow/pull/17649
that PR was released in airflow 2.0.3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15990: Logging bug in a long runs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15990:
URL: https://github.com/apache/airflow/issues/15990#issuecomment-1081683888
And just to clarify - the 2.0.2 issue you mentioned has been (likely) fixed in https://github.com/apache/airflow/pull/17649
We are NOT referring to issues in the changelogs. Our changelogs refer to PRs that solve the problem instead. And those PRs are linked to the issues when they are relevant (like this one is ). So please. Upgrade. Don't expect your particular issue will be refered to in the changelog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] egS27 commented on issue #15990: Logging bug in a long runs
Posted by GitBox <gi...@apache.org>.
egS27 commented on issue #15990:
URL: https://github.com/apache/airflow/issues/15990#issuecomment-1081328848
@dstandish was the original issue resolved? In this case, please, provide corresponded note in the changelog. We are still using custom logging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15990: Logging bug in a long runs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15990:
URL: https://github.com/apache/airflow/issues/15990#issuecomment-1081796709
And I heartily recommend "search" on Airlfow docs site. It really fast and really good:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org