You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/06/01 20:20:47 UTC
[GitHub] [airflow] jhonatanrt opened a new issue #16207: Airflow scheduler - Can't start correctly because it has error message about DagFileProcessor0-Process failed
jhonatanrt opened a new issue #16207:
URL: https://github.com/apache/airflow/issues/16207
Hi guys,
I have a bug problem with the last version of airflow (2.1.0). Because, I can't start the app correctly. If you could help me with another configuration or fix.
When I like to make a schedule in airflow Dag or sync new Dags, Airflow didn't work.
My configuration:
CPU: 1GB
Memory: 2GB
Environment:
"AIRFLOW_DATABASE_NAME": *****,
"AIRFLOW_DATABASE_PORT_NUMBER": "5432",
"AIRFLOW_DATABASE_HOST": ******,
"AIRFLOW_EXECUTOR": "CeleryExecutor",
"AIRFLOW_WEBSERVER_HOST": "webserver.airflow",
"AIRFLOW_LOAD_EXAMPLES": "no",
"AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL": "30",
"REDIS_HOST": *****,
"BUCKET_NAME": *****
_Error screenshot_
Thanks for your help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #16207: Airflow scheduler - Can't start correctly because it has error message about DagFileProcessor0-Process failed
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #16207:
URL: https://github.com/apache/airflow/issues/16207#issuecomment-852456635
@jhonatanrt, can you post the complete traceback?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jhonatanrt edited a comment on issue #16207: Airflow scheduler - Can't start correctly because it has error message about DagFileProcessor0-Process failed
Posted by GitBox <gi...@apache.org>.
jhonatanrt edited a comment on issue #16207:
URL: https://github.com/apache/airflow/issues/16207#issuecomment-852480383
> @jhonatanrt, can you post the complete traceback?
Ok, no problem.
Another thing, You can see "sync s3" because I put bash to sync with s3 bucket to get new Dags File. I'm working in AWS ECS Fargate and I only have problem with the scheduler container.
`
[38;5;6mairflow-scheduler [38;5;5m21:35:27.38 [0m[38;5;2mINFO [0m ==> ** Airflow setup finished! **
--
[38;5;6mairflow-scheduler [38;5;5m21:35:27.40 [0m[38;5;2mINFO [0m ==> ** Starting Airflow **
____________ _____________
____ \|__( )_________ __/__ /________ __
____ /\| \|_ /__ ___/_ /_ __ /_ __ \_ \| /\| / /
___ ___ \| / _ / _ __/ _ / / /_/ /_ \|/ \|/ /
_/_/ \|_/_/ /_/ /_/ /_/ \____/____/\|__/
[[34m2021-06-01 21:35:28,607[0m] {[34mscheduler_job.py:[0m1253} INFO[0m - Starting the scheduler[0m
[[34m2021-06-01 21:35:28,607[0m] {[34mscheduler_job.py:[0m1258} INFO[0m - Processing each file at most -1 times[0m
[[34m2021-06-01 21:35:28,611[0m] {[34mdag_processing.py:[0m254} INFO[0m - Launched DagFileProcessorManager with pid: 154[0m
[[34m2021-06-01 21:35:28,612[0m] {[34mscheduler_job.py:[0m1822} INFO[0m - Resetting orphaned tasks for active dag runs[0m
[[34m2021-06-01 21:35:28,616[0m] {[34msettings.py:[0m52} INFO[0m - Configured default timezone Timezone('UTC')[0m
Process DagFileProcessor0-Process:
Traceback (most recent call last):
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 166, in _run_file_processor
set_context(log, file_path)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 174, in set_context
handler.set_context(value)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 56, in set_context
local_loc = self._init_file(filename)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 144, in _init_file
open(log_file_path, "a").close()
PermissionError: [Errno 13] Permission denied: '/opt/bitnami/airflow/logs/scheduler/2021-06-01/demo.py.log'
Process DagFileProcessor1-Process:
Traceback (most recent call last):
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 166, in _run_file_processor
set_context(log, file_path)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 174, in set_context
handler.set_context(value)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 56, in set_context
local_loc = self._init_file(filename)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 144, in _init_file
open(log_file_path, "a").close()
PermissionError: [Errno 13] Permission denied: '/opt/bitnami/airflow/logs/scheduler/2021-06-01/demo.py.log'
sync s3
Process DagFileProcessor2-Process:
Traceback (most recent call last):
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 166, in _run_file_processor
set_context(log, file_path)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 174, in set_context
handler.set_context(value)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 56, in set_context
local_loc = self._init_file(filename)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 144, in _init_file
open(log_file_path, "a").close()
PermissionError: [Errno 13] Permission denied: '/opt/bitnami/airflow/logs/scheduler/2021-06-01/demo.py.log'
Process DagFileProcessor3-Process:
Traceback (most recent call last):
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 166, in _run_file_processor
set_context(log, file_path)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 174, in set_context
handler.set_context(value)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 56, in set_context
local_loc = self._init_file(filename)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 144, in _init_file
open(log_file_path, "a").close()
PermissionError: [Errno 13] Permission denied: '/opt/bitnami/airflow/logs/scheduler/2021-06-01/demo.py.log'
sync s3
Process DagFileProcessor4-Process:
Traceback (most recent call last):
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 166, in _run_file_processor
set_context(log, file_path)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 174, in set_context
handler.set_context(value)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 56, in set_context
local_loc = self._init_file(filename)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 144, in _init_file
open(log_file_path, "a").close()
PermissionError: [Errno 13] Permission denied: '/opt/bitnami/airflow/logs/scheduler/2021-06-01/demo.py.log'
Process DagFileProcessor5-Process:
Traceback (most recent call last):
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 166, in _run_file_processor
set_context(log, file_path)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 174, in set_context
handler.set_context(value)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 56, in set_context
local_loc = self._init_file(filename)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 144, in _init_file
open(log_file_path, "a").close()
PermissionError: [Errno 13] Permission denied: '/opt/bitnami/airflow/logs/scheduler/2021-06-01/demo.py.log'
sync s3
Process DagFileProcessor6-Process:
Traceback (most recent call last):
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 166, in _run_file_processor
set_context(log, file_path)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 174, in set_context
handler.set_context(value)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 56, in set_context
local_loc = self._init_file(filename)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 144, in _init_file
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jhonatanrt commented on issue #16207: Airflow scheduler - Can't start correctly because it has error message about DagFileProcessor0-Process failed
Posted by GitBox <gi...@apache.org>.
jhonatanrt commented on issue #16207:
URL: https://github.com/apache/airflow/issues/16207#issuecomment-853622861
Thanks @jedcunningham. I had a problem with the user root. when I changed to user 1001 that work well. 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jhonatanrt edited a comment on issue #16207: Airflow scheduler - Can't start correctly because it has error message about DagFileProcessor0-Process failed
Posted by GitBox <gi...@apache.org>.
jhonatanrt edited a comment on issue #16207:
URL: https://github.com/apache/airflow/issues/16207#issuecomment-852480383
> @jhonatanrt, can you post the complete traceback?
Ok, no problem.
Another thing, You can see "sync s3" because I put bash to sync with s3 bucket to get new Dags File. I'm working in AWS ECS Fargate and I only have problem with the scheduler container.
[38;5;6mairflow-scheduler [38;5;5m21:35:27.38 [0m[38;5;2mINFO [0m ==> ** Airflow setup finished! **
--
[38;5;6mairflow-scheduler [38;5;5m21:35:27.40 [0m[38;5;2mINFO [0m ==> ** Starting Airflow **
____________ _____________
____ \|__( )_________ __/__ /________ __
____ /\| \|_ /__ ___/_ /_ __ /_ __ \_ \| /\| / /
___ ___ \| / _ / _ __/ _ / / /_/ /_ \|/ \|/ /
_/_/ \|_/_/ /_/ /_/ /_/ \____/____/\|__/
[[34m2021-06-01 21:35:28,607[0m] {[34mscheduler_job.py:[0m1253} INFO[0m - Starting the scheduler[0m
[[34m2021-06-01 21:35:28,607[0m] {[34mscheduler_job.py:[0m1258} INFO[0m - Processing each file at most -1 times[0m
[[34m2021-06-01 21:35:28,611[0m] {[34mdag_processing.py:[0m254} INFO[0m - Launched DagFileProcessorManager with pid: 154[0m
[[34m2021-06-01 21:35:28,612[0m] {[34mscheduler_job.py:[0m1822} INFO[0m - Resetting orphaned tasks for active dag runs[0m
[[34m2021-06-01 21:35:28,616[0m] {[34msettings.py:[0m52} INFO[0m - Configured default timezone Timezone('UTC')[0m
Process DagFileProcessor0-Process:
Traceback (most recent call last):
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 166, in _run_file_processor
set_context(log, file_path)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 174, in set_context
handler.set_context(value)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 56, in set_context
local_loc = self._init_file(filename)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 144, in _init_file
open(log_file_path, "a").close()
PermissionError: [Errno 13] Permission denied: '/opt/bitnami/airflow/logs/scheduler/2021-06-01/demo.py.log'
Process DagFileProcessor1-Process:
Traceback (most recent call last):
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 166, in _run_file_processor
set_context(log, file_path)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 174, in set_context
handler.set_context(value)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 56, in set_context
local_loc = self._init_file(filename)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 144, in _init_file
open(log_file_path, "a").close()
PermissionError: [Errno 13] Permission denied: '/opt/bitnami/airflow/logs/scheduler/2021-06-01/demo.py.log'
sync s3
Process DagFileProcessor2-Process:
Traceback (most recent call last):
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 166, in _run_file_processor
set_context(log, file_path)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 174, in set_context
handler.set_context(value)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 56, in set_context
local_loc = self._init_file(filename)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 144, in _init_file
open(log_file_path, "a").close()
PermissionError: [Errno 13] Permission denied: '/opt/bitnami/airflow/logs/scheduler/2021-06-01/demo.py.log'
Process DagFileProcessor3-Process:
Traceback (most recent call last):
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 166, in _run_file_processor
set_context(log, file_path)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 174, in set_context
handler.set_context(value)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 56, in set_context
local_loc = self._init_file(filename)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 144, in _init_file
open(log_file_path, "a").close()
PermissionError: [Errno 13] Permission denied: '/opt/bitnami/airflow/logs/scheduler/2021-06-01/demo.py.log'
sync s3
Process DagFileProcessor4-Process:
Traceback (most recent call last):
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 166, in _run_file_processor
set_context(log, file_path)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 174, in set_context
handler.set_context(value)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 56, in set_context
local_loc = self._init_file(filename)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 144, in _init_file
open(log_file_path, "a").close()
PermissionError: [Errno 13] Permission denied: '/opt/bitnami/airflow/logs/scheduler/2021-06-01/demo.py.log'
Process DagFileProcessor5-Process:
Traceback (most recent call last):
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 166, in _run_file_processor
set_context(log, file_path)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 174, in set_context
handler.set_context(value)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 56, in set_context
local_loc = self._init_file(filename)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 144, in _init_file
open(log_file_path, "a").close()
PermissionError: [Errno 13] Permission denied: '/opt/bitnami/airflow/logs/scheduler/2021-06-01/demo.py.log'
sync s3
Process DagFileProcessor6-Process:
Traceback (most recent call last):
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 166, in _run_file_processor
set_context(log, file_path)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 174, in set_context
handler.set_context(value)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 56, in set_context
local_loc = self._init_file(filename)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 144, in _init_file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jhonatanrt closed issue #16207: Airflow scheduler - Can't start correctly because it has error message about DagFileProcessor0-Process failed
Posted by GitBox <gi...@apache.org>.
jhonatanrt closed issue #16207:
URL: https://github.com/apache/airflow/issues/16207
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #16207: Airflow scheduler - Can't start correctly because it has error message about DagFileProcessor0-Process failed
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #16207:
URL: https://github.com/apache/airflow/issues/16207#issuecomment-853161787
```
PermissionError: [Errno 13] Permission denied: '/opt/bitnami/airflow/logs/scheduler/2021-06-01/demo.py.log'
```
It looks like the scheduler can't write log files. You'll need to debug why as that would be something with the bitnami chart, image, or your cluster. If you can reproduce it, it might be worth an issue over at bitnami.
(If you weren't aware, the community also recently released the official community chart: https://airflow.apache.org/docs/helm-chart/stable/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jhonatanrt closed issue #16207: Airflow scheduler - Can't start correctly because it has error message about DagFileProcessor0-Process failed
Posted by GitBox <gi...@apache.org>.
jhonatanrt closed issue #16207:
URL: https://github.com/apache/airflow/issues/16207
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jhonatanrt commented on issue #16207: Airflow scheduler - Can't start correctly because it has error message about DagFileProcessor0-Process failed
Posted by GitBox <gi...@apache.org>.
jhonatanrt commented on issue #16207:
URL: https://github.com/apache/airflow/issues/16207#issuecomment-852480383
> @jhonatanrt, can you post the complete traceback?
Ok, no problem.
Another thing, You can see "sync s3" because I put bash to sync with s3 bucket to get new Dags File. I'm working in AWS ECS Fargate and I only have problem with the scheduler container.
[38;5;6mairflow-scheduler [38;5;5m21:35:27.38 [0m[38;5;2mINFO [0m ==> ** Airflow setup finished! **
--
[38;5;6mairflow-scheduler [38;5;5m21:35:27.40 [0m[38;5;2mINFO [0m ==> ** Starting Airflow **
____________ _____________
____ \|__( )_________ __/__ /________ __
____ /\| \|_ /__ ___/_ /_ __ /_ __ \_ \| /\| / /
___ ___ \| / _ / _ __/ _ / / /_/ /_ \|/ \|/ /
_/_/ \|_/_/ /_/ /_/ /_/ \____/____/\|__/
[[34m2021-06-01 21:35:28,607[0m] {[34mscheduler_job.py:[0m1253} INFO[0m - Starting the scheduler[0m
[[34m2021-06-01 21:35:28,607[0m] {[34mscheduler_job.py:[0m1258} INFO[0m - Processing each file at most -1 times[0m
[[34m2021-06-01 21:35:28,611[0m] {[34mdag_processing.py:[0m254} INFO[0m - Launched DagFileProcessorManager with pid: 154[0m
[[34m2021-06-01 21:35:28,612[0m] {[34mscheduler_job.py:[0m1822} INFO[0m - Resetting orphaned tasks for active dag runs[0m
[[34m2021-06-01 21:35:28,616[0m] {[34msettings.py:[0m52} INFO[0m - Configured default timezone Timezone('UTC')[0m
Process DagFileProcessor0-Process:
Traceback (most recent call last):
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 166, in _run_file_processor
set_context(log, file_path)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 174, in set_context
handler.set_context(value)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 56, in set_context
local_loc = self._init_file(filename)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 144, in _init_file
open(log_file_path, "a").close()
PermissionError: [Errno 13] Permission denied: '/opt/bitnami/airflow/logs/scheduler/2021-06-01/demo.py.log'
Process DagFileProcessor1-Process:
Traceback (most recent call last):
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 166, in _run_file_processor
set_context(log, file_path)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 174, in set_context
handler.set_context(value)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 56, in set_context
local_loc = self._init_file(filename)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 144, in _init_file
open(log_file_path, "a").close()
PermissionError: [Errno 13] Permission denied: '/opt/bitnami/airflow/logs/scheduler/2021-06-01/demo.py.log'
sync s3
Process DagFileProcessor2-Process:
Traceback (most recent call last):
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 166, in _run_file_processor
set_context(log, file_path)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 174, in set_context
handler.set_context(value)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 56, in set_context
local_loc = self._init_file(filename)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 144, in _init_file
open(log_file_path, "a").close()
PermissionError: [Errno 13] Permission denied: '/opt/bitnami/airflow/logs/scheduler/2021-06-01/demo.py.log'
Process DagFileProcessor3-Process:
Traceback (most recent call last):
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 166, in _run_file_processor
set_context(log, file_path)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 174, in set_context
handler.set_context(value)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 56, in set_context
local_loc = self._init_file(filename)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 144, in _init_file
open(log_file_path, "a").close()
PermissionError: [Errno 13] Permission denied: '/opt/bitnami/airflow/logs/scheduler/2021-06-01/demo.py.log'
sync s3
Process DagFileProcessor4-Process:
Traceback (most recent call last):
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 166, in _run_file_processor
set_context(log, file_path)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 174, in set_context
handler.set_context(value)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 56, in set_context
local_loc = self._init_file(filename)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 144, in _init_file
open(log_file_path, "a").close()
PermissionError: [Errno 13] Permission denied: '/opt/bitnami/airflow/logs/scheduler/2021-06-01/demo.py.log'
Process DagFileProcessor5-Process:
Traceback (most recent call last):
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 166, in _run_file_processor
set_context(log, file_path)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 174, in set_context
handler.set_context(value)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 56, in set_context
local_loc = self._init_file(filename)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 144, in _init_file
open(log_file_path, "a").close()
PermissionError: [Errno 13] Permission denied: '/opt/bitnami/airflow/logs/scheduler/2021-06-01/demo.py.log'
sync s3
Process DagFileProcessor6-Process:
Traceback (most recent call last):
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 166, in _run_file_processor
set_context(log, file_path)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 174, in set_context
handler.set_context(value)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 56, in set_context
local_loc = self._init_file(filename)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 144, in _init_file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jhonatanrt commented on issue #16207: Airflow scheduler - Can't start correctly because it has error message about DagFileProcessor0-Process failed
Posted by GitBox <gi...@apache.org>.
jhonatanrt commented on issue #16207:
URL: https://github.com/apache/airflow/issues/16207#issuecomment-853622861
Thanks @jedcunningham. I had a problem with the user root. when I changed to user 1001 that work well. 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] raefezzeldin commented on issue #16207: Airflow scheduler - Can't start correctly because it has error message about DagFileProcessor0-Process failed
Posted by GitBox <gi...@apache.org>.
raefezzeldin commented on issue #16207:
URL: https://github.com/apache/airflow/issues/16207#issuecomment-878656215
@jhonatanrt I am facing the same issue when I tries to presist logs from where did you change the user?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #16207: Airflow scheduler - Can't start correctly because it has error message about DagFileProcessor0-Process failed
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #16207:
URL: https://github.com/apache/airflow/issues/16207#issuecomment-852419897
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org