You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/01/21 10:09:20 UTC
[GitHub] [airflow] ClassyLion opened a new issue #13805: Could not get scheduler_job_id
ClassyLion opened a new issue #13805:
URL: https://github.com/apache/airflow/issues/13805
I upgraded from AirFlow 1.10.14 to AirFlow 2.0.0. AirFlow is deployed in kubernetes remotely.
Every time I try to run a dag, it gets scheduled, but tasks are not started. When I click on task and press run, I get this error.
I've tried creating a new dag, deleting a dag and running after it is added back and I still get this error.
```
Python version: 3.8.7
Airflow version: 2.0.0
Node: airflow-web-ffdd89d6-h98vj
-------------------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.8/site-packages/airflow/www/auth.py", line 34, in decorated
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/airflow/www/decorators.py", line 60, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/airflow/www/views.py", line 1366, in run
executor.start()
File "/usr/local/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 493, in start
raise AirflowException("Could not get scheduler_job_id")
airflow.exceptions.AirflowException: Could not get scheduler_job_id
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil edited a comment on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
kaxil edited a comment on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-767726269
Can you post your DAG please? Does other DAGs run ? Can you post the logs from the Scheduler too with DEBUG level logging.
Did you unpause the DAG too?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-777829618
I'll take a look in my morning.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-776818004
@MaxTaggart you might need to rebase from master to get tests passing
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-764526302
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ihorton1 edited a comment on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
ihorton1 edited a comment on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-844291902
@dimberman unless I'm mistaken, it appears that although @MaxTaggart fixed the backfill issue related to this issue, the run command is still throwing the original error in 2.0.2.
Would adding `job_id = 'manual'` in airflow/cli/commands/task_command.py fix the issue you think? I noticed it doesn't look it job_id getting set when using the Kubernetes Executor.
```
def _run_task_by_executor(args, dag, ti):
"""
Sends the task to the executor for execution. This can result in the task being started by another host
if the executor implementation does
"""
pickle_id = None
if args.ship_dag:
try:
# Running remotely, so pickling the DAG
with create_session() as session:
pickle = DagPickle(dag)
session.add(pickle)
pickle_id = pickle.id
# TODO: This should be written to a log
print(f'Pickled dag {dag} as pickle_id: {pickle_id}')
except Exception as e:
print('Could not pickle the DAG')
print(e)
raise e
executor = ExecutorLoader.get_default_executor()
executor.job_id = 'manual'
executor.start()
print("Sending to executor.")
executor.queue_task_instance(
ti,
mark_success=args.mark_success,
pickle_id=pickle_id,
ignore_all_deps=args.ignore_all_dependencies,
ignore_depends_on_past=args.ignore_depends_on_past,
ignore_task_deps=args.ignore_dependencies,
ignore_ti_state=args.force,
pool=args.pool,
)
executor.heartbeat()
executor.end()
```
If so, wondering how i should test it. Happy to take this on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Reddy1990 edited a comment on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
Reddy1990 edited a comment on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-877090063
HI All, I'm still getting the same error. While using CeleryKubernetesExecutor
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2021-07-09 10:31:52,972] {scheduler_job.py:1251} INFO - Starting the scheduler
[2021-07-09 10:31:52,972] {scheduler_job.py:1256} INFO - Processing each file at most -1 times
[2021-07-09 10:31:52,978] {kubernetes_executor.py:471} INFO - Start Kubernetes executor
[2021-07-09 10:31:52,978] {scheduler_job.py:1302} ERROR - Exception when executing SchedulerJob._run_scheduler_loop
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 1276, in _execute
self.executor.start()
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/executors/celery_kubernetes_executor.py", line 60, in start
self.kubernetes_executor.start()
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/executors/kubernetes_executor.py", line 473, in start
raise AirflowException("Could not get scheduler_job_id")
airflow.exceptions.AirflowException: Could not get scheduler_job_id
[2021-07-09 10:31:52,979] {dag_processing.py:453} WARNING - Ending without manager process.
[2021-07-09 10:31:52,979] {scheduler_job.py:1305} INFO - Exited execute loop
Environment:
Using Airflow 2.0.2 Version
Kubernetes v1.20.7
Cloud provider or hardware configuration: AWS Linux
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jdotjdot commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
jdotjdot commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-818863418
We're actually running into this issue just running the `CeleryKubernetesExecutor` without doing backfill at all. Same issue as airflow-helm/charts#114.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Reddy1990 edited a comment on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
Reddy1990 edited a comment on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-877090063
HI All, I'm still getting the same error. While using CeleryKubernetesExecutor
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2021-07-09 10:31:52,972] {scheduler_job.py:1251} INFO - Starting the scheduler
[2021-07-09 10:31:52,972] {scheduler_job.py:1256} INFO - Processing each file at most -1 times
[2021-07-09 10:31:52,978] {kubernetes_executor.py:471} INFO - Start Kubernetes executor
[2021-07-09 10:31:52,978] {scheduler_job.py:1302} ERROR - Exception when executing SchedulerJob._run_scheduler_loop
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 1276, in _execute
self.executor.start()
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/executors/celery_kubernetes_executor.py", line 60, in start
self.kubernetes_executor.start()
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/executors/kubernetes_executor.py", line 473, in start
raise AirflowException("Could not get scheduler_job_id")
airflow.exceptions.AirflowException: Could not get scheduler_job_id
[2021-07-09 10:31:52,979] {dag_processing.py:453} WARNING - Ending without manager process.
[2021-07-09 10:31:52,979] {scheduler_job.py:1305} INFO - Exited execute loop
Environment:
Using Airflow 2.0.2 Version
Kubernetes v1.20.7
Cloud provider or hardware configuration: AWS Linux
I was using KubernetesExecutor (all was working fine , faced no issues and able to run DAG's/Jobs) and now experimenting with CeleryKubernetesExecutor the issue started showing up and I check if any job/Dag/task is causing the issue where the Scheduler looks for but I tried after deleting DAG's and placing any empty dir but the issue is still there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-767700842
Just checking here --- This only happens when you try to run the "task" in the UI.
Can you try by just triggering the DAG itself and not an individual Task please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ClassyLion commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
ClassyLion commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-764812356
> Does it only happen with Kubernetes Executor?
Yes, it happens only with Kubernetes Executor, apparently. Tried LocalExecutor and task completes without problems when triggering the DAG.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-764526302
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MaxTaggart commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
MaxTaggart commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-777820686
Alright @dimberman or @ashb, I need to raise the flag again. The CI/CD is now failing in two spots. I can't find any error messages and I tried re-running it after catching up with `master` but it still isn't passing. These are the checks that are failing:
1. MySQL8, Py3.6: Always Core Other API CLI Providers WWW Integration Heisentests
2. MySQL8, Py3.8: Always Core Other API CLI Providers WWW Integration Heisentests
Is it possible that the `BackFillJob` test is missing a mark, or is in the wrong section?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ClassyLion commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
ClassyLion commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-776275699
Hey, @MaxTaggart. I understand how it is. I managed to fully setup the dev environment on my side and made some attempts at hardcoding the values, but it didn't work for me. Also, dayjob takes requires attention as well, resulting in me not being able dedicate as much time as I would want to.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MaxTaggart commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
MaxTaggart commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-780923358
Hey @ashb, have you had a chance to take a look at this? I'm also happy to do more digging if you have any hunches about what might be going wrong.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-764696221
Does it only happen with Kubernetes Executor?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-764793241
Looks like a bug with Kubernetes Executor. Related issue: https://github.com/apache/airflow/issues/13805
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-817723855
> Hi all,
> Thank you for investigating into this. May I ask, when can we expect to have this fix released? Because it is preventing us from doing a manual backfill.
The fix will be available in Airflow 2.0.2 -- which should be released in a week or so
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-767726269
Can you post your DAG please? Does other DAGs run ? Can you post the logs from the Scheduler too with DEBUG level logging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-775272675
@ClassyLion Awesome! Glad to help you get set up :).
So here is the documentation on getting a dev environment set up https://github.com/apache/airflow/blob/master/CONTRIBUTORS_QUICK_START.rst
Please let me know if you run into any roadbumps (would be good for me to know so I can fix them)
After that I'm glad to jump on a zoom to help or feel free to comment here and I can answer any questions :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-774241825
So the solution here would be to give an arbitrary scheduler_job_id "a.k.a. 'backfill' or 'manual'" For these two use-cases.
@yjwong @ClassyLion would either of you be interested in taking this on? I'd be glad to help you get set up with a dev environment/take you through the PR process (no problem if you don't have the cycles but this seems like a pretty great first ticket :) ).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MaxTaggart commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
MaxTaggart commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-777608085
@dimberman It looks like PR #14160 is getting hung up on a language check that doesn't like "WhiteListRoundRobin" from the cassandra hook in `airflow/providers/apache/cassandra/hooks/cassandra.py`. What's weird is that the file should be excluded from that check since it is in the exclude list in `.pre-commit-config.yaml:317`:
``` yaml
- id: language-matters
language: pygrep
name: Check for language that we do not accept as community
description: Please use "deny_list" or "allow_list" instead.
entry: "(?i)(black|white)[_-]?list"
pass_filenames: true
exclude: >
(?x)
^airflow/providers/apache/cassandra/hooks/cassandra.py$|
^airflow/providers/apache/hive/operators/hive_stats.py$|
^airflow/providers/apache/hive/.*PROVIDER_CHANGES_*|
^airflow/providers/apache/hive/.*README.md$|
^tests/providers/apache/cassandra/hooks/test_cassandra.py$|
^docs/apache-airflow-providers-apache-cassandra/connections/cassandra.rst$|
^docs/apache-airflow-providers-apache-hive/commits.rst$|git|
^CHANGELOG.txt$
```
Any ideas?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] iwanbolzern commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
iwanbolzern commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-817733036
> > Hi all,
> > Thank you for investigating into this. May I ask, when can we expect to have this fix released? Because it is preventing us from doing a manual backfill.
>
> The fix will be available in Airflow 2.0.2 -- which should be released in a week or so
Hi @kaxil, Thanks for getting back. In this case we will wait a few more weeks 😉
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ClassyLion commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
ClassyLion commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-767683603
I currently don't have access to the specific AirFlow deployment, but I checked logs thoroughly multiple times and there were no issues with anything starting. This was the only problematic issue that appeared.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ClassyLion commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
ClassyLion commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-767727886
Other DAGs don't run aswell. I have logging set to DEBUG and it didn't change anything in the logs. I will share logs tomorrow, when I get access to deployment.
I unpaused the dag and it got scheduled and started, but the tasks were not running, therefore, not finishing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-776322924
Thank you @MaxTaggart.
Here is a test that should work for the BackFillJob
```
def test_backfill_has_job_id(self):
dag = self.dagbag.get_dag("test_start_date_scheduling")
dag.clear()
executor = MockExecutor(parallelism=16)
job = BackfillJob(
executor=executor,
dag=dag,
start_date=DEFAULT_DATE,
end_date=DEFAULT_DATE + datetime.timedelta(days=1),
run_backwards=True,
)
job.run()
assert executor.job_id != None
```
and here is one that should work for test_views.py
```
@mock.patch('airflow.executors.executor_loader.ExecutorLoader.get_default_executor')
def test_run_executor_has_job_id(self, get_default_executor_function):
executor = CeleryExecutor()
executor.heartbeat = lambda: True
get_default_executor_function.return_value = executor
task_id = 'runme_0'
form = dict(
task_id=task_id,
dag_id="example_bash_operator",
ignore_all_deps="false",
ignore_ti_state="false",
execution_date=self.EXAMPLE_DAG_DEFAULT_DATE,
origin='/home',
)
resp = self.client.post('run', data=form, follow_redirects=True)
assert executor.job_id !=None
```
Please let me know if those work as expected.
@ClassyLion Sorry this one didn't work out but now that you're set up if you wanna pick any of the "first issue" tagged issues for the project I'm still glad to help you get a commit in :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Reddy1990 commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
Reddy1990 commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-877090063
HI All, I'm still getting the same error.
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2021-07-09 10:31:52,972] {scheduler_job.py:1251} INFO - Starting the scheduler
[2021-07-09 10:31:52,972] {scheduler_job.py:1256} INFO - Processing each file at most -1 times
[2021-07-09 10:31:52,978] {kubernetes_executor.py:471} INFO - Start Kubernetes executor
[2021-07-09 10:31:52,978] {scheduler_job.py:1302} ERROR - Exception when executing SchedulerJob._run_scheduler_loop
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 1276, in _execute
self.executor.start()
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/executors/celery_kubernetes_executor.py", line 60, in start
self.kubernetes_executor.start()
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/executors/kubernetes_executor.py", line 473, in start
raise AirflowException("Could not get scheduler_job_id")
airflow.exceptions.AirflowException: Could not get scheduler_job_id
[2021-07-09 10:31:52,979] {dag_processing.py:453} WARNING - Ending without manager process.
[2021-07-09 10:31:52,979] {scheduler_job.py:1305} INFO - Exited execute loop
Environment:
Using Airflow 2.0.2 Version
Kubernetes v1.20.7
Cloud provider or hardware configuration: AWS Linux
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yjwong commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
yjwong commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-771751589
Seems to be related to this commit: https://github.com/apache/airflow/commit/3ca11eb9b02a2c2591292fd6b76e0e98b8f22656
In `airflow/jobs/scheduler_job.py`, `executor.job_id` is set but not for `airflow/jobs/backfill_job.py`. The same can be observed in `airflow/www/views.py` too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ihorton1 commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
ihorton1 commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-844291902
@dimberman unless I'm mistaken, it appears that although @MaxTaggart fixed the backfill issue related to this issue, the run command is still throwing the original error in 2.0.2.
Would adding `job_id = 'manual'` in airflow/cli/commands/task_command.py fix the issue you think?
`def _run_task_by_executor(args, dag, ti):
"""
Sends the task to the executor for execution. This can result in the task being started by another host
if the executor implementation does
"""
pickle_id = None
if args.ship_dag:
try:
# Running remotely, so pickling the DAG
with create_session() as session:
pickle = DagPickle(dag)
session.add(pickle)
pickle_id = pickle.id
# TODO: This should be written to a log
print(f'Pickled dag {dag} as pickle_id: {pickle_id}')
except Exception as e:
print('Could not pickle the DAG')
print(e)
raise e
executor = ExecutorLoader.get_default_executor()
executor.job_id = 'manual'
executor.start()
print("Sending to executor.")
executor.queue_task_instance(
ti,
mark_success=args.mark_success,
pickle_id=pickle_id,
ignore_all_deps=args.ignore_all_dependencies,
ignore_depends_on_past=args.ignore_depends_on_past,
ignore_task_deps=args.ignore_dependencies,
ignore_ti_state=args.force,
pool=args.pool,
)
executor.heartbeat()
executor.end()`
If so, wondering how i should test it. Happy to take this on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MaxTaggart commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
MaxTaggart commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-776337759
@dimberman Thanks for those tests, they are passing now and I pushed a new commit to that PR that includes them. Just out of curiosity, do we not need to create a unique `job_id` for each backfill/manual job? (Instead of hardcoding a constant)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-777630973
@MaxTaggart It is complaining about the `tags` file you added, not cassandra.py itself. You probably didn't mean to commit the tags file anyway :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ClassyLion commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
ClassyLion commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-768180676
> Can you post your DAG please? Does other DAGs run ? Can you post the logs from the Scheduler too with DEBUG level logging.
>
> Did you unpause the DAG too?
```
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2021-01-27 10:07:10,148] {scheduler_job.py:1241} INFO - Starting the scheduler
[2021-01-27 10:07:10,149] {scheduler_job.py:1246} INFO - Processing each file at most -1 times
[2021-01-27 10:07:10,150] {kubernetes_executor.py:491} INFO - Start Kubernetes executor
[2021-01-27 10:07:10,186] {kubernetes_executor.py:462} INFO - When executor started up, found 0 queued task instances
[2021-01-27 10:07:10,192] {dag_processing.py:250} INFO - Launched DagFileProcessorManager with pid: 42
[2021-01-27 10:07:10,193] {scheduler_job.py:1751} INFO - Resetting orphaned tasks for active dag runs
[2021-01-27 10:07:10,193] {kubernetes_executor.py:126} INFO - Event: and now my watch begins starting at resource_version: 0
[2021-01-27 10:07:10,211] {settings.py:52} INFO - Configured default timezone Timezone('UTC')
[2021-01-27 10:11:15,626] {scheduler_job.py:938} INFO - 2 tasks up for execution:
<TaskInstance: test.monthly.weekly.trigger.test_task 2021-01-26 16:58:38.208907+00:00 [scheduled]>
<TaskInstance: test.monthly.weekly.trigger.test_task 2021-01-27 10:11:06.043581+00:00 [scheduled]>
[2021-01-27 10:11:15,635] {scheduler_job.py:967} INFO - Figuring out tasks to run in Pool(name=default_pool) with 128 open slots and 2 task instances ready to be queued
[2021-01-27 10:11:15,635] {scheduler_job.py:995} INFO - DAG test.monthly.weekly.trigger has 0/16 running and queued tasks
[2021-01-27 10:11:15,635] {scheduler_job.py:995} INFO - DAG test.monthly.weekly.trigger has 1/16 running and queued tasks
[2021-01-27 10:11:15,636] {scheduler_job.py:1060} INFO - Setting the following tasks to queued state:
<TaskInstance: test.monthly.weekly.trigger.test_task 2021-01-26 16:58:38.208907+00:00 [scheduled]>
<TaskInstance: test.monthly.weekly.trigger.test_task 2021-01-27 10:11:06.043581+00:00 [scheduled]>
[2021-01-27 10:11:15,639] {scheduler_job.py:1102} INFO - Sending TaskInstanceKey(dag_id='test.monthly.weekly.trigger', task_id='test_task', execution_date=datetime.datetime(2021, 1, 26, 16, 58, 38, 208907, tzinfo=Timezone('UTC')), try_number=1) to executor with priority 1 and queue default
[2021-01-27 10:11:15,640] {base_executor.py:79} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'test.monthly.weekly.trigger', 'test_task', '2021-01-26T16:58:38.208907+00:00', '--local', '--pool', 'default_pool', '--subdir', '/usr/local/airflow/dags/repo/dags/test.monthly.weekly.trigger.py']
[2021-01-27 10:11:15,640] {scheduler_job.py:1102} INFO - Sending TaskInstanceKey(dag_id='test.monthly.weekly.trigger', task_id='test_task', execution_date=datetime.datetime(2021, 1, 27, 10, 11, 6, 43581, tzinfo=Timezone('UTC')), try_number=1) to executor with priority 1 and queue default
[2021-01-27 10:11:15,640] {base_executor.py:79} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'test.monthly.weekly.trigger', 'test_task', '2021-01-27T10:11:06.043581+00:00', '--local', '--pool', 'default_pool', '--subdir', '/usr/local/airflow/dags/repo/dags/test.monthly.weekly.trigger.py']
[2021-01-27 10:11:15,684] {kubernetes_executor.py:510} INFO - Add task TaskInstanceKey(dag_id='test.monthly.weekly.trigger', task_id='test_task', execution_date=datetime.datetime(2021, 1, 26, 16, 58, 38, 208907, tzinfo=Timezone('UTC')), try_number=1) with command ['airflow', 'tasks', 'run', 'test.monthly.weekly.trigger', 'test_task', '2021-01-26T16:58:38.208907+00:00', '--local', '--pool', 'default_pool', '--subdir', '/usr/local/airflow/dags/repo/dags/test.monthly.weekly.trigger.py'] with executor_config {'KubernetesExecutor': {'request_memory': '256Mi', 'limit_memory': '512Mi'}}
/usr/local/lib/python3.8/site-packages/airflow/kubernetes/pod_generator.py:193 DeprecationWarning: Using a dictionary for the executor_config is deprecated and will soon be removed.please use a `kubernetes.client.models.V1Pod` class with a "pod_override" key instead.
[2021-01-27 10:11:15,901] {kubernetes_executor.py:510} INFO - Add task TaskInstanceKey(dag_id='test.monthly.weekly.trigger', task_id='test_task', execution_date=datetime.datetime(2021, 1, 27, 10, 11, 6, 43581, tzinfo=Timezone('UTC')), try_number=1) with command ['airflow', 'tasks', 'run', 'test.monthly.weekly.trigger', 'test_task', '2021-01-27T10:11:06.043581+00:00', '--local', '--pool', 'default_pool', '--subdir', '/usr/local/airflow/dags/repo/dags/test.monthly.weekly.trigger.py'] with executor_config {'KubernetesExecutor': {'request_memory': '256Mi', 'limit_memory': '512Mi'}}
[2021-01-27 10:11:15,906] {kubernetes_executor.py:277} INFO - Kubernetes job is (TaskInstanceKey(dag_id='test.monthly.weekly.trigger', task_id='test_task', execution_date=datetime.datetime(2021, 1, 26, 16, 58, 38, 208907, tzinfo=Timezone('UTC')), try_number=1), ['airflow', 'tasks', 'run', 'test.monthly.weekly.trigger', 'test_task', '2021-01-26T16:58:38.208907+00:00', '--local', '--pool', 'default_pool', '--subdir', '/usr/local/airflow/dags/repo/dags/test.monthly.weekly.trigger.py'], {'api_version': 'v1',
'kind': 'Pod',
'metadata': {'annotations': None,
'cluster_name': None,
'creation_timestamp': None,
'deletion_grace_period_seconds': None,
'deletion_timestamp': None,
'finalizers': None,
'generate_name': None,
'generation': None,
'initializers': None,
'labels': None,
'managed_fields': None,
'name': None,
'namespace': None,
'owner_references': None,
'resource_version': None,
'self_link': None,
'uid': None},
'spec': {'active_deadline_seconds': None,
'affinity': None,
'automount_service_account_token': None,
'containers': [{'args': [],
'command': [],
'env': [],
'env_from': [],
'image': None,
'image_pull_policy': None,
'lifecycle': None,
'liveness_probe': None,
'name': 'base',
'ports': [],
'readiness_probe': None,
'resources': {'limits': {'memory': '512Mi'},
'requests': {'memory': '256Mi'}},
'security_context': None,
'stdin': None,
'stdin_once': None,
'termination_message_path': None,
'termination_message_policy': None,
'tty': None,
'volume_devices': None,
'volume_mounts': [],
'working_dir': None}],
'dns_config': None,
'dns_policy': None,
'enable_service_links': None,
'host_aliases': None,
'host_ipc': None,
'host_network': False,
'host_pid': None,
'hostname': None,
'image_pull_secrets': [],
'init_containers': None,
'node_name': None,
'node_selector': None,
'preemption_policy': None,
'priority': None,
'priority_class_name': None,
'readiness_gates': None,
'restart_policy': None,
'runtime_class_name': None,
'scheduler_name': None,
'security_context': None,
'service_account': None,
'service_account_name': None,
'share_process_namespace': None,
'subdomain': None,
'termination_grace_period_seconds': None,
'tolerations': None,
'volumes': []},
'status': None}, None)
[2021-01-27 10:11:15,984] {scheduler_job.py:1193} INFO - Executor reports execution of test.monthly.weekly.trigger.test_task execution_date=2021-01-26 16:58:38.208907+00:00 exited with status queued for try_number 1
[2021-01-27 10:11:15,984] {scheduler_job.py:1193} INFO - Executor reports execution of test.monthly.weekly.trigger.test_task execution_date=2021-01-27 10:11:06.043581+00:00 exited with status queued for try_number 1
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yjwong commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
yjwong commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-771751589
Seems to be related to this commit: https://github.com/apache/airflow/commit/3ca11eb9b02a2c2591292fd6b76e0e98b8f22656
In `airflow/jobs/scheduler_job.py`, `executor.job_id` is set but not for `airflow/jobs/backfill_job.py`. The same can be observed in `airflow/www/views.py` too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ihorton1 edited a comment on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
ihorton1 edited a comment on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-844291902
@dimberman unless I'm mistaken, it appears that although @MaxTaggart fixed the backfill issue related to this issue, the run command is still throwing the original error in 2.0.2.
Would adding `job_id = 'manual'` in airflow/cli/commands/task_command.py fix the issue you think?
```
def _run_task_by_executor(args, dag, ti):
"""
Sends the task to the executor for execution. This can result in the task being started by another host
if the executor implementation does
"""
pickle_id = None
if args.ship_dag:
try:
# Running remotely, so pickling the DAG
with create_session() as session:
pickle = DagPickle(dag)
session.add(pickle)
pickle_id = pickle.id
# TODO: This should be written to a log
print(f'Pickled dag {dag} as pickle_id: {pickle_id}')
except Exception as e:
print('Could not pickle the DAG')
print(e)
raise e
executor = ExecutorLoader.get_default_executor()
executor.job_id = 'manual'
executor.start()
print("Sending to executor.")
executor.queue_task_instance(
ti,
mark_success=args.mark_success,
pickle_id=pickle_id,
ignore_all_deps=args.ignore_all_dependencies,
ignore_depends_on_past=args.ignore_depends_on_past,
ignore_task_deps=args.ignore_dependencies,
ignore_ti_state=args.force,
pool=args.pool,
)
executor.heartbeat()
executor.end()
```
If so, wondering how i should test it. Happy to take this on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ClassyLion commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
ClassyLion commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-774297920
> So the solution here would be to give an arbitrary scheduler_job_id "a.k.a. 'backfill' or 'manual'" For these two use-cases.
>
> @yjwong @ClassyLion would either of you be interested in taking this on? I'd be glad to help you get set up with a dev environment/take you through the PR process (no problem if you don't have the cycles but this seems like a pretty great first ticket :) ).
I can say that I am interested and I'm up for it... At least to try it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-767625496
Hi @ClassyLion can you post the logs leading up to that error? Does the executor just never start?
What's strange is that this error seems to suggest that the executor never receives a job_id.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zhang699 commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
zhang699 commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-781793430
I also encounter this problem, but curious about another questions, why load same DAG three times before the exception happen, and the last two of loaded DAG path seems incorrect.
Â
```
[2021-02-19 03:35:50,204] {dagbag.py:413} DEBUG - Loaded DAG <DAG: etl_dag1>
[2021-02-19 03:35:50,205] {dagbag.py:287} DEBUG - Importing /home/airflow/dags/..data/code-dag.py
[2021-02-19 03:35:50,222] {dagbag.py:413} DEBUG - Loaded DAG <DAG: etl_dag1>
[2021-02-19 03:35:50,223] {dagbag.py:287} DEBUG - Importing /home/airflow/dags/..2021_02_19_03_05_19.306169647/code-dag.py
[2021-02-19 03:35:50,239] {dagbag.py:413} DEBUG - Loaded DAG <DAG: etl_dag1>
```
Â
```
airflow@webserver-6b975954d-pf6h8:/opt/airflow$ airflow dags backfill etl_dag1 -s 2020-01-20 -e 2020-01-21
[2021-02-19 03:35:48,536] {settings.py:210} DEBUG - Setting up DB connection pool (PID 214)
[2021-02-19 03:35:48,537] {settings.py:281} DEBUG - settings.prepare_engine_args(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=214
[2021-02-19 03:35:48,636] {cli_action_loggers.py:40} DEBUG - Adding <function default_action_log at 0x7f7b77054320> to pre execution callback
[2021-02-19 03:35:50,016] {cli_action_loggers.py:66} DEBUG - Calling callbacks: [<function default_action_log at 0x7f7b77054320>]
/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/dag_command.py:62 PendingDeprecationWarning: --ignore-first-depends-on-past is deprecated as the value is always set to True
[2021-02-19 03:35:50,050] {dagbag.py:448} INFO - Filling up the DagBag from /home/airflow/dags
[2021-02-19 03:35:50,051] {dagbag.py:287} DEBUG - Importing /home/airflow/dags/code-dag.py
/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:26 DeprecationWarning: This module is deprecated. Please use `kubernetes.client.models.V1Volume`.
/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:27 DeprecationWarning: This module is deprecated. Please use `kubernetes.client.models.V1VolumeMount`.
[2021-02-19 03:35:50,204] {dagbag.py:413} DEBUG - Loaded DAG <DAG: etl_dag1>
[2021-02-19 03:35:50,205] {dagbag.py:287} DEBUG - Importing /home/airflow/dags/..data/code-dag.py
[2021-02-19 03:35:50,222] {dagbag.py:413} DEBUG - Loaded DAG <DAG: etl_dag1>
[2021-02-19 03:35:50,223] {dagbag.py:287} DEBUG - Importing /home/airflow/dags/..2021_02_19_03_05_19.306169647/code-dag.py
[2021-02-19 03:35:50,239] {dagbag.py:413} DEBUG - Loaded DAG <DAG: etl_dag1>
[2021-02-19 03:35:50,240] {executor_loader.py:82} DEBUG - Loading core executor: KubernetesExecutor
[2021-02-19 03:35:50,398] {kubernetes_executor.py:473} INFO - Start Kubernetes executor
[2021-02-19 03:35:50,415] {cli_action_loggers.py:84} DEBUG - Calling callbacks: []
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 89, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/dag_command.py", line 116, in dag_backfill
run_backwards=args.run_backwards,
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/dag.py", line 1706, in run
job.run()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 237, in run
self._execute()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 65, in wrapper
return func(*args, session=session, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/backfill_job.py", line 788, in _execute
executor.start()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/kubernetes_executor.py", line 475, in start
raise AirflowException("Could not get scheduler_job_id")
airflow.exceptions.AirflowException: Could not get scheduler_job_id
[2021-02-19 03:35:50,429] {settings.py:292} DEBUG - Disposing DB connection pool (PID 214)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ClassyLion edited a comment on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
ClassyLion edited a comment on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-767683603
I currently don't have access to the specific AirFlow deployment, but I checked logs thoroughly multiple times and there were no issues with anything starting. This was the only problematic issue that appeared.
When I get access to the deployment, I'll update with specific information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ClassyLion edited a comment on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
ClassyLion edited a comment on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-768180676
> Can you post your DAG please? Does other DAGs run ? Can you post the logs from the Scheduler too with DEBUG level logging.
>
> Did you unpause the DAG too?
```
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2021-01-27 10:07:10,148] {scheduler_job.py:1241} INFO - Starting the scheduler
[2021-01-27 10:07:10,149] {scheduler_job.py:1246} INFO - Processing each file at most -1 times
[2021-01-27 10:07:10,150] {kubernetes_executor.py:491} INFO - Start Kubernetes executor
[2021-01-27 10:07:10,186] {kubernetes_executor.py:462} INFO - When executor started up, found 0 queued task instances
[2021-01-27 10:07:10,192] {dag_processing.py:250} INFO - Launched DagFileProcessorManager with pid: 42
[2021-01-27 10:07:10,193] {scheduler_job.py:1751} INFO - Resetting orphaned tasks for active dag runs
[2021-01-27 10:07:10,193] {kubernetes_executor.py:126} INFO - Event: and now my watch begins starting at resource_version: 0
[2021-01-27 10:07:10,211] {settings.py:52} INFO - Configured default timezone Timezone('UTC')
[2021-01-27 10:11:15,626] {scheduler_job.py:938} INFO - 2 tasks up for execution:
<TaskInstance: test.monthly.weekly.trigger.test_task 2021-01-26 16:58:38.208907+00:00 [scheduled]>
<TaskInstance: test.monthly.weekly.trigger.test_task 2021-01-27 10:11:06.043581+00:00 [scheduled]>
[2021-01-27 10:11:15,635] {scheduler_job.py:967} INFO - Figuring out tasks to run in Pool(name=default_pool) with 128 open slots and 2 task instances ready to be queued
[2021-01-27 10:11:15,635] {scheduler_job.py:995} INFO - DAG test.monthly.weekly.trigger has 0/16 running and queued tasks
[2021-01-27 10:11:15,635] {scheduler_job.py:995} INFO - DAG test.monthly.weekly.trigger has 1/16 running and queued tasks
[2021-01-27 10:11:15,636] {scheduler_job.py:1060} INFO - Setting the following tasks to queued state:
<TaskInstance: test.monthly.weekly.trigger.test_task 2021-01-26 16:58:38.208907+00:00 [scheduled]>
<TaskInstance: test.monthly.weekly.trigger.test_task 2021-01-27 10:11:06.043581+00:00 [scheduled]>
[2021-01-27 10:11:15,639] {scheduler_job.py:1102} INFO - Sending TaskInstanceKey(dag_id='test.monthly.weekly.trigger', task_id='test_task', execution_date=datetime.datetime(2021, 1, 26, 16, 58, 38, 208907, tzinfo=Timezone('UTC')), try_number=1) to executor with priority 1 and queue default
[2021-01-27 10:11:15,640] {base_executor.py:79} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'test.monthly.weekly.trigger', 'test_task', '2021-01-26T16:58:38.208907+00:00', '--local', '--pool', 'default_pool', '--subdir', '/usr/local/airflow/dags/repo/dags/test.monthly.weekly.trigger.py']
[2021-01-27 10:11:15,640] {scheduler_job.py:1102} INFO - Sending TaskInstanceKey(dag_id='test.monthly.weekly.trigger', task_id='test_task', execution_date=datetime.datetime(2021, 1, 27, 10, 11, 6, 43581, tzinfo=Timezone('UTC')), try_number=1) to executor with priority 1 and queue default
[2021-01-27 10:11:15,640] {base_executor.py:79} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'test.monthly.weekly.trigger', 'test_task', '2021-01-27T10:11:06.043581+00:00', '--local', '--pool', 'default_pool', '--subdir', '/usr/local/airflow/dags/repo/dags/test.monthly.weekly.trigger.py']
[2021-01-27 10:11:15,684] {kubernetes_executor.py:510} INFO - Add task TaskInstanceKey(dag_id='test.monthly.weekly.trigger', task_id='test_task', execution_date=datetime.datetime(2021, 1, 26, 16, 58, 38, 208907, tzinfo=Timezone('UTC')), try_number=1) with command ['airflow', 'tasks', 'run', 'test.monthly.weekly.trigger', 'test_task', '2021-01-26T16:58:38.208907+00:00', '--local', '--pool', 'default_pool', '--subdir', '/usr/local/airflow/dags/repo/dags/test.monthly.weekly.trigger.py'] with executor_config {'KubernetesExecutor': {'request_memory': '256Mi', 'limit_memory': '512Mi'}}
/usr/local/lib/python3.8/site-packages/airflow/kubernetes/pod_generator.py:193 DeprecationWarning: Using a dictionary for the executor_config is deprecated and will soon be removed.please use a `kubernetes.client.models.V1Pod` class with a "pod_override" key instead.
[2021-01-27 10:11:15,901] {kubernetes_executor.py:510} INFO - Add task TaskInstanceKey(dag_id='test.monthly.weekly.trigger', task_id='test_task', execution_date=datetime.datetime(2021, 1, 27, 10, 11, 6, 43581, tzinfo=Timezone('UTC')), try_number=1) with command ['airflow', 'tasks', 'run', 'test.monthly.weekly.trigger', 'test_task', '2021-01-27T10:11:06.043581+00:00', '--local', '--pool', 'default_pool', '--subdir', '/usr/local/airflow/dags/repo/dags/test.monthly.weekly.trigger.py'] with executor_config {'KubernetesExecutor': {'request_memory': '256Mi', 'limit_memory': '512Mi'}}
[2021-01-27 10:11:15,906] {kubernetes_executor.py:277} INFO - Kubernetes job is (TaskInstanceKey(dag_id='test.monthly.weekly.trigger', task_id='test_task', execution_date=datetime.datetime(2021, 1, 26, 16, 58, 38, 208907, tzinfo=Timezone('UTC')), try_number=1), ['airflow', 'tasks', 'run', 'test.monthly.weekly.trigger', 'test_task', '2021-01-26T16:58:38.208907+00:00', '--local', '--pool', 'default_pool', '--subdir', '/usr/local/airflow/dags/repo/dags/test.monthly.weekly.trigger.py'], {'api_version': 'v1',
'kind': 'Pod',
'metadata': {'annotations': None,
'cluster_name': None,
'creation_timestamp': None,
'deletion_grace_period_seconds': None,
'deletion_timestamp': None,
'finalizers': None,
'generate_name': None,
'generation': None,
'initializers': None,
'labels': None,
'managed_fields': None,
'name': None,
'namespace': None,
'owner_references': None,
'resource_version': None,
'self_link': None,
'uid': None},
'spec': {'active_deadline_seconds': None,
'affinity': None,
'automount_service_account_token': None,
'containers': [{'args': [],
'command': [],
'env': [],
'env_from': [],
'image': None,
'image_pull_policy': None,
'lifecycle': None,
'liveness_probe': None,
'name': 'base',
'ports': [],
'readiness_probe': None,
'resources': {'limits': {'memory': '512Mi'},
'requests': {'memory': '256Mi'}},
'security_context': None,
'stdin': None,
'stdin_once': None,
'termination_message_path': None,
'termination_message_policy': None,
'tty': None,
'volume_devices': None,
'volume_mounts': [],
'working_dir': None}],
'dns_config': None,
'dns_policy': None,
'enable_service_links': None,
'host_aliases': None,
'host_ipc': None,
'host_network': False,
'host_pid': None,
'hostname': None,
'image_pull_secrets': [],
'init_containers': None,
'node_name': None,
'node_selector': None,
'preemption_policy': None,
'priority': None,
'priority_class_name': None,
'readiness_gates': None,
'restart_policy': None,
'runtime_class_name': None,
'scheduler_name': None,
'security_context': None,
'service_account': None,
'service_account_name': None,
'share_process_namespace': None,
'subdomain': None,
'termination_grace_period_seconds': None,
'tolerations': None,
'volumes': []},
'status': None}, None)
[2021-01-27 10:11:15,984] {scheduler_job.py:1193} INFO - Executor reports execution of test.monthly.weekly.trigger.test_task execution_date=2021-01-26 16:58:38.208907+00:00 exited with status queued for try_number 1
[2021-01-27 10:11:15,984] {scheduler_job.py:1193} INFO - Executor reports execution of test.monthly.weekly.trigger.test_task execution_date=2021-01-27 10:11:06.043581+00:00 exited with status queued for try_number 1
[2021-01-27 10:11:15,987] {kubernetes_executor.py:147} INFO - Event: testmonthlyweeklytriggertesttask-b2dda5ecac3c44b9870e38a018c4c421 had an event of type ADDED
[2021-01-27 10:11:15,988] {kubernetes_executor.py:202} INFO - Event: testmonthlyweeklytriggertesttask-b2dda5ecac3c44b9870e38a018c4c421 Pending
[2021-01-27 10:11:15,992] {kubernetes_executor.py:147} INFO - Event: testmonthlyweeklytriggertesttask-b2dda5ecac3c44b9870e38a018c4c421 had an event of type MODIFIED
[2021-01-27 10:11:15,992] {kubernetes_executor.py:202} INFO - Event: testmonthlyweeklytriggertesttask-b2dda5ecac3c44b9870e38a018c4c421 Pending
[2021-01-27 10:11:15,996] {kubernetes_executor.py:147} INFO - Event: testmonthlyweeklytriggertesttask-b2dda5ecac3c44b9870e38a018c4c421 had an event of type MODIFIED
[2021-01-27 10:11:15,996] {kubernetes_executor.py:202} INFO - Event: testmonthlyweeklytriggertesttask-b2dda5ecac3c44b9870e38a018c4c421 Pending
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ClassyLion commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
ClassyLion commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-813850685
Is it possible to implement this fix in deployment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-776817345
@MaxTaggart the job_id is mostly to identify which scheduler launched the job. With how Scheduler HA works in 2.0, once the tasks launches, the other schedulers should be able to adopt the orphan task. No unique names needed :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ClassyLion commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
ClassyLion commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-764812356
> Does it only happen with Kubernetes Executor?
Yes, it happens only with Kubernetes Executor, apparently. Tried LocalExecutor and task completes without problems when triggering the DAG.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MaxTaggart commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
MaxTaggart commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-776268465
Hey, @ClassyLion I don't mean to poach this one and I would be happy to collaborate, but this is holding up our deployment so I thought I would get started. @dimberman PR #14160 adds the hardcoded `job_id`'s, but I could use some guidance on how best to add test coverage. i.e. I'm still getting familiar with the testing landscape and I'm not sure if it would make more sense to add tests to an existing file or if this is new territory.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MaxTaggart commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
MaxTaggart commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-777632648
Aha, good call, I did not. Embarassing...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-764696221
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ClassyLion commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
ClassyLion commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-767705384
If I only trigger the DAG, the task will not run. It will be queued but not finished, or even running. And the DAG stays as running without any progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ClassyLion edited a comment on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
ClassyLion edited a comment on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-767705384
If I only trigger the DAG, the task will not run. It will be queued but not finished, or even running. And the DAG stays as running without any progress.
EDIT: That is the reason, why I tried to run the "task" in the UI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] iwanbolzern commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
iwanbolzern commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-817559307
Hi all,
Thank you for investigating into this. May I ask, when can we expect to have this fix released? Because it is preventing us from doing a manual backfill.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #13805:
URL: https://github.com/apache/airflow/issues/13805
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] arijitdey commented on issue #13805: Could not get scheduler_job_id
Posted by GitBox <gi...@apache.org>.
arijitdey commented on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-768070771
I am facing the same error while trying to backfill:
Logs:
```
[2021-01-27 06:35:50,209] {airflow-log-cleanup.py:44} INFO - ENABLE_DELETE_CHILD_LOG False
/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py:320 DeprecationWarning: The statsd_on option in [scheduler] has been moved to the statsd_on option in [metrics] - the old setting has been used, but please update your config.
[2021-01-27 06:35:50,376] {kubernetes_executor.py:491} INFO - Start Kubernetes executor
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 89, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/dag_command.py", line 116, in dag_backfill
run_backwards=args.run_backwards,
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/dag.py", line 1701, in run
job.run()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 237, in run
self._execute()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 65, in wrapper
return func(*args, session=session, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/backfill_job.py", line 788, in _execute
executor.start()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/kubernetes_executor.py", line 493, in start
raise AirflowException("Could not get scheduler_job_id")
airflow.exceptions.AirflowException: Could not get scheduler_job_id
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org