You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "pranav agrawal (JIRA)" <ji...@apache.org> on 2019/01/29 06:50:00 UTC
[jira] [Commented] (AIRFLOW-3750) SSHOperator Not Ending Process On
Remote Machine
[ https://issues.apache.org/jira/browse/AIRFLOW-3750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754657#comment-16754657 ]
pranav agrawal commented on AIRFLOW-3750:
-----------------------------------------
please provide update on this, thanks!
> SSHOperator Not Ending Process On Remote Machine
> ------------------------------------------------
>
> Key: AIRFLOW-3750
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3750
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
> Affects Versions: 1.10.1
> Reporter: pranav agrawal
> Priority: Major
> Attachments: Screen Shot 2019-01-22 at 5.17.27 PM.png
>
>
> *Airflow Version: 1.10.1*
> I have run a DAG containing SSHOperator,
> {code:java}
> from airflow.contrib.operators.ssh_operator import SSHOperator
> SSHOperator(
> task_id='mongo_transformation',
> ssh_conn_id='datapl_utils',
> command='java -cp /home/airflowclient/application-0.0.1-SNAPSHOT/application-0.0.1-SNAPSHOT-jar-with-dependencies.jar com.oyo.spark.application.MongoIngestionNew {}'.format(Variable.get('mongo_new_transformation')),
> retries=99999,
> dag=dag)
> {code}
> (ATTACHED ssh_conn_id details as image)
> It starts to run as expected on remote machine,
>
> {code:java}
> root 2638 1 0 Jan18 ? 00:00:00 /usr/sbin/sshd -D
> root 2705 1 0 Jan18 ? 00:00:00 /usr/lib/systemd/systemd-udevd
> root 2808 1 0 Jan18 ? 00:00:00 /usr/sbin/crond -n
> chrony 2953 1 0 Jan18 ? 00:00:00 /usr/sbin/chronyd
> root 2988 1 0 Jan18 ? 00:00:04 /usr/bin/amazon-ssm-agent
> root 3019 1 0 Jan18 ? 00:00:05 /usr/sbin/irqbalance --foreground --hintpolicy=subset
> root 3942 2638 0 05:36 ? 00:00:00 sshd: pranav.agrawal1 [priv]
> pranav.+ 3944 3942 0 05:36 ? 00:02:37 sshd: pranav.agrawal1@pts/0
> pranav.+ 3945 3944 0 05:36 pts/0 00:00:00 -bash
> root 4080 3945 0 05:50 pts/0 00:00:00 sudo su airflowclient
> root 4081 4080 0 05:50 pts/0 00:00:00 su airflowclient
> airflow+ 4082 4081 0 05:50 pts/0 00:00:00 bash
> root 4657 2638 0 06:41 ? 00:00:00 sshd: nitin.nagpal [priv]
> nitin.n+ 4659 4657 0 06:41 ? 00:00:00 sshd: nitin.nagpal@pts/1
> nitin.n+ 4660 4659 0 06:41 pts/1 00:00:00 -bash
> root 4683 4660 0 06:41 pts/1 00:00:00 sudo su - ec2-user
> root 4684 4683 0 06:41 pts/1 00:00:00 su - ec2-user
> ec2-user 4685 4684 0 06:41 pts/1 00:00:00 -bash
> root 4718 4685 0 06:42 pts/1 00:00:00 sudo su - airflowclient
> root 4719 4718 0 06:42 pts/1 00:00:00 su - airflowclient
> airflow+ 4720 4719 0 06:42 pts/1 00:00:00 -bash
> root 7120 2 0 10:10 ? 00:00:00 [kworker/0:0]
> postfix 7141 2524 0 10:12 ? 00:00:00 pickup -l -t unix -u
> root 7257 2 0 10:22 ? 00:00:07 [kworker/u4:1]
> root 7488 2 0 10:45 ? 00:00:00 [kworker/1:0]
> root 7527 2 0 10:50 ? 00:00:00 [kworker/0:3]
> root 7532 2 0 10:50 ? 00:00:00 [kworker/1:2]
> root 7538 2 0 10:50 ? 00:00:00 [kworker/u4:2]
> root 7573 2 0 10:55 ? 00:00:00 [kworker/1:1]
> root 7575 2638 0 10:55 ? 00:00:00 sshd: airflowclient [priv]
> airflow+ 7577 7575 0 10:55 ? 00:00:00 sshd: airflowclient@notty
> airflow+ 7578 7577 23 10:55 ? 00:00:12 java -cp /home/airflowclient/application-0.0.1-SNAPSHOT/application-0.0.1-SNAPSHOT-jar-with-dependencies.jar c
> {code}
> After a while, I marked the DAG as Success via UI, I get following log in airflow logs,
> {code:java}
> [2019-01-22 11:14:20,866] {logging_mixin.py:95} INFO - [2019-01-22 11:14:20,866] {jobs.py:2695} WARNING - State of this instance has been externally set to success. Taking the poison pill.
> [2019-01-22 11:14:20,911] {helpers.py:240} INFO - Sending 15 to GPID 8260
> [2019-01-22 11:14:20,911] {models.py:1636} ERROR - Received SIGTERM. Terminating subprocesses.
> [2019-01-22 11:14:22,341] {helpers.py:230} INFO - Process psutil.Process(pid=8260 (terminated)) (8260) terminated with exit code 0
> [2019-01-22 11:14:22,342] {logging_mixin.py:95} INFO - [2019-01-22 11:14:22,342] {jobs.py:2627} INFO - Task exited with return code 0{code}
> But on the remote machine, the java process is still running even after several minutes,
> {code:java}
> [airflowclient@prod-datapl-common-utils ~]$ ps -ef | grep "airflow+"
> airflow+ 4082 4081 0 05:50 pts/0 00:00:00 bash
> airflow+ 4720 4719 0 06:42 pts/1 00:00:00 -bash
> airflow+ 7578 1 99 10:55 ? 01:43:26 java -cp /home/airflowclient/application-0.0.1-SNAPSHOT/application-0.0.1-SNAPSHOT-jar-with-dependencies.jar com.oyo.spark.application.MongoIngestionNew prod 10.20.3.33 owner_portal app-jan21-test-1{code}
> server details,
> {code:java}
> [airflowclient@prod-datapl-common-utils ~]$ cat /etc/system-release
> Amazon Linux release 2 (Karoo){code}
> Please help me with fixing this at the earliest.
> Thanks!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)