You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/03/17 18:45:35 UTC

[GitHub] [airflow] anthonyp97 opened a new issue #14859: Smart Sensors - Inconsistent Logging

anthonyp97 opened a new issue #14859:
URL: https://github.com/apache/airflow/issues/14859

<!--

Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)

If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>

Please delete these comment blocks before submitting the issue.

-->

<!--

IMPORTANT!!!

PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!

PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!

Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.

-->

**Apache Airflow version**: 2.0.1

**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): N/A

**Environment**:

- **Cloud provider or hardware configuration**: AWS
- **OS** (e.g. from /etc/os-release): Ubuntu 18.04.2 LTS
- **Kernel** (e.g. `uname -a`): GNU/Linux 4.15.0-1043-aws x86_64
- **Others**: `LocalExecutor` with a `PARALLELISM` of 32, smart sensors enabled using 2 shards.

**What happened**:

The logs for the 2 smart sensor tasks that we run always show: `[2021-03-16 21:07:45,415] {smart_sensor.py:373} INFO - Loaded 0 sensor_works`. However, I can confirm that our FTP sensors are getting registered properly in the smart sensors in the FTP sensor logs. Strangely, the logs for a certain FTP sensor will occasionally show `{smart_sensor.py:373} INFO - 4 tasks detected.` and then the poke information for these 4 sensors appear in this FTP sensor's log, so it looks like the logs are going to the wrong location.

**What you expected to happen**:

The logs in the smart sensor tasks themselves should show the number of sensor_works loaded in each smart sensor shard (it should not alwasys be 0), and this information should not be in a random FTP sensor's logs. Also the logs for a specific FTP sensor should not include logs from different sensors.

**How to reproduce it**:

Run a `LocalExecutor` with several sensors running in parallel, and 2+ shards enabled for the smart sensors. The logs in `smart_sensor_group_shard_0`, `smart_sensor_group_shard_1` etc. should always show `Loaded 0 sensor_works` when that is not what is expected. You may be able to find the sensor_works loaded information in the logs of one of the sensors in your DAG.
<!---

As minimally and precisely as possible. Keep in mind we do not have access to your cluster or dags.

If you are using kubernetes, please attempt to recreate the issue using minikube or kind.

## Install minikube/kind

- Minikube https://minikube.sigs.k8s.io/docs/start/
- Kind https://kind.sigs.k8s.io/docs/user/quick-start/

If this is a UI bug, please provide a screenshot of the bug or a link to a youtube video of the bug in action

You can include images using the .md style of
![alt text](http://url/to/img.png)

To record a screencast, mac users can use QuickTime and then create an unlisted youtube video with the resulting .mov file.

--->

<!--

How often does this problem occur? Once? Every time etc?

Any relevant logs to include? Put them here in side a detail tag:
<details><summary>x.log</summary> lots of stuff </details>

-->

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org