You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "pruoff (via GitHub)" <gi...@apache.org> on 2024/02/15 07:41:52 UTC

[PR] Added Feature: search dags by task id with suggestions [airflow]

pruoff opened a new pull request, #37436:
URL: https://github.com/apache/airflow/pull/37436

   This PR closes: #37106
   
   Previously the DAG search bar could be used to search by DAG id and owner.
   
   Changes in /home UI from user's perspective:
   * DAG search with prefix "task: <task_id>" filters all DAGs by case-insensitive pattern-matching of the provided task_id.
   * suggestions of task_ids are automatically shown when typing. Since the same task_id can be present in many dags, an italic " in <dag_id>" is shown following the task_id suggestion.
   * actively limit the DAG search scope by prefixing "dag:" for dag_id search and "owner:" for owner search.
   * autocompleted suggestions are prefixed by their type e.g. "owner: airflow"
   
   ![Screenshot 2024-02-11 at 14 25 41](https://github.com/apache/airflow/assets/61174725/f7212303-dbfd-44e3-93d4-bae27d42efe3)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #37436:
URL: https://github.com/apache/airflow/pull/37436#discussion_r1699933006


##########
airflow/www/views.py:
##########
@@ -839,21 +841,92 @@ def index(self):
         with create_session() as session:
             # read orm_dags from the db
             dags_query = select(DagModel).where(~DagModel.is_subdag, DagModel.is_active)
-
+            dags_query = dags_query.where(DagModel.dag_id.in_(filter_dag_ids))
             if arg_search_query:
-                escaped_arg_search_query = arg_search_query.replace("_", r"\_")
-                dags_query = dags_query.where(
-                    DagModel.dag_id.ilike("%" + escaped_arg_search_query + "%", escape="\\")
-                    | DagModel._dag_display_property_value.ilike(
-                        "%" + escaped_arg_search_query + "%", escape="\\"
+                prefix_search_match = re2.match(r"(?i)(dag|owner|task):\s*", arg_search_query)
+                if prefix_search_match:
+                    query_prefix = prefix_search_match[0].lower()
+                    query_value = arg_search_query[len(query_prefix) :]
+                    if query_value:
+                        escaped_query_value = "%" + query_value.replace("_", r"\_") + "%"
+                        if query_prefix.startswith("task:"):

Review Comment:
   Nit: Item 0 is the whole group, item 1 is the first capturing group, and since we have that there's no need to use `.startswith`
   ```suggestion
                       query_prefix = prefix_search_match[1].lower()
                       query_value = arg_search_query[len(prefix_search_match[0]) :]
                       if query_value:
                           if query_prefix == "task":
   ```
   
   etc.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #37436:
URL: https://github.com/apache/airflow/pull/37436#discussion_r1699922154


##########
airflow/www/views.py:
##########
@@ -839,21 +841,92 @@ def index(self):
         with create_session() as session:
             # read orm_dags from the db
             dags_query = select(DagModel).where(~DagModel.is_subdag, DagModel.is_active)
-
+            dags_query = dags_query.where(DagModel.dag_id.in_(filter_dag_ids))
             if arg_search_query:
-                escaped_arg_search_query = arg_search_query.replace("_", r"\_")
-                dags_query = dags_query.where(
-                    DagModel.dag_id.ilike("%" + escaped_arg_search_query + "%", escape="\\")
-                    | DagModel._dag_display_property_value.ilike(
-                        "%" + escaped_arg_search_query + "%", escape="\\"
+                prefix_search_match = re2.match(r"(?i)(dag|owner|task):\s*", arg_search_query)
+                if prefix_search_match:
+                    query_prefix = prefix_search_match[0].lower()
+                    query_value = arg_search_query[len(query_prefix) :]
+                    if query_value:
+                        escaped_query_value = "%" + query_value.replace("_", r"\_") + "%"
+                        if query_prefix.startswith("task:"):
+                            if session.bind.dialect.name == "mysql":

Review Comment:
   Having four levels of nesting makes this code very hard to understand



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #37436:
URL: https://github.com/apache/airflow/pull/37436#discussion_r1699925294


##########
airflow/www/views.py:
##########
@@ -839,21 +841,92 @@ def index(self):
         with create_session() as session:
             # read orm_dags from the db
             dags_query = select(DagModel).where(~DagModel.is_subdag, DagModel.is_active)
-
+            dags_query = dags_query.where(DagModel.dag_id.in_(filter_dag_ids))
             if arg_search_query:
-                escaped_arg_search_query = arg_search_query.replace("_", r"\_")
-                dags_query = dags_query.where(
-                    DagModel.dag_id.ilike("%" + escaped_arg_search_query + "%", escape="\\")
-                    | DagModel._dag_display_property_value.ilike(
-                        "%" + escaped_arg_search_query + "%", escape="\\"
+                prefix_search_match = re2.match(r"(?i)(dag|owner|task):\s*", arg_search_query)
+                if prefix_search_match:
+                    query_prefix = prefix_search_match[0].lower()
+                    query_value = arg_search_query[len(query_prefix) :]
+                    if query_value:
+                        escaped_query_value = "%" + query_value.replace("_", r"\_") + "%"
+                        if query_prefix.startswith("task:"):
+                            if session.bind.dialect.name == "mysql":
+                                dag_id_to_task_subq = (
+                                    select(SerializedDagModel.dag_id)
+                                    .where(
+                                        func.regexp_like(
+                                            func.json_keys(
+                                                func.json_extract(
+                                                    SerializedDagModel._data, "$.dag._task_group.children"
+                                                )
+                                            ),
+                                            f".*{query_value}.*",
+                                            "i",
+                                        )
+                                    )
+                                    .subquery()
+                                )
+                                dags_query = dags_query.join(
+                                    dag_id_to_task_subq, DagModel.dag_id == dag_id_to_task_subq.c.dag_id
+                                )
+                            elif session.bind.dialect.name == "postgresql":
+                                dag_id_to_task_subq = select(
+                                    SerializedDagModel.dag_id,
+                                    cast(
+                                        func.json_extract_path(
+                                            func.json_array_elements(
+                                                func.json_extract_path(
+                                                    SerializedDagModel._data, "dag", "tasks"
+                                                )
+                                            ),
+                                            "__var",
+                                            "task_id",
+                                        ),
+                                        String,
+                                    ).label("task_id"),
+                                ).subquery()
+                                dags_query = dags_query.join(
+                                    dag_id_to_task_subq, DagModel.dag_id == dag_id_to_task_subq.c.dag_id
+                                ).where(dag_id_to_task_subq.c.task_id.ilike(escaped_query_value, escape="\\"))
+                            else:
+                                filtered_dag_models = session.scalars(dags_query).all()
+                                dag_bag = get_airflow_app().dag_bag
+                                filtered_dags = [
+                                    dag_bag.get_dag(DM.dag_id, session=session) for DM in filtered_dag_models
+                                ]
+                                filtered_by_tasks_dag_ids = [
+                                    dag.dag_id
+                                    for dag in filtered_dags
+                                    if [
+                                        task
+                                        for task in dag.tasks
+                                        if re2.search(r"(?i)" + query_value, task.task_id)
+                                    ]
+                                ]
+                                dags_query = dags_query.where(DagModel.dag_id.in_(filtered_by_tasks_dag_ids))
+                        else:
+                            if query_prefix.startswith("dag:"):
+                                dags_query = dags_query.where(
+                                    DagModel.dag_id.ilike(escaped_query_value, escape="\\")
+                                    | DagModel._dag_display_property_value.ilike(
+                                        escaped_query_value, escape="\\"
+                                    )
+                                )
+                            elif query_prefix.startswith("dag:"):

Review Comment:
   ```
                               if query_prefix.startswith("dag:"):
                                   ...
                               elif query_prefix.startswith("dag:"):
   ```
   
   Something seems wrong here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #37436:
URL: https://github.com/apache/airflow/pull/37436#issuecomment-2029323779

   @pruoff can you rebase and resolve conflicts?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "pruoff (via GitHub)" <gi...@apache.org>.
pruoff commented on PR #37436:
URL: https://github.com/apache/airflow/pull/37436#issuecomment-2041956580

   > Thanks for the fix, I can confirm it works now as advertised.
   > 
   > I approve from my side. I just kindly request to also alter the documentation a bit. Else the risk is high that nobody will be able to find and discover this cool freature - especially searching for tasks. Can you add / adjust documentation in `docs/apache-airflow/ui.rst`
   
   Thanks a lot! I now also extended the UI docs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on PR #37436:
URL: https://github.com/apache/airflow/pull/37436#issuecomment-2262715763

   (@pruoff And sorry for not noticing the original ping! My GH notifications are a mess)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "patrick-ruoff (via GitHub)" <gi...@apache.org>.
patrick-ruoff commented on PR #37436:
URL: https://github.com/apache/airflow/pull/37436#issuecomment-2472483271

   If anybody has the time to do the refactoring, please go ahead. I won't find any in the next weeks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on PR #37436:
URL: https://github.com/apache/airflow/pull/37436#issuecomment-2258917527

   Looking better! I think we should add an info tooltip telling users the new keywords search.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "pruoff (via GitHub)" <gi...@apache.org>.
pruoff commented on PR #37436:
URL: https://github.com/apache/airflow/pull/37436#issuecomment-2076827199

   > Let's rebase.
   
   Quite some effort this time. Included the display_dag_name logic


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "pruoff (via GitHub)" <gi...@apache.org>.
pruoff commented on PR #37436:
URL: https://github.com/apache/airflow/pull/37436#issuecomment-2130811150

   @ashb thanks for the review. I share your concern about resource consumption issues when loading all dags from the DB.
   
   I tried to query the JSON via many ways. One would be:
   `session.query(SerializedDagModel).filter(SerializedDagModel.__table__.columns.data['dag_id'].astext == dag_id)`
   yet I always end up in `NotImplementedError: Operator 'getitem' is not supported on this expression`.
   
   Can somebody support me here with a minimal working code snippet? In the repo, I cannot find any comparable code that queries the serialized dags in the DB.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #37436:
URL: https://github.com/apache/airflow/pull/37436#issuecomment-2067295317

   @bbovenzi or @eladkal Can yo umake a 2nd pass review, then I thing we should merge this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #37436:
URL: https://github.com/apache/airflow/pull/37436#discussion_r1699923636


##########
airflow/www/views.py:
##########
@@ -839,21 +841,92 @@ def index(self):
         with create_session() as session:
             # read orm_dags from the db
             dags_query = select(DagModel).where(~DagModel.is_subdag, DagModel.is_active)
-
+            dags_query = dags_query.where(DagModel.dag_id.in_(filter_dag_ids))
             if arg_search_query:
-                escaped_arg_search_query = arg_search_query.replace("_", r"\_")
-                dags_query = dags_query.where(
-                    DagModel.dag_id.ilike("%" + escaped_arg_search_query + "%", escape="\\")
-                    | DagModel._dag_display_property_value.ilike(
-                        "%" + escaped_arg_search_query + "%", escape="\\"
+                prefix_search_match = re2.match(r"(?i)(dag|owner|task):\s*", arg_search_query)
+                if prefix_search_match:
+                    query_prefix = prefix_search_match[0].lower()
+                    query_value = arg_search_query[len(query_prefix) :]
+                    if query_value:
+                        escaped_query_value = "%" + query_value.replace("_", r"\_") + "%"
+                        if query_prefix.startswith("task:"):
+                            if session.bind.dialect.name == "mysql":

Review Comment:
   Rather than having the database specific knowledge live in the view code this would make more sense if it was added as a class method on the model



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37436:
URL: https://github.com/apache/airflow/pull/37436#issuecomment-2471993811

   This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on PR #37436:
URL: https://github.com/apache/airflow/pull/37436#issuecomment-2473637625

   Also there is no feature release ahead for airflow 2.x. This could be directly incorporated to the new UI under development for airflow 3.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on PR #37436:
URL: https://github.com/apache/airflow/pull/37436#issuecomment-2474174094

   Yes, feel free to help on our advanced search for Airflow 3.0! https://github.com/apache/airflow/issues/42699


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on code in PR #37436:
URL: https://github.com/apache/airflow/pull/37436#discussion_r1496812864


##########
airflow/www/static/js/dags.js:
##########
@@ -126,16 +126,75 @@ $(".typeahead").typeahead({
       success: callback,
     });
   },
+  matcher(item) {
+    const it = this.displayText(item);
+    const query = this.query.toLowerCase();
+    const queryValue = query.replace(/^([a-z]+:\s*)/i, "");
+
+    return it.toLowerCase().indexOf(queryValue) !== -1;
+  },
+  displayText(item) {
+    if (typeof item !== "undefined" && typeof item.name !== "undefined") {
+      if (item.type === "task") {
+        return `${item.type}: ${item.name} <i>in ${item.dag_id}</i>`;
+      }
+      return `${item.type}: ${item.name}`;
+    }
+    return item;
+  },
+  sorter(items) {
+    const beginswith = [];
+    const caseSensitive = [];
+    const caseInsensitive = [];
+    let item;
+    let { query } = this;
+    query = query.replace(/^([a-z]+:\s*)/i, "");

Review Comment:
   Let's create a new variable for query instead of using `let` and reassigning its value.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "pruoff (via GitHub)" <gi...@apache.org>.
pruoff commented on PR #37436:
URL: https://github.com/apache/airflow/pull/37436#issuecomment-2257443570

   Here it finally is! I invested many hours in finding the correct methods. Asking for help did not create any responses neither here nor on Slack. There’s only one single occurrence of func.json_extract in the code that I found more or less by accident.
   
   Tested with MySQL, Postgres, and sqlite; sqlite supports func.json_extract but nothing like func.json_keys so I kept the original code for that backend.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on code in PR #37436:
URL: https://github.com/apache/airflow/pull/37436#discussion_r1496809388


##########
airflow/www/static/js/dags.js:
##########
@@ -126,16 +126,75 @@ $(".typeahead").typeahead({
       success: callback,
     });
   },
+  matcher(item) {
+    const it = this.displayText(item);
+    const query = this.query.toLowerCase();
+    const queryValue = query.replace(/^([a-z]+:\s*)/i, "");
+
+    return it.toLowerCase().indexOf(queryValue) !== -1;
+  },
+  displayText(item) {
+    if (typeof item !== "undefined" && typeof item.name !== "undefined") {
+      if (item.type === "task") {
+        return `${item.type}: ${item.name} <i>in ${item.dag_id}</i>`;
+      }
+      return `${item.type}: ${item.name}`;
+    }
+    return item;
+  },
+  sorter(items) {
+    const beginswith = [];
+    const caseSensitive = [];
+    const caseInsensitive = [];
+    let item;
+    let { query } = this;
+    query = query.replace(/^([a-z]+:\s*)/i, "");
+
+    while (items.length) {
+      item = items.shift();
+      if (!item.name.toLowerCase().indexOf(query.toLowerCase())) {
+        beginswith.push(item);
+      } else if (item.name.indexOf(query) !== -1) {
+        caseSensitive.push(item);
+      } else {
+        caseInsensitive.push(item);
+      }
+    }
+
+    return beginswith.concat(caseSensitive, caseInsensitive);
+  },
+  highlighter(displayText, item) {
+    let { query } = this;
+    query = query.replace(/^([a-z]+:\s*)/i, "");
+
+    if (query === "") {
+      return displayText;
+    }
+
+    let { name } = item;
+    query = query.replace(/[()/.*+?[\]]/g, (mat) => `\\${mat}`);
+    const reg = new RegExp(query, "gi");
+    name = name.replace(name, name.replace(reg, "<strong>$&</strong>"));
+    return displayText.replace(item.name, name);
+  },
   autoSelect: false,
   afterSelect(value) {
     const query = new URLSearchParams(window.location.search);
-    query.set("search", value.name);
     if (value.type === "owner") {
+      query.set("search", value.name);
       window.location = `${DAGS_INDEX}?${query}`;
     }
     if (value.type === "dag") {
+      query.set("search", value.name);

Review Comment:
   We don't want the query for the gridUrl since it's already directly in the url



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on code in PR #37436:
URL: https://github.com/apache/airflow/pull/37436#discussion_r1496808744


##########
airflow/www/static/js/dags.js:
##########
@@ -126,16 +126,75 @@ $(".typeahead").typeahead({
       success: callback,
     });
   },
+  matcher(item) {
+    const it = this.displayText(item);
+    const query = this.query.toLowerCase();
+    const queryValue = query.replace(/^([a-z]+:\s*)/i, "");
+
+    return it.toLowerCase().indexOf(queryValue) !== -1;
+  },
+  displayText(item) {
+    if (typeof item !== "undefined" && typeof item.name !== "undefined") {
+      if (item.type === "task") {
+        return `${item.type}: ${item.name} <i>in ${item.dag_id}</i>`;
+      }
+      return `${item.type}: ${item.name}`;
+    }
+    return item;
+  },
+  sorter(items) {
+    const beginswith = [];
+    const caseSensitive = [];
+    const caseInsensitive = [];
+    let item;
+    let { query } = this;
+    query = query.replace(/^([a-z]+:\s*)/i, "");
+
+    while (items.length) {
+      item = items.shift();
+      if (!item.name.toLowerCase().indexOf(query.toLowerCase())) {
+        beginswith.push(item);
+      } else if (item.name.indexOf(query) !== -1) {
+        caseSensitive.push(item);
+      } else {
+        caseInsensitive.push(item);
+      }
+    }
+
+    return beginswith.concat(caseSensitive, caseInsensitive);
+  },
+  highlighter(displayText, item) {
+    let { query } = this;
+    query = query.replace(/^([a-z]+:\s*)/i, "");
+
+    if (query === "") {
+      return displayText;
+    }
+
+    let { name } = item;
+    query = query.replace(/[()/.*+?[\]]/g, (mat) => `\\${mat}`);
+    const reg = new RegExp(query, "gi");
+    name = name.replace(name, name.replace(reg, "<strong>$&</strong>"));
+    return displayText.replace(item.name, name);
+  },
   autoSelect: false,
   afterSelect(value) {
     const query = new URLSearchParams(window.location.search);
-    query.set("search", value.name);
     if (value.type === "owner") {
+      query.set("search", value.name);
       window.location = `${DAGS_INDEX}?${query}`;
     }
     if (value.type === "dag") {
+      query.set("search", value.name);
       window.location = `${gridUrl.replace("__DAG_ID__", value.name)}?${query}`;
     }
+    if (value.type === "task") {
+      query.set("search", value.dag_id);

Review Comment:
   ```suggestion
         query.set("task_id", value.name);
   ```
   Rebase and then changing the searh param will allow you to navigate directly to the task in question



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #37436:
URL: https://github.com/apache/airflow/pull/37436#issuecomment-2030075088

   I checked and I have no auto-complete and suggestions when I type as described in the PR descriptions. Tested with Firefox and Chromium.
   Did I mis-undertstand the PR that suggestions are not there? Or are the suggestions browser specific?
   
   Also when not prefixing `dag: ` in the search the text entered always filters for the DAG ID. Search for tasks and owner though is working. But `dag: ` prefix smells un-needed if this is the default. Or shall the default be to make a global search?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "pruoff (via GitHub)" <gi...@apache.org>.
pruoff commented on PR #37436:
URL: https://github.com/apache/airflow/pull/37436#issuecomment-2041368166

   @jscheffl 
   > I checked and I have no auto-complete and suggestions when I type as described in the PR descriptions. Tested with Firefox and Chromium. Did I mis-undertstand the PR that suggestions are not there? Or are the suggestions browser specific?
   
   A bug was introduced when rebasing, which caused the autocomplete to fail - see the FIX commit. The feature works for me now in both Firefox and Chromium.
   
   > Also when not prefixing `dag: ` in the search the text entered always filters for the DAG ID. Search for tasks and owner though is working. But `dag: ` prefix smells un-needed if this is the default. Or shall the default be to make a global search?
   
   The default is to search by both dag id and owner. The `dag: ` prefix resolves confusion in case a dag id and owner are named the same. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi closed pull request #37436: Added Feature: search dags by task id with suggestions
URL: https://github.com/apache/airflow/pull/37436


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on code in PR #37436:
URL: https://github.com/apache/airflow/pull/37436#discussion_r1496812008


##########
airflow/www/static/js/dags.js:
##########
@@ -126,16 +126,75 @@ $(".typeahead").typeahead({
       success: callback,
     });
   },
+  matcher(item) {
+    const it = this.displayText(item);
+    const query = this.query.toLowerCase();
+    const queryValue = query.replace(/^([a-z]+:\s*)/i, "");
+
+    return it.toLowerCase().indexOf(queryValue) !== -1;
+  },
+  displayText(item) {
+    if (typeof item !== "undefined" && typeof item.name !== "undefined") {

Review Comment:
   Do we want to check against the string `"undefined"` or the javascript value `undefined`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on code in PR #37436:
URL: https://github.com/apache/airflow/pull/37436#discussion_r1496808744


##########
airflow/www/static/js/dags.js:
##########
@@ -126,16 +126,75 @@ $(".typeahead").typeahead({
       success: callback,
     });
   },
+  matcher(item) {
+    const it = this.displayText(item);
+    const query = this.query.toLowerCase();
+    const queryValue = query.replace(/^([a-z]+:\s*)/i, "");
+
+    return it.toLowerCase().indexOf(queryValue) !== -1;
+  },
+  displayText(item) {
+    if (typeof item !== "undefined" && typeof item.name !== "undefined") {
+      if (item.type === "task") {
+        return `${item.type}: ${item.name} <i>in ${item.dag_id}</i>`;
+      }
+      return `${item.type}: ${item.name}`;
+    }
+    return item;
+  },
+  sorter(items) {
+    const beginswith = [];
+    const caseSensitive = [];
+    const caseInsensitive = [];
+    let item;
+    let { query } = this;
+    query = query.replace(/^([a-z]+:\s*)/i, "");
+
+    while (items.length) {
+      item = items.shift();
+      if (!item.name.toLowerCase().indexOf(query.toLowerCase())) {
+        beginswith.push(item);
+      } else if (item.name.indexOf(query) !== -1) {
+        caseSensitive.push(item);
+      } else {
+        caseInsensitive.push(item);
+      }
+    }
+
+    return beginswith.concat(caseSensitive, caseInsensitive);
+  },
+  highlighter(displayText, item) {
+    let { query } = this;
+    query = query.replace(/^([a-z]+:\s*)/i, "");
+
+    if (query === "") {
+      return displayText;
+    }
+
+    let { name } = item;
+    query = query.replace(/[()/.*+?[\]]/g, (mat) => `\\${mat}`);
+    const reg = new RegExp(query, "gi");
+    name = name.replace(name, name.replace(reg, "<strong>$&</strong>"));
+    return displayText.replace(item.name, name);
+  },
   autoSelect: false,
   afterSelect(value) {
     const query = new URLSearchParams(window.location.search);
-    query.set("search", value.name);
     if (value.type === "owner") {
+      query.set("search", value.name);
       window.location = `${DAGS_INDEX}?${query}`;
     }
     if (value.type === "dag") {
+      query.set("search", value.name);
       window.location = `${gridUrl.replace("__DAG_ID__", value.name)}?${query}`;
     }
+    if (value.type === "task") {
+      query.set("search", value.dag_id);

Review Comment:
   ```suggestion
         query.set("task_id", value.name);
   ```
   Rebase and then changing the search param will allow you to navigate directly to the task in question



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #37436:
URL: https://github.com/apache/airflow/pull/37436#discussion_r1699924198


##########
airflow/www/views.py:
##########
@@ -839,21 +841,92 @@ def index(self):
         with create_session() as session:
             # read orm_dags from the db
             dags_query = select(DagModel).where(~DagModel.is_subdag, DagModel.is_active)
-
+            dags_query = dags_query.where(DagModel.dag_id.in_(filter_dag_ids))
             if arg_search_query:
-                escaped_arg_search_query = arg_search_query.replace("_", r"\_")
-                dags_query = dags_query.where(
-                    DagModel.dag_id.ilike("%" + escaped_arg_search_query + "%", escape="\\")
-                    | DagModel._dag_display_property_value.ilike(
-                        "%" + escaped_arg_search_query + "%", escape="\\"
+                prefix_search_match = re2.match(r"(?i)(dag|owner|task):\s*", arg_search_query)
+                if prefix_search_match:
+                    query_prefix = prefix_search_match[0].lower()
+                    query_value = arg_search_query[len(query_prefix) :]
+                    if query_value:
+                        escaped_query_value = "%" + query_value.replace("_", r"\_") + "%"
+                        if query_prefix.startswith("task:"):
+                            if session.bind.dialect.name == "mysql":
+                                dag_id_to_task_subq = (
+                                    select(SerializedDagModel.dag_id)
+                                    .where(
+                                        func.regexp_like(
+                                            func.json_keys(
+                                                func.json_extract(
+                                                    SerializedDagModel._data, "$.dag._task_group.children"
+                                                )
+                                            ),
+                                            f".*{query_value}.*",
+                                            "i",
+                                        )
+                                    )
+                                    .subquery()
+                                )
+                                dags_query = dags_query.join(
+                                    dag_id_to_task_subq, DagModel.dag_id == dag_id_to_task_subq.c.dag_id
+                                )
+                            elif session.bind.dialect.name == "postgresql":
+                                dag_id_to_task_subq = select(
+                                    SerializedDagModel.dag_id,
+                                    cast(
+                                        func.json_extract_path(
+                                            func.json_array_elements(
+                                                func.json_extract_path(
+                                                    SerializedDagModel._data, "dag", "tasks"

Review Comment:
   Why do we look at a totally different part of the JSON structure for mysql than postgres?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "pruoff (via GitHub)" <gi...@apache.org>.
pruoff commented on PR #37436:
URL: https://github.com/apache/airflow/pull/37436#issuecomment-2079612257

   > Do we want to include searching by task_display_name in this PR or in a follow up?
   
   Searching by task_display_name is already included and the PR does not change this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on code in PR #37436:
URL: https://github.com/apache/airflow/pull/37436#discussion_r1581435304


##########
airflow/www/views.py:
##########
@@ -5415,51 +5453,78 @@ class AutocompleteView(AirflowBaseView):
     def autocomplete(self, session: Session = NEW_SESSION):
         """Autocomplete."""
         query = unquote(request.args.get("query", ""))
+        query_prefix = ""
+        prefix_search_match = re2.match(r"(?i)(dag|owner|task):\s*", query)
+        if prefix_search_match:
+            query_prefix = prefix_search_match[0].lower()
+            query = query[len(query_prefix) :]
 
         if not query:
             return flask.json.jsonify([])
 
-        # Provide suggestions of dag_ids and owners
-        dag_ids_query = select(
-            sqla.literal("dag").label("type"),
-            DagModel.dag_id.label("name"),
-            DagModel._dag_display_property_value.label("dag_display_name"),
-        ).where(
-            ~DagModel.is_subdag,
-            DagModel.is_active,
-            or_(
-                DagModel.dag_id.ilike(f"%{query}%"),
-                DagModel._dag_display_property_value.ilike(f"%{query}%"),
-            ),
-        )
+        status = flask_session.get(FILTER_STATUS_COOKIE)
+        filter_dag_ids = get_auth_manager().get_permitted_dag_ids(user=g.user)
 
-        owners_query = (
-            select(
-                sqla.literal("owner").label("type"),
-                DagModel.owners.label("name"),
-                sqla.literal(None).label("dag_display_name"),
+        def _filter_dags_query(dags_query: Select) -> Select:
+            # Hide DAGs if not showing status: "all"
+            if status == "active":
+                dags_query = dags_query.where(~DagModel.is_paused)
+            elif status == "paused":
+                dags_query = dags_query.where(DagModel.is_paused)
+            return dags_query.where(
+                ~DagModel.is_subdag, DagModel.is_active, DagModel.dag_id.in_(filter_dag_ids)
             )
-            .distinct()
-            .where(~DagModel.is_subdag, DagModel.is_active, DagModel.owners.ilike(f"%{query}%"))
-        )
 
-        # Hide DAGs if not showing status: "all"
-        status = flask_session.get(FILTER_STATUS_COOKIE)
-        if status == "active":
-            dag_ids_query = dag_ids_query.where(~DagModel.is_paused)
-            owners_query = owners_query.where(~DagModel.is_paused)
-        elif status == "paused":
-            dag_ids_query = dag_ids_query.where(DagModel.is_paused)
-            owners_query = owners_query.where(DagModel.is_paused)
+        if query_prefix.startswith("task:"):
+            # Provide suggestions of task_ids
+            dags_query = _filter_dags_query(dags_query=select(DagModel.dag_id))
+            filtered_dag_ids = session.scalars(dags_query).all()
+            dag_bag = get_airflow_app().dag_bag
+            filtered_dags = [dag_bag.get_dag(dag_id, session=session) for dag_id in filtered_dag_ids]

Review Comment:
   In testing locally, I am keep getting errors from these dag_bag fetches:
   
   ```ValueError("Invalid value for `name`, must not be `None`")```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #37436:
URL: https://github.com/apache/airflow/pull/37436#discussion_r1581455829


##########
airflow/www/views.py:
##########
@@ -5415,51 +5453,78 @@ class AutocompleteView(AirflowBaseView):
     def autocomplete(self, session: Session = NEW_SESSION):
         """Autocomplete."""
         query = unquote(request.args.get("query", ""))
+        query_prefix = ""
+        prefix_search_match = re2.match(r"(?i)(dag|owner|task):\s*", query)
+        if prefix_search_match:
+            query_prefix = prefix_search_match[0].lower()
+            query = query[len(query_prefix) :]
 
         if not query:
             return flask.json.jsonify([])
 
-        # Provide suggestions of dag_ids and owners
-        dag_ids_query = select(
-            sqla.literal("dag").label("type"),
-            DagModel.dag_id.label("name"),
-            DagModel._dag_display_property_value.label("dag_display_name"),
-        ).where(
-            ~DagModel.is_subdag,
-            DagModel.is_active,
-            or_(
-                DagModel.dag_id.ilike(f"%{query}%"),
-                DagModel._dag_display_property_value.ilike(f"%{query}%"),
-            ),
-        )
+        status = flask_session.get(FILTER_STATUS_COOKIE)
+        filter_dag_ids = get_auth_manager().get_permitted_dag_ids(user=g.user)
 
-        owners_query = (
-            select(
-                sqla.literal("owner").label("type"),
-                DagModel.owners.label("name"),
-                sqla.literal(None).label("dag_display_name"),
+        def _filter_dags_query(dags_query: Select) -> Select:
+            # Hide DAGs if not showing status: "all"
+            if status == "active":
+                dags_query = dags_query.where(~DagModel.is_paused)
+            elif status == "paused":
+                dags_query = dags_query.where(DagModel.is_paused)
+            return dags_query.where(
+                ~DagModel.is_subdag, DagModel.is_active, DagModel.dag_id.in_(filter_dag_ids)
             )
-            .distinct()
-            .where(~DagModel.is_subdag, DagModel.is_active, DagModel.owners.ilike(f"%{query}%"))
-        )
 
-        # Hide DAGs if not showing status: "all"
-        status = flask_session.get(FILTER_STATUS_COOKIE)
-        if status == "active":
-            dag_ids_query = dag_ids_query.where(~DagModel.is_paused)
-            owners_query = owners_query.where(~DagModel.is_paused)
-        elif status == "paused":
-            dag_ids_query = dag_ids_query.where(DagModel.is_paused)
-            owners_query = owners_query.where(DagModel.is_paused)
+        if query_prefix.startswith("task:"):
+            # Provide suggestions of task_ids
+            dags_query = _filter_dags_query(dags_query=select(DagModel.dag_id))
+            filtered_dag_ids = session.scalars(dags_query).all()
+            dag_bag = get_airflow_app().dag_bag
+            filtered_dags = [dag_bag.get_dag(dag_id, session=session) for dag_id in filtered_dag_ids]

Review Comment:
   Do you have specific DAGs loaded? Which chars are you entering? Can you post the stack trace?
   I tested locally on my side and for me it looks good.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org