Re: [PR] Add partition_key and partition_key_pattern filters for asset events [airflow]

["hussein-awala (via GitHub)" <gi...@apache.org>]

hussein-awala commented on PR #64610:
URL: https://github.com/apache/airflow/pull/64610#issuecomment-4657453435

   Sorry for the late reply @pierrejeambrun, and thanks for the review.
   
   You raise a fair point about consistency, and I agree the regex introduction deserves a dedicated discussion. Let me explain the concrete gap so we can decide together.
   
   The existing `pattern` (substring `ILIKE '%term%'`) and `prefix_pattern` (prefix range scan) filters are great for free-text columns like `dag_id`, but they fall short for **partition keys**, which are typically *structured composite strings* (e.g. `us|2024-01-15`, `region=us/date=2024-01-15`). A few queries that are common for partition keys and that `pattern`/`prefix_pattern` cannot express:
   
   **1. The `|` delimiter collides with the OR operator**
   
   Both filters treat `|` as logical OR (`val_str.split("|")`). But partition keys very frequently use `|` as a field separator (see this PR's own examples like `us|2026-03-10`). So:
   - `prefix_pattern=us|2024-01-15` is parsed as `OR(prefix "us", prefix "2024-01-15")` — not a match on the literal key.
   - With regex you escape it: `partition_key_pattern=^us\|2024-01-15$`.
   
   **2. Suffix / "ends with" matching**
   
   Find all partitions for a given date regardless of the region prefix:
   - regex: `partition_key_pattern=\|2024-01-15$`
   - `pattern` can only do unanchored substring (`%2024-01-15%`), which also matches `2024-01-15T09:00`; `prefix_pattern` only matches from the start. Neither can anchor to the end.
   
   **3. Character-class / format constraints**
   
   Match only well-formed date partitions for a region:
   - regex: `partition_key_pattern=^us\|\d{4}-\d{2}-\d{2}$`
   - LIKE's `_` matches *any* character, so `us|2024-__-__` would also match `us|2024-ab-cd`. There's no way to require digits.
   
   **4. Structured mid-string alternation**
   
   Match either region, but only for a specific month:
   - regex: `partition_key_pattern=^(us|eu)\|2024-03`
   - The `|` OR in `pattern`/`prefix_pattern` splits the whole term, so it can't express "us OR eu, each followed by `|2024-03`".


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org