You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "obarisk (via GitHub)" <gi...@apache.org> on 2024/02/23 15:01:45 UTC

[I] PostgresToGCSOperator with server_side_cursor use unique_name that breaks postgres identifier limit [airflow]

obarisk opened a new issue, #37654:
URL: https://github.com/apache/airflow/issues/37654

   ### Apache Airflow Provider(s)
   
   google
   
   ### Versions of Apache Airflow Providers
   
   <= 10.15.0
   
   ### Apache Airflow version
   
   2.8.1
   
   ### Operating System
   
   Debian GNU/Linux 12
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   _No response_
   
   ### What happened
   
   when total characters of dag_id and task_id greater than 24. (64 - 36 - 4)
   PostgresToGCSOperator with argument `use_server_side_cursor = True` run into psycopg2 error
   > psycopg2.errors.SyntaxError: unterminated quoted identifier
   
   ### What you think should happen instead
   
   possible expectations
   
   - Add a configuration options for the limitation of postgres identifier length. (default = 64)
   - Detect postgres identifier length before execution. Error out as earier as possible
   - Shorten the `_unique_name` generation for the cursor, currently
     ```
     def _unique_name(self):
       return f"{self.dag_id}__{self.task_id}__{uuid.uuid4()}" if self.use_server_side_cursor else None
     ```
   
   ### How to reproduce
   
   ```python
   from airflow.providers.google.cloud.transfers.gcs_to_bigquery import GCSToBigQueryOperator
   
   task = PostgresToGCSOperator(
     task_id="t" * 25,
     use_server_side_cursor=True,
     doc_md=f"""postgres identifier overflow""",
   )
   ```
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] PostgresToGCSOperator with server_side_cursor use unique_name that breaks postgres identifier limit [airflow]

Posted by "dybolo (via GitHub)" <gi...@apache.org>.
dybolo commented on issue #37654:
URL: https://github.com/apache/airflow/issues/37654#issuecomment-1977591590

   Hello,
   Can I try to fix this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] PostgresToGCSOperator with server_side_cursor use unique_name that breaks postgres identifier limit [airflow]

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis closed issue #37654: PostgresToGCSOperator with server_side_cursor use unique_name that breaks postgres identifier limit
URL: https://github.com/apache/airflow/issues/37654


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] PostgresToGCSOperator with server_side_cursor use unique_name that breaks postgres identifier limit [airflow]

Posted by "obarisk (via GitHub)" <gi...@apache.org>.
obarisk commented on issue #37654:
URL: https://github.com/apache/airflow/issues/37654#issuecomment-1977611687

   yes. please. I've heaivy workloads currently. But please share the fix with me. I'd love to learn more about the development process. @dybolo 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] PostgresToGCSOperator with server_side_cursor use unique_name that breaks postgres identifier limit [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #37654:
URL: https://github.com/apache/airflow/issues/37654#issuecomment-1961488638

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] PostgresToGCSOperator with server_side_cursor use unique_name that breaks postgres identifier limit [airflow]

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #37654:
URL: https://github.com/apache/airflow/issues/37654#issuecomment-1970129857

   I guess it might be use something like hash instead, md5, sha1 or similar.
   
   E.g. n Amazon Provider exists simple helper for generate deterministic identifiers: 
   https://github.com/apache/airflow/blob/main/airflow/providers/amazon/aws/utils/identifiers.py
   
   https://github.com/apache/airflow/blob/390490fdd4e72960a7cdac2dd09ebd503a8e30df/airflow/providers/amazon/aws/operators/ecs.py#L528-L531
   
   If it shouldn't be a deterministic it could use [`slugify`](https://github.com/un33k/python-slugify) to truncate and fit to a limits
   
   Pretty much different solutions for someone who might want to fix it.
   
   BTW, postgres identifiers limited by 63 bytes, not 64 characters: https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org