You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@beam.apache.org by "cipheraxat (via GitHub)" <gi...@apache.org> on 2025/10/30 16:03:30 UTC

[PR] [BEAM YAML] Add documentation and examples for SQL calcite_connection… [beam]

cipheraxat opened a new pull request, #36684:
URL: https://github.com/apache/beam/pull/36684

   ### Issue Components
   
   - [x] Component: Beam YAML
   
   ### Changes Made
   
   1. **Added example YAML pipeline** in \`sdks/python/apache_beam/yaml/examples/transforms/sql/calcite_connection_properties.yaml\`
      - Shows how to provide \`calcite_connection_properties\` under the top-level \`options:\` key
      - Demonstrates both YAML mapping format (preferred) and JSON string format (for compatibility)
      - Includes a working SQL transform example using PostgreSQL functions
   
   2. **Updated YAML docs generator** in \`sdks/python/apache_beam/yaml/generate_yaml_docs.py\`
      - Added special handling for the SQL transform to include a callout about calcite connection properties
      - The generated transform catalog page will now include clear documentation and examples
      - Shows both approaches for providing connection properties
   
   ### Example Usage
   
   **Preferred YAML mapping approach:**
   \`\`\`yaml
   options:
     calcite_connection_properties:
       fun: postgresql
   \`\`\`
   
   **Alternative JSON string approach:**
   \`\`\`yaml
   options:
     calcite_connection_properties: '{\"fun\": \"postgresql\"}'
   \`\`\`
   
   ### Testing
   
   - Syntax validation passed for the modified Python generator
   - Example YAML follows existing patterns in the examples directory
   - Changes are backwards compatible and don't affect existing functionality


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [BEAM YAML] Add documentation and examples for SQL calcite_connection… [beam]

Posted by "mohamedawnallah (via GitHub)" <gi...@apache.org>.

mohamedawnallah commented on PR #36684:
URL: https://github.com/apache/beam/pull/36684#issuecomment-3468915178

   @cipheraxat – Just curious what Beam issue this PR closes?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [BEAM YAML] Add documentation and examples for SQL calcite_connection… [beam]

Posted by "mohamedawnallah (via GitHub)" <gi...@apache.org>.

mohamedawnallah commented on PR #36684:
URL: https://github.com/apache/beam/pull/36684#issuecomment-3468951389

   And seems formatting workflows complains. This can be done by following the `CONTRIBUTING.md` guide which references the following:
   https://cwiki.apache.org/confluence/display/BEAM/Python+Tips#PythonTips-LintandFormattingChecks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [BEAM YAML] Add documentation and examples for SQL calcite_connection… [beam]

Posted by "mohamedawnallah (via GitHub)" <gi...@apache.org>.

mohamedawnallah commented on PR #36684:
URL: https://github.com/apache/beam/pull/36684#issuecomment-3468940144

   > @mohamedawnallah - This is the issue : #36614
   
   Let's add in the PR description "Closes #36614". That way the reviewers and people seeing PR becomes clear what it closes. Also it would be nice for that issue to be automatically closed when that PR is merged


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add documentation and examples for SQL calcite_connection… [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] commented on PR #36684:
URL: https://github.com/apache/beam/pull/36684#issuecomment-3699253441

   This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add documentation and examples for SQL calcite_connection… [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] commented on PR #36684:
URL: https://github.com/apache/beam/pull/36684#issuecomment-3469753518

   Assigning reviewers:
   
   R: @jrmccluskey for label python.
   
   
   Note: If you would like to opt out of this review, comment `assign to next reviewer`.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add documentation and examples for SQL calcite_connection… [beam]

Posted by "mohamedawnallah (via GitHub)" <gi...@apache.org>.

mohamedawnallah commented on PR #36684:
URL: https://github.com/apache/beam/pull/36684#issuecomment-3469788160

   On initial glance, It looks like we need to register the example YAML file name as processor in `examples_test.py` perhaps seeing `test_sqlserver_to_bigquery_yaml` on how the file is created and registered in `examples_test.py`: 
   
   **Where It is Created:**
   
   https://github.com/apache/beam/blob/1e51a886f5cb2904a4c0b331274f328692291fcd/sdks/python/apache_beam/yaml/examples/transforms/blueprint/sqlserver_to_bigquery.yaml#L18-L58
   
   **Where It is Registered:**
   
   https://github.com/apache/beam/blob/1e51a886f5cb2904a4c0b331274f328692291fcd/sdks/python/apache_beam/yaml/examples/testing/examples_test.py#L915-L926
   
   https://github.com/apache/beam/blob/1e51a886f5cb2904a4c0b331274f328692291fcd/sdks/python/apache_beam/yaml/examples/testing/examples_test.py#L669
   
   Relatedly:
   https://github.com/apache/beam/tree/master/sdks/python/apache_beam/yaml/examples#testing


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [BEAM YAML] Add documentation and examples for SQL calcite_connection… [beam]

Posted by "mohamedawnallah (via GitHub)" <gi...@apache.org>.

mohamedawnallah commented on PR #36684:
URL: https://github.com/apache/beam/pull/36684#issuecomment-3468954442

   It would be nice as well to state how someone's supposed to test this (any unit/integration tests) or if it is tested automatically in the CI?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [BEAM YAML] Add documentation and examples for SQL calcite_connection… [beam]

Posted by "cipheraxat (via GitHub)" <gi...@apache.org>.

cipheraxat commented on PR #36684:
URL: https://github.com/apache/beam/pull/36684#issuecomment-3468932937

   @mohamedawnallah - This is the issue :  https://github.com/apache/beam/issues/36614


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add documentation and examples for SQL calcite_connection… [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] commented on PR #36684:
URL: https://github.com/apache/beam/pull/36684#issuecomment-3469240548

   Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add documentation and examples for SQL calcite_connection… [beam]

Posted by "mohamedawnallah (via GitHub)" <gi...@apache.org>.

mohamedawnallah commented on PR #36684:
URL: https://github.com/apache/beam/pull/36684#issuecomment-3469756009

   waiting on author


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [BEAM YAML] Add documentation and examples for SQL calcite_connection… [beam]

Posted by "mohamedawnallah (via GitHub)" <gi...@apache.org>.

mohamedawnallah commented on PR #36684:
URL: https://github.com/apache/beam/pull/36684#issuecomment-3468968430

   > It would be nice as well to state how someone's supposed to test this (any unit/integration tests) and/or if it is tested automatically in the CI?
   
   Let me know if you need any helping hands regards that


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [BEAM YAML] Add documentation and examples for SQL calcite_connection… [beam]

Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.

gemini-code-assist[bot] commented on PR #36684:
URL: https://github.com/apache/beam/pull/36684#issuecomment-3468772279

   ## Summary of Changes
   
   Hello @cipheraxat, I'm Gemini Code Assist[^1]! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
   
   This pull request improves the usability of Beam YAML for SQL transforms by adding comprehensive documentation and an example for "calcite_connection_properties". This ensures users can easily configure dialect-specific SQL functions, with clear guidance on using both YAML mapping and JSON string formats for these properties, thereby enhancing pipeline flexibility and clarity.
   
   ### Highlights
   
   * **New Example Pipeline**: Introduced a YAML pipeline example demonstrating how to configure "calcite_connection_properties" for SQL transforms, showcasing both YAML mapping and JSON string formats.
   * **Documentation Enhancement**: Modified the YAML documentation generator to automatically include a specific callout for the SQL transform, detailing how to provide "calcite_connection_properties" in the generated docs.
   
   
   
   
   
   
   
   <details>
   <summary><b>Using Gemini Code Assist</b></summary>
   <br>
   
   The full guide for Gemini Code Assist can be found on our [documentation page](https://developers.google.com/gemini-code-assist/docs/review-github-code), here are some quick tips.
   
   <b>Invoking Gemini</b>
   
   You can request assistance from Gemini at any point by creating a comment using either `/gemini <command>` or `@gemini-code-assist <command>`. Below is a summary of the supported commands on the current page.
   
   Feature | Command | Description
   --- | --- | ---
   Code Review | `/gemini review` | Performs a code review for the current pull request in its current state.
   Pull Request Summary | `/gemini summary` | Provides a summary of the current pull request in its current state.
   Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in pull request comments and review comments.
   Help | `/gemini help` | Displays a list of available commands.
   
   <b>Customization</b>
   
   To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a `.gemini/` folder in the base of the repository. Detailed instructions can be found [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github).
   
   <b>Limitations & Feedback</b>
   
   Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with :thumbsup: and :thumbsdown: on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up [here](https://google.qualtrics.com/jfe/form/SV_2cyuGuTWsEw84yG).
   
   <b>You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the [Gemini Code Assist IDE Extension](https://cloud.google.com/products/gemini/code-assist).</b>
   </details>
   
   
   
   
   [^1]: Review the [Privacy Notices](https://policies.google.com/privacy), [Generative AI Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy), [Terms of Service](https://policies.google.com/terms), and learn how to configure Gemini Code Assist in GitHub [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github). Gemini can make mistakes, so double check it and [use code with caution](https://support.google.com/legal/answer/13505487).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Closes #36614 : Add documentation and examples for SQL calcite_connection… [beam]

Posted by "cipheraxat (via GitHub)" <gi...@apache.org>.

cipheraxat commented on PR #36684:
URL: https://github.com/apache/beam/pull/36684#issuecomment-3469009950

   Sure @mohamedawnallah, let me check through formatting issues. I will reach out if I am stuck anywhere.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add documentation and examples for SQL calcite_connection… [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] commented on PR #36684:
URL: https://github.com/apache/beam/pull/36684#issuecomment-3714578213

   This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add documentation and examples for SQL calcite_connection… [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] closed pull request #36684: Add documentation and examples for SQL calcite_connection…
URL: https://github.com/apache/beam/pull/36684


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org