You are viewing a plain text version of this content. The canonical link for it is here.

Posted to notifications@hamilton.apache.org by "AvinashYerra (via GitHub)" <gi...@apache.org> on 2026/01/03 17:04:08 UTC

[PR] Add NDJSON support in polars [hamilton]

AvinashYerra opened a new pull request, #1431:
URL: https://github.com/apache/hamilton/pull/1431

   This PR implements NDJSON (newline-delimited JSON) reader and writer support for Polars, following the same pattern as the existing JSON reader/writer implementation.
   
   ## Changes
   - Added PolarsNDJSONReader class in hamilton/plugins/polars_post_1_0_0_extensions.py
   - Added PolarsNDJSONWriter class in hamilton/plugins/polars_post_1_0_0_extensions.py
   - Supports writing NDJSON files using data.write_ndjson()
   
   - Registered both classes in `register_data_loaders()` function to make them available through the materialization system
   - Added the following tests:
     - test_polars_ndjson() in tests/plugins/test_polars_extensions.py for DataFrame support
     - test_polars_ndjson() in tests/plugins/test_polars_lazyframe_extensions.py for LazyFrame support
   - Added example in examples/polars/materialization/my_script.py
   
   ## How I tested this
   Ran the specific NDJSON tests:
   - pytest tests/plugins/test_polars_extensions.py::test_polars_ndjson -v
   - pytest tests/plugins/test_polars_lazyframe_extensions.py::test_polars_ndjson -v
   Both tests passed
   
   ## Notes
   
   ## Checklist
   
   - [Y] PR has an informative and human-readable title (this will be pulled into the release notes)
   - [ ] Changes are limited to a single goal (no scope creep)
   - [Y] Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
   - [ ] Any _change_ in functionality is tested
   - [ ] New functions are documented (with a description, list of inputs, and expected output)
   - [ ] Placeholder code is flagged / future TODOs are captured in comments
   - [ ] Project documentation has been updated if adding/changing functionality.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@hamilton.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add NDJSON support in polars [hamilton]

Posted by "skrawcz (via GitHub)" <gi...@apache.org>.

skrawcz merged PR #1431:
URL: https://github.com/apache/hamilton/pull/1431


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@hamilton.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add NDJSON support in polars [hamilton]

Posted by "skrawcz (via GitHub)" <gi...@apache.org>.

skrawcz commented on PR #1431:
URL: https://github.com/apache/hamilton/pull/1431#issuecomment-3710130822

   @AvinashYerra would you mind rebasing against the latest hamilton main? I should have merged a fix for a bunch of unit tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@hamilton.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add NDJSON support in polars [hamilton]

Posted by "skrawcz (via GitHub)" <gi...@apache.org>.

skrawcz commented on PR #1431:
URL: https://github.com/apache/hamilton/pull/1431#issuecomment-3722068647

   Strange. You can see that I merged https://github.com/apache/hamilton/pull/1437 ? So it's on main. Do you have that commit?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@hamilton.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org