You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Jefffrey (via GitHub)" <gi...@apache.org> on 2023/11/12 00:11:48 UTC

[PR] Allow writing null valued keys in JSON [arrow-rs]

Jefffrey opened a new pull request, #5065:
URL: https://github.com/apache/arrow-rs/pull/5065

   # Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   Closes #3774
   
   # Rationale for this change
    
   <!--
   Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
   Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
   -->
   
   Allow users to write JSON with null values for keys, where previous (and default) is to skip the keys.
   
   # What changes are included in this PR?
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   In arrow-json writer mod, new type parameter for `Writer` to configure how null valued keys (and null arrays) should be handled. Will default to SkipNulls for backward compatibility.
   
   # Are there any user-facing changes?
   
   Yes
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!---
   If there are any breaking changes to public APIs, please add the `breaking change` label.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow writing null valued keys in JSON [arrow-rs]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #5065:
URL: https://github.com/apache/arrow-rs/pull/5065#issuecomment-1819762082

   > This looks good to me, my only comment is I think it should be `keep_null_values` or possibly `preserve_nulls`, as null keys are actually illegal in JSON 😅
   
   Yeah I wasn't sure on the proper name. I was gonna leave it as `keep_nulls` but I felt that could be confusing since it might imply affecting null values in arrays which should always be kept regardless.
   
   Maybe something like `preserve_object_nulls`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow writing null valued keys in JSON [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #5065:
URL: https://github.com/apache/arrow-rs/pull/5065#issuecomment-1819770043

   Perhaps explicit_nulls ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow writing null valued keys in JSON [arrow-rs]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #5065:
URL: https://github.com/apache/arrow-rs/pull/5065#discussion_r1399064293


##########
arrow-json/src/writer.rs:
##########
@@ -494,21 +575,36 @@ impl JsonFormat for JsonArray {
     }
 }
 
-/// A JSON writer which serializes [`RecordBatch`]es to newline delimited JSON objects
-pub type LineDelimitedWriter<W> = Writer<W, LineDelimited>;
+/// A JSON writer which serializes [`RecordBatch`]es to newline delimited JSON objects.
+///
+/// Will skip writing keys with null values.
+pub type LineDelimitedWriter<W> = Writer<W, LineDelimited, SkipNulls>;
 
-/// A JSON writer which serializes [`RecordBatch`]es to JSON arrays
-pub type ArrayWriter<W> = Writer<W, JsonArray>;
+/// Similar to [`LineDelimitedWriter`] but will keep keys with null values.
+pub type LineDelimitedWriterWithNulls<W> = Writer<W, LineDelimited, KeepNulls>;
+
+/// A JSON writer which serializes [`RecordBatch`]es to JSON arrays.
+///
+/// Will skip writing keys with null values.
+pub type ArrayWriter<W> = Writer<W, JsonArray, SkipNulls>;
+
+/// Similar to [`ArrayWriter`] but will keep keys with null values.
+pub type ArrayWriterWithNulls<W> = Writer<W, JsonArray, KeepNulls>;
 
 /// A JSON writer which serializes [`RecordBatch`]es to a stream of
-/// `u8` encoded JSON objects. See the module level documentation for
-/// detailed usage and examples. The specific format of the stream is
-/// controlled by the [`JsonFormat`] type parameter.
+/// `u8` encoded JSON objects.
+///
+/// See the module level documentation for detailed usage and examples.
+/// The specific format of the stream is controlled by the [`JsonFormat`]
+/// type parameter. The [`NullHandler`] type parameter controls whether
+/// nulls should be written explicitly for keys or skipped. Default is
+/// [`SkipNulls`] for backward compatibility.
 #[derive(Debug)]
-pub struct Writer<W, F>
+pub struct Writer<W, F, N = SkipNulls>
 where
     W: Write,
     F: JsonFormat,
+    N: NullHandler,

Review Comment:
   Changed to be configured via a WriterBuilder now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow writing null valued keys in JSON [arrow-rs]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #5065:
URL: https://github.com/apache/arrow-rs/pull/5065#issuecomment-1819786453

   > Perhaps explicit_nulls ?
   
   Sounds better, renamed to that :+1: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow writing null valued keys in JSON [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold merged PR #5065:
URL: https://github.com/apache/arrow-rs/pull/5065


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow writing null valued keys in JSON [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #5065:
URL: https://github.com/apache/arrow-rs/pull/5065#issuecomment-1807877553

   I could be mistaken but i thought adding new parameters was always a breaking change. Regardless using generics here will result in additional codegen and I'm doubtful will improve performance. Adding a WriterOptions or similar should be more future proof and more consistent with other implementations


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow writing null valued keys in JSON [arrow-rs]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #5065:
URL: https://github.com/apache/arrow-rs/pull/5065#issuecomment-1807246716

   > As commented on the ticket, the current behaviour is simply incorrect as it will not survive a round-trip. I therefore don't think there is a need to make the prior logic configurable, which I think should simplify this PR
   
   If you're referring to #5066 then that is a separate issue which this PR doesn't cover.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow writing null valued keys in JSON [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #5065:
URL: https://github.com/apache/arrow-rs/pull/5065#issuecomment-1807247645

   Aah I see, in that case is there a way we could make this an option instead of a generic. As it stands this is a breaking change, which I'm not sure about


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow writing null valued keys in JSON [arrow-rs]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #5065:
URL: https://github.com/apache/arrow-rs/pull/5065#issuecomment-1807949206

   > Regardless using generics here will result in additional codegen and I'm doubtful will improve performance. Adding a WriterOptions or similar should be more future proof and more consistent with other implementations
   
   Yeah that makes 👍
   
   I'll work on switching to such


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow writing null valued keys in JSON [arrow-rs]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #5065:
URL: https://github.com/apache/arrow-rs/pull/5065#issuecomment-1807711118

   Can you give an example where this might break existing uses? I thought including a default for the new generic parameter for `Writer` would make this backwards compatible,  but would be happy to be proven wrong.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow writing null valued keys in JSON [arrow-rs]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #5065:
URL: https://github.com/apache/arrow-rs/pull/5065#discussion_r1390304461


##########
arrow-json/src/writer.rs:
##########
@@ -494,21 +575,36 @@ impl JsonFormat for JsonArray {
     }
 }
 
-/// A JSON writer which serializes [`RecordBatch`]es to newline delimited JSON objects
-pub type LineDelimitedWriter<W> = Writer<W, LineDelimited>;
+/// A JSON writer which serializes [`RecordBatch`]es to newline delimited JSON objects.
+///
+/// Will skip writing keys with null values.
+pub type LineDelimitedWriter<W> = Writer<W, LineDelimited, SkipNulls>;
 
-/// A JSON writer which serializes [`RecordBatch`]es to JSON arrays
-pub type ArrayWriter<W> = Writer<W, JsonArray>;
+/// Similar to [`LineDelimitedWriter`] but will keep keys with null values.
+pub type LineDelimitedWriterWithNulls<W> = Writer<W, LineDelimited, KeepNulls>;
+
+/// A JSON writer which serializes [`RecordBatch`]es to JSON arrays.
+///
+/// Will skip writing keys with null values.
+pub type ArrayWriter<W> = Writer<W, JsonArray, SkipNulls>;
+
+/// Similar to [`ArrayWriter`] but will keep keys with null values.
+pub type ArrayWriterWithNulls<W> = Writer<W, JsonArray, KeepNulls>;
 
 /// A JSON writer which serializes [`RecordBatch`]es to a stream of
-/// `u8` encoded JSON objects. See the module level documentation for
-/// detailed usage and examples. The specific format of the stream is
-/// controlled by the [`JsonFormat`] type parameter.
+/// `u8` encoded JSON objects.
+///
+/// See the module level documentation for detailed usage and examples.
+/// The specific format of the stream is controlled by the [`JsonFormat`]
+/// type parameter. The [`NullHandler`] type parameter controls whether
+/// nulls should be written explicitly for keys or skipped. Default is
+/// [`SkipNulls`] for backward compatibility.
 #[derive(Debug)]
-pub struct Writer<W, F>
+pub struct Writer<W, F, N = SkipNulls>
 where
     W: Write,
     F: JsonFormat,
+    N: NullHandler,

Review Comment:
   I wasn't sure if it would be better to include this null handling logic in `JsonFormat` instead. Open to suggestions



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow writing null valued keys in JSON [arrow-rs]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #5065:
URL: https://github.com/apache/arrow-rs/pull/5065#issuecomment-1818875831

   Removed configuring this null behaviour as a generic trait, and changed to be a config that can be set via a new WriterBuilder (akin to arrow-csv writing behaviour)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow writing null valued keys in JSON [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #5065:
URL: https://github.com/apache/arrow-rs/pull/5065#issuecomment-1807218117

   As commented on the ticket, the current behaviour is simply incorrect as it will not survive a round-trip. I therefore don't think there is a need to make the prior logic configurable, which I think should simplify this PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org