You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datasketches.apache.org by GitBox <gi...@apache.org> on 2021/12/03 04:57:44 UTC
[GitHub] [datasketches-java] kfaraz commented on pull request #375: Add a nullSafe mode in ArrayOfStringsSerDe
kfaraz commented on pull request #375:
URL: https://github.com/apache/datasketches-java/pull/375#issuecomment-985217093
Thanks for the feedback, @leerho !
> I assume from the discussion in the referenced Druid PR, that the underlying sketch is a quantiles sketch. Is this correct?
That's correct, the underlying sketch is a quantiles `ItemsSketch`.
> Given an array of strings, why is a null string element interpreted differently from an empty string element?
Could null be made equivalent to empty? For example, an empty string be substituted for every occurrence of a null element? This would eliminate the need for a magic null symbol, which does require changing the sketch code for all sketches that might use that class.
This would not be desirable as in most of our datasets, `null` typically signifies "No Data" whereas an empty String would be valid data explicitly set to empty by a user. There are often queries just to find out rows which have `null` values (i.e. "No Data") for a particular column.
> Could a null be converted to a magic string, specified by the user and based on the use case?
This could be done but if the user needs to specify this, it would perhaps be an additional cognitive burden. If hardcoded into the system as a specific String, there would still be a little chance of that string occurring in a dataset.
If you believe there is a string we can use for this purpose, would it make sense to do the conversion from `null` to the magic string and vice-versa in `ArrayOfStringsSerDe` itself?
> This PR would require changing sketch code of all these sketches to interpret nulls differently and either allow or disallow them, depending on the sketch and the use case, which increases the complexity of these sketches.
None of the Java classes would have to be changed. But I am not sure about how the C++ compat would be affected. I have added both a nullSafe and a non nullSafe mode in the modified classes such that the existing sketch classes can continue to rely on the non nullSafe behaviour without any change at all. They would have to handle nulls only if they _explicitly_ start constructing the SerDe as `new ArrayOfStringsSerDe(true)` instead of the existing `new ArrayOfStringsSerDe()`
Code Snippet:
```java
private final boolean nullSafe;
/**
* Creates an instance of {@code ArrayOfStringsSerDe} which does not handle
* null values.
*/
public ArrayOfStringsSerDe() {
this(false);
}
/**
* Creates an instance of {@code ArrayOfStringsSerDe}.
*
* @param nullSafe true if null values should be serialized/deserialized safely.
*/
public ArrayOfStringsSerDe(boolean nullSafe) {
this.nullSafe = nullSafe;
}
```
Please let me know if this would be insufficient and if it would be better to separate the nullSafe mode into a different class altogether.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org