Map UDF : The Nothing type cannot have a serializer

[shkob1 <sh...@gmail.com>]

Hey,

As im building a SQL query, im trying to conditionally build a map such that
there won't be any keys with null values in it. AFAIK from Calcite there's
no native way to do it (other than using case to build the map in different
ways, but then i have a lot of key/value pairs so thats not reasonable). I
tried to implement it using a flink UDF but failed - i wonder if a table
function can return a map since im getting "The Nothing type cannot have a
serializer".

Example SQL:

Select a, b, c, removeNulls('key1', d, 'key2', e, 'key3', f) AS my_map
FROM... 

My Table Function code is:

public class RemoveNullValuesFunction extends
TableFunction<Map&lt;String,String>> {

    public static final String NAME = "removeNulls";

    public void eval(String... keyValues) {

        if (keyValues.length == 0) {
            collect(Collections.emptyMap());
            return;
        }
        final List<String> keyValuesList = Arrays.asList(keyValues);
        Map<String,String> output = Maps.newHashMap();
        for (int i = 0; i < keyValuesList.size(); i = i + 2){
            final String key = keyValuesList.get(i);
            final String value = keyValuesList.get(i + 1);
            if (value != null)
                output.put(key, value);
        }
        collect(output);

    }

    @Override
    public TypeInformation<Map&lt;String, String>> getResultType() {
        return Types.MAP(Types.STRING(),Types.STRING());
    }
}


I wonder if thats not possible or am i missing something in how the
serialization works?

Thanks!
Shahar







--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Map UDF : The Nothing type cannot have a serializer

[Shahar Cizer Kobrinsky <sh...@gmail.com>]

Thanks Rong,

Yes for some reason i thought i need a table function, but scalar works!
Yes the map constructor is what i started with but then figured it doesn't
support conditional entries.

On Thu, Mar 21, 2019 at 6:07 PM Rong Rong <wa...@gmail.com> wrote:

> Based on what I saw in the implementation, I think you meant to implement
> a ScalarFunction right? since you are only trying to structure a VarArg
> string into a Map.
>
> If my understanding was correct. I think the Map constructor[1] is
> something you might be able to leverage. It doesn't resolve your
> Nullability issue though.
> Otherwise you can use the Scalar UDF [2]
>
> --
> Rong
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/table/functions.html#value-construction-functions
> [2]
> https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/table/udfs.html#scalar-functions
>
>
>
> On Thu, Mar 21, 2019 at 5:02 PM shkob1 <sh...@gmail.com> wrote:
>
>> Looking further into the RowType it seems like this field is translated
>> as a
>> CURSOR rather than a map.. not sure why
>>
>>
>>
>> --
>> Sent from:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>>
>

Re: Map UDF : The Nothing type cannot have a serializer

[Rong Rong <wa...@gmail.com>]

Based on what I saw in the implementation, I think you meant to implement a
ScalarFunction right? since you are only trying to structure a VarArg
string into a Map.

If my understanding was correct. I think the Map constructor[1] is
something you might be able to leverage. It doesn't resolve your
Nullability issue though.
Otherwise you can use the Scalar UDF [2]

--
Rong

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/table/functions.html#value-construction-functions
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/table/udfs.html#scalar-functions



On Thu, Mar 21, 2019 at 5:02 PM shkob1 <sh...@gmail.com> wrote:

> Looking further into the RowType it seems like this field is translated as
> a
> CURSOR rather than a map.. not sure why
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>

Re: Map UDF : The Nothing type cannot have a serializer

[shkob1 <sh...@gmail.com>]

Looking further into the RowType it seems like this field is translated as a
CURSOR rather than a map.. not sure why



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/