Are metadata columns required to get declared in the table's schema?

[Jie Han <tu...@gmail.com>]

Hi community, I want to query a metadata column from my table t. Do I need to declare it in the table schema explicitly? 

In spark, metadata columns are hidden columns, which means we don’t need to declare it in the table ddl, we only explicitly reference it in our query. For instance, select *, _metadata from t.

Re: Are metadata columns required to get declared in the table's schema?

[Hang Ruan <ru...@gmail.com>]

Hi, Jie,

If you don't need these metadata columns, you don't need to declare them
for the table. Then metadata columns will not be read from sources and will
not be written into the sink.
You can query a table that is without the metadata column declaration. It
depends on your requests.

Best,
Hang

Jie Han <tu...@gmail.com> 于2023年3月26日周日 21:42写道：

> Thank you for your respond.
> Actually I noticed that the doc says 'However, declaring a metadata column
> in a table’s schema is optional’.
> So, does it mean that we don’t need to declare it when we don't query it
> rather than we can query it without the declaration?
>
> Best,
> Jay
>
>
>
>
>
>
>

Re: Are metadata columns required to get declared in the table's schema?

[Jie Han <tu...@gmail.com>]

Thank you for your respond.
Actually I noticed that the doc says 'However, declaring a metadata column in a table’s schema is optional’.
So, does it mean that we don’t need to declare it when we don't query it rather than we can query it without the declaration?

Best,
Jay

Re: Are metadata columns required to get declared in the table's schema?

[Hang Ruan <ru...@gmail.com>]

ps : DDL I said is the CREATE TABLE statements.

Best,
Hang

Hang Ruan <ru...@gmail.com> 于2023年3月26日周日 21:33写道：

> Hi, Jie,
>
> In Flink, if we want to access a metadata column, we need to declare it in
> the DDL.
> More details could be found here[1].
>
> Best,
> Hang
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/create/#columns
>
> Jie Han <tu...@gmail.com> 于2023年3月26日周日 14:58写道：
>
>> Hi community, I want to query a metadata column from my table t. Do I
>> need to declare it in the table schema explicitly?
>>
>> In spark, metadata columns are *hidden* columns, which means we don’t
>> need to declare it in the table ddl, we only explicitly reference it in our
>> query. For instance, select *, _metadata from t.
>>
>>
>>

Re: Are metadata columns required to get declared in the table's schema?

[Hang Ruan <ru...@gmail.com>]

Hi, Jie,

In Flink, if we want to access a metadata column, we need to declare it in
the DDL.
More details could be found here[1].

Best,
Hang

[1]
https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/create/#columns

Jie Han <tu...@gmail.com> 于2023年3月26日周日 14:58写道：

> Hi community, I want to query a metadata column from my table t. Do I
> need to declare it in the table schema explicitly?
>
> In spark, metadata columns are *hidden* columns, which means we don’t
> need to declare it in the table ddl, we only explicitly reference it in our
> query. For instance, select *, _metadata from t.
>
>
>