[Python] Is there a way to specify a column as non-nullable with parquet.write_table?

[Tim Swast <sw...@google.com.INVALID>]

I'm currently using parquet as the intermediate format when uploading a
pandas DataFrame to Google BigQuery. We encounter a problem when trying to
append a parquet file to a table with required fields (issue:
https://github.com/googleapis/google-cloud-python/issues/8093).

Is there a way to mark fields as required / non-nullable in parquet files?
If there is, is there a way to set that option with
pyarrow.parquet.write_table?

https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html#pyarrow.parquet.write_table

*  •  **Tim Swast*
*  •  *Software Friendliness Engineer
*  •  *Google Cloud Developer Relations
*  •  *Seattle, WA, USA

Re: [Python] Is there a way to specify a column as non-nullable with parquet.write_table?

[Wes McKinney <we...@gmail.com>]

Yes, but you will need to resolve

https://issues.apache.org/jira/browse/ARROW-5169

write_table should respect the field-level nullability in the schema
of the Table you pass

On Thu, May 23, 2019 at 10:34 AM Tim Swast <sw...@google.com.invalid> wrote:
>
> I'm currently using parquet as the intermediate format when uploading a
> pandas DataFrame to Google BigQuery. We encounter a problem when trying to
> append a parquet file to a table with required fields (issue:
> https://github.com/googleapis/google-cloud-python/issues/8093).
>
> Is there a way to mark fields as required / non-nullable in parquet files?
> If there is, is there a way to set that option with
> pyarrow.parquet.write_table?
>
> https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html#pyarrow.parquet.write_table
>
> *  •  **Tim Swast*
> *  •  *Software Friendliness Engineer
> *  •  *Google Cloud Developer Relations
> *  •  *Seattle, WA, USA