You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Killian (Jira)" <ji...@apache.org> on 2021/04/18 06:05:00 UTC
[jira] [Updated] (SPARK-35123) read partitionned parquet:
my_col=NOW replaced by on read()
[ https://issues.apache.org/jira/browse/SPARK-35123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Killian updated SPARK-35123:
----------------------------
Description:
When reading parquet file partitioned with a column containing the value "NOW", The value is interpreted as now() and replaced by the current time at the moment of the read() funct is executed
{code:java}
// step to reproduce
df = spark.createDataFrame(data=["NOW", "TEST"], schema = ["col1"])
df.write.partitionBy("col1").parquet("/home/test/test.parquet")
>>> /home/test/test.parquet/col1=NOW
df_loaded = spark.read.option(
"basePath",
"/home/test/test.parquet",
).parquet("/home/test/test.parquet/col1=*")
>>>
+--------------------------+
|col1 |
+--------------------------+
|2021-04-18 12:49:13.590431|
|TEST |{code}
was:
When reading parquet file partitioned with a column containing the value "NOW", The value is interpreted as now() and replaced by the current time at the moment of the read() funct is executed
{code:java}
// df = spark.createDataFrame(data=["NOW", "TEST"], schema = ["col1"])
df.write.partitionBy("col1").parquet("/home/test/test.parquet")
>>> /home/test/test.parquet/col1=NOW
df_loaded = spark.read.option(
"basePath",
"/home/test/test.parquet",
).parquet("/home/test/test.parquet/col1=*")
>>>
+--------------------------+
|col1 |
+--------------------------+
|2021-04-18 12:49:13.590431|
|TEST |{code}
> read partitionned parquet: my_col=NOW replaced by <current_date> on read()
> --------------------------------------------------------------------------
>
> Key: SPARK-35123
> URL: https://issues.apache.org/jira/browse/SPARK-35123
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 3.1.1
> Reporter: Killian
> Priority: Major
>
> When reading parquet file partitioned with a column containing the value "NOW", The value is interpreted as now() and replaced by the current time at the moment of the read() funct is executed
> {code:java}
> // step to reproduce
> df = spark.createDataFrame(data=["NOW", "TEST"], schema = ["col1"])
> df.write.partitionBy("col1").parquet("/home/test/test.parquet")
> >>> /home/test/test.parquet/col1=NOW
> df_loaded = spark.read.option(
> "basePath",
> "/home/test/test.parquet",
> ).parquet("/home/test/test.parquet/col1=*")
> >>>
> +--------------------------+
> |col1 |
> +--------------------------+
> |2021-04-18 12:49:13.590431|
> |TEST |{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org