[jira] [Updated] (SPARK-20357) Expose Calendar.getWeekYear() as Spark SQL date function to be consistent with weekofyear()

["Hyukjin Kwon (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-20357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon updated SPARK-20357:
---------------------------------
    Labels: bulk-closed  (was: )

> Expose Calendar.getWeekYear() as Spark SQL date function to be consistent with weekofyear()
> -------------------------------------------------------------------------------------------
>
>                 Key: SPARK-20357
>                 URL: https://issues.apache.org/jira/browse/SPARK-20357
>             Project: Spark
>          Issue Type: New Feature
>          Components: SQL
>    Affects Versions: 2.1.0, 2.2.0
>            Reporter: Jeeyoung Kim
>            Priority: Minor
>              Labels: bulk-closed
>
> Since weeks and years are extracted using different boundaries (weeks happen every 7 days, years happen every 365-ish days, which is not divisible by 7), there are weird inconsistencies around how end-of-the year dates are handled if you use {{year}} and {{weekofyear}} Spark SQL functions. The example below shows how "2016-01-01" and "2016-12-30" has the same {{(year, week)}} pair.
> This happens because the week for "2016-01-01" is calculated as "last week of 2015". the Year function in Spark SQL ignores this and returns YYYY component of YYYY-MM-DD.
> The correct way to fix this is by exposing {{Java.util.dates.getWeekYear}}. This function calculates week-based years, so "2016-01-01" will return 2015 instead. in this case.
> {noformat}
> # Trying out the bug for date - using PySpark
> import pyspark.sql.functions as F
> df = spark.createDataFrame([("2016-12-31",),("2016-12-30",), ("2017-01-01",), ("2017-01-02",),("2017-12-30",)], ['id'])
> df_parsed = (
>     df
>     .withColumn("year", F.year(df['id'].cast("date")))
>     .withColumn("weekofyear", F.weekofyear(df['id'].cast("date")))
> )
> df_parsed.show()
> {noformat}
> Prints 
> {noformat}
> +----------+----+----------+
> |        id|year|weekofyear|
> +----------+----+----------+
> |2016-12-31|2016|        52|
> |2016-12-30|2016|        52|
> |2017-01-01|2017|        52| <- same (year, weekofyear) output
> |2017-01-02|2017|         1|
> |2017-12-30|2017|        52| <- 
> +----------+----+----------+
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org