You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Daeho Ro (Jira)" <ji...@apache.org> on 2020/08/21 12:35:00 UTC
[jira] [Updated] (SPARK-32683) Datetime Pattern F not working as
expected
[ https://issues.apache.org/jira/browse/SPARK-32683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daeho Ro updated SPARK-32683:
-----------------------------
Summary: Datetime Pattern F not working as expected (was: Datetime Pattern F not working expected)
> Datetime Pattern F not working as expected
> ------------------------------------------
>
> Key: SPARK-32683
> URL: https://issues.apache.org/jira/browse/SPARK-32683
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.0.0
> Environment: Windows 10 Pro with Jupyter Lab Docker Image for the spark 3.0.0 and python 3.8.5.
> REPOSITORY : jupyter/all-spark-notebook
> TAG: f1811928b3dd
> Reporter: Daeho Ro
> Priority: Major
>
> From the docs, the pattern F should give a week of the month.
> |*Symbol*|*Meaning*|*Presentation*|*Example*|
> |F|week-of-month|number(1)|3|
> I have tested in the scala spark 3.0.0 and pyspark 3.0.0:
> {code:java}
> from pyspark.sql.functions import *df.withColumn('date', to_timestamp('date', 'yyyy-MM-dd')) \
> .withColumn('month', month('date')) \
> .withColumn('week', date_format('date', 'F')) \
> .show(10, False)
> +-------------------+-----+----+
> |date |month|week|
> +-------------------+-----+----+
> |2020-08-01 00:00:00|8 |1 |
> |2020-08-02 00:00:00|8 |2 |
> |2020-08-03 00:00:00|8 |3 |
> |2020-08-04 00:00:00|8 |4 |
> |2020-08-05 00:00:00|8 |5 |
> |2020-08-06 00:00:00|8 |6 |
> |2020-08-07 00:00:00|8 |7 |
> |2020-08-08 00:00:00|8 |1 |
> |2020-08-09 00:00:00|8 |2 |
> |2020-08-10 00:00:00|8 |3 |
> +-------------------+-----+----+ {code}
> The `week` column is not the week of the month. It is a day of the week as a number.
> !image-2020-08-21-21-31-32-297.png!
> From my calendar, the first day of August should have 1 for the week-of-month and from 2nd to 8th should have 2 and so on.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org