You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Tangruilin (via GitHub)" <gi...@apache.org> on 2023/11/15 14:13:51 UTC
[PR] [fix #5044] Support converting 'yyyymmdd' format to date [arrow-rs]
Tangruilin opened a new pull request, #5078:
URL: https://github.com/apache/arrow-rs/pull/5078
# Which issue does this PR close?
Closes #5044 .
# Rationale for this change
format "yyyymmdd" and "yymmdd" cannot be converted to DateType, which do not comply with the ISO standard
# What changes are included in this PR?
the way to parse "yyyymmdd" and "yymmdd" are added to parse_date
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
# Are there any user-facing changes?
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix #5044] Support converting 'yyyymmdd' format to date [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #5078:
URL: https://github.com/apache/arrow-rs/pull/5078#discussion_r1401678853
##########
arrow-cast/src/parse.rs:
##########
@@ -544,6 +544,18 @@ const EPOCH_DAYS_FROM_CE: i32 = 719_163;
/// Error message if nanosecond conversion request beyond supported interval
const ERR_NANOSECONDS_NOT_SUPPORTED: &str = "The dates that can be represented as nanoseconds have to be between 1677-09-21T00:12:44.0 and 2262-04-11T23:47:16.854775804";
+fn parse_two_digit_year(input_year: i32) -> i32 {
Review Comment:
I think we should not support them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix #5044] Support converting 'yyyymmdd' format to date [arrow-rs]
Posted by "Tangruilin (via GitHub)" <gi...@apache.org>.
Tangruilin commented on code in PR #5078:
URL: https://github.com/apache/arrow-rs/pull/5078#discussion_r1401505540
##########
arrow-cast/src/parse.rs:
##########
@@ -544,6 +544,18 @@ const EPOCH_DAYS_FROM_CE: i32 = 719_163;
/// Error message if nanosecond conversion request beyond supported interval
const ERR_NANOSECONDS_NOT_SUPPORTED: &str = "The dates that can be represented as nanoseconds have to be between 1677-09-21T00:12:44.0 and 2262-04-11T23:47:16.854775804";
+fn parse_two_digit_year(input_year: i32) -> i32 {
Review Comment:
> FWIW https://www.rfc-editor.org/rfc/rfc3339#section-3 explicitly disallows 2 digit years as they are ambiguous

Maybe you need to make a decision if two digit year should be supported, and I will finish this PR by the decision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix #5044] Support converting 'yyyymmdd' format to date [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold merged PR #5078:
URL: https://github.com/apache/arrow-rs/pull/5078
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix #5044] Support converting 'yyyymmdd' format to date [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #5078:
URL: https://github.com/apache/arrow-rs/pull/5078#discussion_r1394274321
##########
arrow-cast/src/parse.rs:
##########
@@ -544,6 +544,18 @@ const EPOCH_DAYS_FROM_CE: i32 = 719_163;
/// Error message if nanosecond conversion request beyond supported interval
const ERR_NANOSECONDS_NOT_SUPPORTED: &str = "The dates that can be represented as nanoseconds have to be between 1677-09-21T00:12:44.0 and 2262-04-11T23:47:16.854775804";
+fn parse_two_digit_year(input_year: i32) -> i32 {
Review Comment:
Could we get a doc comment explaining what this is doing, and perhaps linking to the docs of the system it is based on?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix #5044] Support converting 'yyyymmdd' format to date [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #5078:
URL: https://github.com/apache/arrow-rs/pull/5078#issuecomment-1817644333
Marking as draft to show this isn't waiting on a review, feel free to unmark when you would like me to take another look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix #5044] Support converting 'yyyymmdd' format to date [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #5078:
URL: https://github.com/apache/arrow-rs/pull/5078#discussion_r1394275928
##########
arrow-cast/src/parse.rs:
##########
@@ -544,6 +544,18 @@ const EPOCH_DAYS_FROM_CE: i32 = 719_163;
/// Error message if nanosecond conversion request beyond supported interval
const ERR_NANOSECONDS_NOT_SUPPORTED: &str = "The dates that can be represented as nanoseconds have to be between 1677-09-21T00:12:44.0 and 2262-04-11T23:47:16.854775804";
+fn parse_two_digit_year(input_year: i32) -> i32 {
Review Comment:
FWIW https://www.rfc-editor.org/rfc/rfc3339#section-3 explicitly disallows 2 digit years as they are ambiguous
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix #5044] Support converting 'yyyymmdd' format to date [arrow-rs]
Posted by "Tangruilin (via GitHub)" <gi...@apache.org>.
Tangruilin commented on code in PR #5078:
URL: https://github.com/apache/arrow-rs/pull/5078#discussion_r1401504423
##########
arrow-cast/src/parse.rs:
##########
@@ -544,6 +544,18 @@ const EPOCH_DAYS_FROM_CE: i32 = 719_163;
/// Error message if nanosecond conversion request beyond supported interval
const ERR_NANOSECONDS_NOT_SUPPORTED: &str = "The dates that can be represented as nanoseconds have to be between 1677-09-21T00:12:44.0 and 2262-04-11T23:47:16.854775804";
+fn parse_two_digit_year(input_year: i32) -> i32 {
Review Comment:
> FWIW https://www.rfc-editor.org/rfc/rfc3339#section-3 explicitly disallows 2 digit years as they are ambiguous
It refer to https://www.postgresql.org/docs/current/datatype-datetime.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix #5044] Support converting 'yyyymmdd' format to date [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #5078:
URL: https://github.com/apache/arrow-rs/pull/5078#discussion_r1405471882
##########
arrow-cast/src/parse.rs:
##########
@@ -559,8 +559,30 @@ fn parse_date(string: &str) -> Option<NaiveDate> {
const HYPHEN: u8 = b'-'.wrapping_sub(b'0');
+ // refer to https://www.rfc-editor.org/rfc/rfc3339#section-3
if digits[4] != HYPHEN {
- return None;
+ if string.len() != 8 {
+ return None;
+ }
+
+ for ch in string.bytes() {
+ if !ch.is_ascii_digit() {
+ return None;
+ }
+ }
+
Review Comment:
```suggestion
```
The mask already validates this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix #5044] Support converting 'yyyymmdd' format to date [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #5078:
URL: https://github.com/apache/arrow-rs/pull/5078#discussion_r1394272576
##########
arrow-cast/src/parse.rs:
##########
@@ -560,7 +572,32 @@ fn parse_date(string: &str) -> Option<NaiveDate> {
const HYPHEN: u8 = b'-'.wrapping_sub(b'0');
if digits[4] != HYPHEN {
- return None;
+ // In this case, the string may be type "yymmdd" or "yyyymmdd", so we should check it
+ if string.len() != 6 && string.len() != 8 {
+ return None;
+ }
+ for ch in string.bytes() {
+ if ch < b'0' || ch > b'9' {
+ return None;
+ }
+ }
+ let (year, month, day) = match string.len() {
+ 6 => (
+ parse_two_digit_year(digits[0] as i32 * 10 + digits[1] as i32) as u16,
+ digits[2] * 10 + digits[3],
+ digits[4] * 10 + digits[5],
+ ),
+ 8 => (
+ digits[0] as u16 * 1000
+ + digits[1] as u16 * 100
+ + digits[2] as u16 * 10
+ + digits[3] as u16,
+ digits[4] * 10 + digits[5],
+ digits[6] * 10 + digits[7],
+ ),
+ _ => return None,
+ };
Review Comment:
```suggestion
let (year, month, day) = match (mask, string.len()) {
(0b111111, 6) => (
parse_two_digit_year(digits[0] as i32 * 10 + digits[1] as i32) as u16,
digits[2] * 10 + digits[3],
digits[4] * 10 + digits[5],
),
(0b11111111, 8) => (
digits[0] as u16 * 1000
+ digits[1] as u16 * 100
+ digits[2] as u16 * 10
+ digits[3] as u16,
digits[4] * 10 + digits[5],
digits[6] * 10 + digits[7],
),
_ => return None,
};
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix #5044] Support converting 'yyyymmdd' format to date [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #5078:
URL: https://github.com/apache/arrow-rs/pull/5078#discussion_r1405471937
##########
arrow-cast/src/parse.rs:
##########
@@ -559,8 +559,30 @@ fn parse_date(string: &str) -> Option<NaiveDate> {
const HYPHEN: u8 = b'-'.wrapping_sub(b'0');
+ // refer to https://www.rfc-editor.org/rfc/rfc3339#section-3
if digits[4] != HYPHEN {
- return None;
+ if string.len() != 8 {
+ return None;
+ }
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix #5044] Support converting 'yyyymmdd' format to date [arrow-rs]
Posted by "Tangruilin (via GitHub)" <gi...@apache.org>.
Tangruilin commented on PR #5078:
URL: https://github.com/apache/arrow-rs/pull/5078#issuecomment-1827048580
> I took the liberty of removing some redundant checks, this looks good now thank you
I modify the code as comment, now you can retry it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix #5044] Support converting 'yyyymmdd' format to date [arrow-rs]
Posted by "Tangruilin (via GitHub)" <gi...@apache.org>.
Tangruilin commented on code in PR #5078:
URL: https://github.com/apache/arrow-rs/pull/5078#discussion_r1401504423
##########
arrow-cast/src/parse.rs:
##########
@@ -544,6 +544,18 @@ const EPOCH_DAYS_FROM_CE: i32 = 719_163;
/// Error message if nanosecond conversion request beyond supported interval
const ERR_NANOSECONDS_NOT_SUPPORTED: &str = "The dates that can be represented as nanoseconds have to be between 1677-09-21T00:12:44.0 and 2262-04-11T23:47:16.854775804";
+fn parse_two_digit_year(input_year: i32) -> i32 {
Review Comment:
> FWIW https://www.rfc-editor.org/rfc/rfc3339#section-3 explicitly disallows 2 digit years as they are ambiguous
It refer to ![https://www.postgresql.org/docs/current/datatype-datetime.html]https://www.postgresql.org/docs/current/datatype-datetime.html.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix #5044] Support converting 'yyyymmdd' format to date [arrow-rs]
Posted by "Tangruilin (via GitHub)" <gi...@apache.org>.
Tangruilin commented on code in PR #5078:
URL: https://github.com/apache/arrow-rs/pull/5078#discussion_r1401694017
##########
arrow-cast/src/parse.rs:
##########
@@ -544,6 +544,18 @@ const EPOCH_DAYS_FROM_CE: i32 = 719_163;
/// Error message if nanosecond conversion request beyond supported interval
const ERR_NANOSECONDS_NOT_SUPPORTED: &str = "The dates that can be represented as nanoseconds have to be between 1677-09-21T00:12:44.0 and 2262-04-11T23:47:16.854775804";
+fn parse_two_digit_year(input_year: i32) -> i32 {
Review Comment:
> I think we should not support them
Okey. I will modify the code. Thanks for suggestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org