You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/02/03 15:34:15 UTC
[GitHub] [pinot] kbastani opened a new issue #8117: Infer SDF for Timestamp in dimension field spec
kbastani opened a new issue #8117:
URL: https://github.com/apache/pinot/issues/8117
When ingesting a `Timestamp` field as a String that has an SDF (Simple Date Format) pattern that is non-default for `java.sql.Timestamp` in a schema's dimension field spec (which requires `yyyy-mm-dd hh:mm:ss[.fffffffff]`) Pinot will fail to import the record. To work around this, the source data must either be modified to match the standard SDF or transformed in the table spec.
Example:
```json
"dimensionFieldSpecs": [{
"name": "Date",
"dataType": "TIMESTAMP"
}]
```
Source data:
```csv
ID, DATE
1, "10/1/2005",
```
The SDF format here is not specified, since time granularity is not allowed as a dimension. The same issue can be experienced for ingesting a String field as a `Timestamp` in the date field specification of a schema, which does allow a user to specify an SDF. There are some scenarios where using the dimension field specification to automatically convert from a String in the source dataset to a `Timestamp` is useful and simpler than using the date time field spec.
The solution to this issue is to allow for conversions to a `Timestamp` from a String as an input that automatically infers popular SDFs that already have a pre-defined granularity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #8117: Infer SDF for Timestamp in dimension field spec
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #8117:
URL: https://github.com/apache/pinot/issues/8117#issuecomment-1029358666
Auto-infer SDF by trying common patterns can be quite expensive. Currently in order to ingest date time as timestamp, user can use ingestion transform `fromDateTime(dateTimeCol, 'pattern'` to convert the date time to timestamp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] kbastani commented on issue #8117: Infer SDF for Timestamp in dimension field spec
Posted by GitBox <gi...@apache.org>.
kbastani commented on issue #8117:
URL: https://github.com/apache/pinot/issues/8117#issuecomment-1029364494
The overhead for this would be the same as it was before for the default condition, meaning that only if the time format for a `TIMESTAMP` in a dimension is not recognized, it will go on to the inference logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org