You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by je...@apache.org on 2018/11/28 00:38:47 UTC
[incubator-pinot] branch master updated: Removing warnings in docs
(#3560)
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new d9ad5c3 Removing warnings in docs (#3560)
d9ad5c3 is described below
commit d9ad5c3e05901fb4ea7ee33c26e88fdf011f65b3
Author: Jennifer Dai <je...@users.noreply.github.com>
AuthorDate: Tue Nov 27 16:38:43 2018 -0800
Removing warnings in docs (#3560)
* Removing some warnings when new documentation is compiled
---
docs/conf.py | 2 +-
docs/schema_timespec.rst | 76 +++++++++++++++++++++++++-----------------------
2 files changed, 40 insertions(+), 38 deletions(-)
diff --git a/docs/conf.py b/docs/conf.py
index 9163bda..a95d37e 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -157,7 +157,7 @@ html_theme = 'sphinx_rtd_theme'
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['_static']
+html_static_path = []
# Add any extra paths that contain custom files (such as robots.txt or
# .htaccess) here, relative to this directory. These files are copied
diff --git a/docs/schema_timespec.rst b/docs/schema_timespec.rst
index 94b69b9..1d7e769 100644
--- a/docs/schema_timespec.rst
+++ b/docs/schema_timespec.rst
@@ -8,18 +8,19 @@ The pinot schema timespec looks like this:
::
- "timeFieldSpec":
- {
- "name" : <name of time column>,
- "dataType" : <datatype of time column>,
- "timeFormat" : <format of time column, EPOCH or SIMPLE_DATE_FORMAT:format>,
- "timeUnitSize" : <time column granularity size>,
- "timeType" : <time unit of time column>
- }
-
+ {
+ "timeFieldSpec":
+ {
+ "name" : <name of time column>,
+ "dataType" : <datatype of time column>,
+ "timeFormat" : <format of time column, EPOCH or SIMPLE_DATE_FORMAT:format>,
+ "timeUnitSize" : <time column granularity size>,
+ "timeType" : <time unit of time column>
+ }
+ }
We are missing data granularity information in pinot schema.
-TimeUnitSize, timeType and timeFormat allow us to define the granularity of the time column, but don’t provide a way for applications to know in what buckets the data granularity is.
+TimeUnitSize, timeType and timeFormat allow us to define the granularity of the time column, but don’t provide a way for applications to know in what buckets the data granularity is.
Currently, we can only have one time column in the table which is limiting some use cases. We should allow multiple time columns and even allow derived time columns. Derived columns can be useful in performing roll ups or generating star tree aggregate nodes.
@@ -30,13 +31,15 @@ We have added a List<DateTimeFieldSpec> _dateTimeFieldSpecs to the pinot schema
::
- “dateTimeFieldSpec”:
- {`
- “name” : <name of the date time column>,
- “dataType” : <datetype of the date time column>,
- “format” : <string for interpreting the datetime column>,
- “granularity” : <string for data granularity buckets>,
- “dateTimeType” : <DateTimeType enum PRIMARY,SECONDARY or DERIVED>
+ {
+ “dateTimeFieldSpec”:
+ {
+ “name” : <name of the date time column>,
+ “dataType” : <datetype of the date time column>,
+ “format” : <string for interpreting the datetime column>,
+ “granularity” : <string for data granularity buckets>,
+ “dateTimeType” : <DateTimeType enum PRIMARY,SECONDARY or DERIVED>
+ }
}
#. name - this if the name of the date time column, similar to the older timeSpec
@@ -48,62 +51,61 @@ We have added a List<DateTimeFieldSpec> _dateTimeFieldSpecs to the pinot schema
<br>Size is the integer value of the granularity size.
<br>TimeFormat tells us whether the time column value is expressed in epoch or is a simple date format pattern.
<br>Consider 2 date time values for example 2017/07/01 00:00:00 and 2017/08/29 05:20:00:
- 1. If the time column value is defined in millisSinceEpoch (1498892400000, 1504009200000)
- this configuration will be 1:MILLISECONDS:EPOCH
- 2. If the time column value is defined in 5 minutes since epoch (4996308, 5013364)
- this configuration will be 5:MINUTES:EPOCH
- 3. If the time column value is defined in a simple date format of a day (e.g. 20170701, 20170829),
- this configuration will be 1:DAYS:SIMPLE_DATE_FORMAT:yyyyMMdd (the pattern can be configured as desired)
+1. If the time column value is defined in millisSinceEpoch (1498892400000, 1504009200000), this configuration will be 1:MILLISECONDS:EPOCH
+2. If the time column value is defined in 5 minutes since epoch (4996308, 5013364), this configuration will be 5:MINUTES:EPOCH
+3. If the time column value is defined in a simple date format of a day (e.g. 20170701, 20170829), this configuration will be 1:DAYS:SIMPLE_DATE_FORMAT:yyyyMMdd (the pattern can be configured as desired)
#. granularity - defines in what granularity the data is bucketed.
<br>Granularity has to follow pattern- size:timeunit, where size and timeUnit together define the bucket granularity of the data. This is independent of the format, which is purely defining how to interpret the numeric value in the datetime column.
- 1. if a time column is defined in millisSinceEpoch (format=1:MILLISECONDS:EPOCH), but the data buckets
- are 5 minutes, the granularity will be 5:MINUTES.
- 2. if a time column is defined in hoursSinceEpoch (format=1:HOURS:EPOCH), and the data buckets are 1
- hours, the granularity will be 1:HOURS
+1. if a time column is defined in millisSinceEpoch (format=1:MILLISECONDS:EPOCH), but the data buckets are 5 minutes, the granularity will be 5:MINUTES.
+2. if a time column is defined in hoursSinceEpoch (format=1:HOURS:EPOCH), and the data buckets are 1 hours, the granularity will be 1:HOURS
#. dateTimeType - this is an enum of values
- 1. PRIMARY: The primary date time column. This will be the date time column which keeps the milliseconds value. This will be used as the default time column, in references by pinot code (e.g. retention manager)
- 2. SECONDARY: The date time columns which are not the primary columns with milliseconds value. These can be date time columns in other granularity, put in by applications for their specific use cases
- 3. DERIVED: The date time columns which are derived, say using other columns, generated via rollups, etc
-
+1. PRIMARY: The primary date time column. This will be the date time column which keeps the milliseconds value. This will be used as the default time column, in references by pinot code (e.g. retention manager)
+2. SECONDARY: The date time columns which are not the primary columns with milliseconds value. These can be date time columns in other granularity, put in by applications for their specific use cases
+3. DERIVED: The date time columns which are derived, say using other columns, generated via rollups, etc
+
Examples:
::
“dateTimeFieldSpec”:
{
+
“name” : “Date”,
“dataType” : “LONG”,
“format” : “1:HOURS:EPOCH”,
“granularity” : “1:HOURS”,
“dateTimeType” : "PRIMARY"
- }
-
+ }
“dateTimeFieldSpec”:
{
+
“name” : “Date”,
“dataType” : “LONG”,
“format” : “1:MILLISECONDS:EPOCH”,
“granularity” : “5:MINUTES”,
“dateTimeType” : "PRIMARY"
+
}
“dateTimeFieldSpec”:
{
- `“name” : “Date”,
+
+ “name” : “Date”,
“dataType” : “LONG”,
“format” : “1:DAYS:SIMPLE_DATE_FORMAT:yyyyMMdd”,
“granularity” : “1:DAYS”,
“dateTimeType” : "SECONDARY"
+
}
Migration
~~~~~~~~~
-Once this change is pushed in, we will migrate all our clients to start populating the new DateTimeFieldSpec, along with the TimeSpec.
-<br>We can then go over all older schemas, and fill up the DateTimeFieldSpec referring to the TimeFieldSpec.
-<br>We then migrate our clients to start using DateTimeFieldSpec instead of TimeFieldSpec.
+Once this change is pushed in, we will migrate all our clients to start populating the new DateTimeFieldSpec, along with the TimeSpec.
+<br>We can then go over all older schemas, and fill up the DateTimeFieldSpec referring to the TimeFieldSpec.
+<br>We then migrate our clients to start using DateTimeFieldSpec instead of TimeFieldSpec.
<br>At this point, we can deprecate the TimeFieldSpec.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org