You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/06/05 12:05:58 UTC
[GitHub] [druid] enenuki opened a new issue #9992: Additional functions in transformation expressions
enenuki opened a new issue #9992:
URL: https://github.com/apache/druid/issues/9992
### Description
We would like to see several useful functions available in transformation expressions. These functions
should not be very complex to add but certainly have some added value:
1. Current system timestamp: while there are different timestamp parsing functions available, this function seem to be missing.
2. Hash(code) function: Ability to create a hash from a field, subset of fields or all fields would be nice
### Motivation
Primary motivation for this both of these functions is generation of unique ID field as transformation expression during ingestion time. Maybe there is better way to achieve this?
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] enenuki commented on issue #9992: Additional functions in transformation expressions
Posted by GitBox <gi...@apache.org>.
enenuki commented on issue #9992:
URL: https://github.com/apache/druid/issues/9992#issuecomment-640457378
We are using druid to store syslog records in it. Later on we run those records against some set of rules to check for existence of some incidents. When incident is detected we need ability to refer to a set of related records and we need a record ID for this. Using timestamp, hashcode and/or UUID generator function would be beneficial for this. I realize that this may not be a best use case for druid as it is analytics database.
Regardless to our use case, having function for system time gives "ingestion" timestamp which seems useful as many realtime/streaming frameworks have this option. For example, ability to set __time field to be "system (ingestion) time" instead some value parsed from input seems logical.
Regards,
Nenad
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #9992: Additional functions in transformation expressions
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #9992:
URL: https://github.com/apache/druid/issues/9992#issuecomment-639989880
What are the use cases of such fields (system timestamp, hash code) ? Could you give an example?
If you do need a HASH code field, you could calculate the code by yourself.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #9992: Additional functions in transformation expressions
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #9992:
URL: https://github.com/apache/druid/issues/9992#issuecomment-640673797
I think this is not the use case of Druid.
These fields, such as UUID/hashcode/ingestion timestamp, will be treated as druid DEMENSIONs. And because the value of UUID field of each record is unique globally, it prevents DRUID from pre-aggregating records. In my opinion, this is what DRUID does not expect.
Maybe ElasticSearch is more appropriate in your case
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org