You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "martin-raymond (via GitHub)" <gi...@apache.org> on 2023/10/24 20:48:42 UTC

[I] refactor(pinot) regression on area chart (DB engine error) [superset]

martin-raymond opened a new issue, #25749:
URL: https://github.com/apache/superset/issues/25749

   We are using a pinot table as a dataset, with a temporal column _startDate_ in _epoch_ms_. Before the 3.0.0.rc4 version, we were using it to create the followin area chart : 
   
   ![areachart](https://github.com/apache/superset/assets/10060967/2b3386fd-f425-44f1-b02a-56d2382fc39c)
   
   Since the release of the 3.0.0.rc4 version, our chart is not loading : 
   
   **DB engine Erreur : could not convert string to Timestamp**
   
   Before 3.0.0.rc4, the query generated by the simple area chart was : 
   
   `SELECT DATETIMECONVERT(startDate, '1:MILLISECONDS:EPOCH', '1:MILLISECONDS:EPOCH', '1:DAYS'),
          count(*)
   FROM "dataset"
   WHERE startDate >= 1694908800000
     AND startDate < 1697500800000
   GROUP BY DATETIMECONVERT(startDate, '1:MILLISECONDS:EPOCH', '1:MILLISECONDS:EPOCH', '1:DAYS')
   ORDER BY count(*) DESC
   LIMIT 10000;`
   
   (1694908800000 and 1697500800000 are just examples of our time filter)
   
   After the 3.0.0.rc4, the query that causes the error is : 
   
   `SELECT CAST(DATE_TRUNC('day', CAST(DATETIMECONVERT((startDate/1000), '1:SECONDS:EPOCH', '1:SECONDS:EPOCH', '1:SECONDS') AS TIMESTAMP)) AS TIMESTAMP),
          count(*)
   FROM "dataset"
   WHERE startDate >= 1694908800000
     AND startDate < 1697500800000
   GROUP BY CAST(DATE_TRUNC('day', CAST(DATETIMECONVERT((startDate/1000), '1:SECONDS:EPOCH', '1:SECONDS:EPOCH', '1:SECONDS') AS TIMESTAMP)) AS TIMESTAMP)
   ORDER BY count(*) DESC
   LIMIT 10000;`
   
   The part : 
   
   `DATETIMECONVERT((startDate/1000), '1:SECONDS:EPOCH', '1:SECONDS:EPOCH', '1:SECONDS')` 
   
   seems odd because it literally said "convert seconds in seconds", and we think it might causes the crash.
   
   We think the problem is caused by : https://github.com/apache/superset/pull/24942
   
   Thx for your help
   
   - superset version: `3.0.0.rc4`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "martin-raymond (via GitHub)" <gi...@apache.org>.
martin-raymond commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1802413554

   @zhaoyongjie well, it seems to have solved the issue (but we have to use a calculated column as you adviced).
   we have a lots of charts to verify now, but it seems to work fine on the first
   
   thank you very much @zhaoyongjie for your help. i will keep you informed here if it works on every charts, then i will close my issue. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-2043678000

   Agreed. I'll be optimistic here and just close this one. If @martin-raymond is still facing this issue, we can revisit and/or reopen as necessary.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "zhaoyongjie (via GitHub)" <gi...@apache.org>.
zhaoyongjie commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1779379248

   @martin-raymond I'm wondering it might be a bug in Apache Calcite side because it can't correctly calculate expression `DATETIMECONVERT((startDate/1000), '1:SECONDS:EPOCH', '1:SECONDS:EPOCH', '1:SECONDS')`. 
   
   There is a workaround might be useful, 1) try to create a derived column based on the _startDate_  in Pinot as `timestamp` type instead of `long` type,  2) then **remove** epoch_ms tag in column config, 3), syncing column metadata in Dataset Model in Superset. 4) use the new derived column replace with previous one in Chart.
   
   Hope it helps you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "zhaoyongjie (via GitHub)" <gi...@apache.org>.
zhaoyongjie commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1782871421

   > ![image](https://user-images.githubusercontent.com/10060967/278645399-7d9780e0-dd4a-4f4e-814a-ba41a02d93e5.png)
   > 
   > The drill to detail shows that we are able to get events, but for some reason the display is bogged
   
   Let's focus on the Pinot and Area chart.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "zhaoyongjie (via GitHub)" <gi...@apache.org>.
zhaoyongjie commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1782832154

   > @zhaoyongjie yes we tried TIMESTAMP, STRING, with a format datetime, without a format, same results in the end
   
   Could you please post screenshot of explore? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "martin-raymond (via GitHub)" <gi...@apache.org>.
martin-raymond commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-2045831803

   we changed all of our temporal field in our pinot table, so we skipped the problem completely. the thread can stay closed, thx


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "zhaoyongjie (via GitHub)" <gi...@apache.org>.
zhaoyongjie commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1802162821

   https://pypi.org/project/pinotdb/
   
   The latest version is 5.1.0, try to update it.
   
   On Wed, Nov 8, 2023 at 4:38 PM martin-raymond ***@***.***>
   wrote:
   
   > pinotdb : 0.3.8
   >
   > —
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/superset/issues/25749#issuecomment-1802144922>,
   > or unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/AAPMKUVJNPMETVVPENBIUGLYDORRHAVCNFSM6AAAAAA6OJMWLCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBSGE2DIOJSGI>
   > .
   > You are receiving this because you were mentioned.Message ID:
   > ***@***.***>
   >
   
   
   -- 
   
   Best regards,
   
   Yongjie
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "zhaoyongjie (via GitHub)" <gi...@apache.org>.
zhaoyongjie commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1782872843

   > ![image](https://user-images.githubusercontent.com/10060967/278644368-a53f1bc9-5e18-4783-90cf-c76ecda17394.png)
   > 
   > The query :
   > 
   > ```
   > SELECT CAST(DATE_TRUNC('day', CAST(cast(dateHeureDebut as timestamp) AS TIMESTAMP)) AS TIMESTAMP),
   >        COUNT(*)
   > FROM "knada-indicateur-visite-v2"."knada-indicateur-visite-v2"
   > GROUP BY CAST(DATE_TRUNC('day', CAST(cast(dateHeureDebut as timestamp) AS TIMESTAMP)) AS TIMESTAMP)
   > ORDER BY COUNT(*) DESC
   > LIMIT 10000;
   > ```
   > 
   > which works fine in pinot
   > 
   > @zhaoyongjie that is what you wanted ? and thank you very much for your help by the way !
   
   Please clear up the "Contribution Mode" and try it again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "zhaoyongjie (via GitHub)" <gi...@apache.org>.
zhaoyongjie commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1782897382

   > Aucun is the french word for "no contribution mode" so it is already cleared up @zhaoyongjie
   
   It's weird. Could you click the three dots on the right corner of Explore page, and check out what Query is? and please check the logs of Superset, does superset a acturally send query?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "zhaoyongjie (via GitHub)" <gi...@apache.org>.
zhaoyongjie commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1802084217

   The first column of result seems like TEXT instead of DATATIME from the screenshot of SQLLab. Please check out Pinot side or what did you change in Superset codebase?
   
   ![](https://user-images.githubusercontent.com/10060967/281446414-457e339c-b08c-46d3-8db8-a88bc7ff32e9.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "martin-raymond (via GitHub)" <gi...@apache.org>.
martin-raymond commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1781806269

   @zhaoyongjie we tried to use a calculated column as you said
   
   superset seems to not be able to display the result from pinot. here are the logs when we try to create a chart : 
   
   ```
   10.42.9.0 - - [26/Oct/2023:19:47:43 +0000] "GET /api/v1/me/ HTTP/1.1" 200 150 "https://superset.knada.rancher.kosmos.fr/explore/?form_data_key=BjW4HfY8NeJzcurrBzb6L16PZtorS2K4FnVzTPuFXrk6Pfk0YUH1mQOFAuNUK4jc&datasource_type=table&datasource_id=54" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"
   2023-10-26 19:47:45,016:INFO:pinotdb.sqlalchemy:Updated pinot dialect args from {'debug': False}: None and False
   2023-10-26 19:47:45,038:INFO:pinotdb.sqlalchemy:Updated pinot dialect args from {'dbapi': <module 'pinotdb' from '/usr/local/lib/python3.9/site-packages/pinotdb/__init__.py'>, 'compiler_linting': 1, 'debug': False}: None and False
   2023-10-26 19:47:45,038:INFO:pinotdb.sqlalchemy:Updated pinot dialect args from {'host': 'pinot-broker.pinot.svc.cluster.local', 'port': 8099, 'path': 'query/sql', 'scheme': 'http', 'username': None, 'password': None, 'debug': False}: http://pinot-controller.pinot.svc.cluster.local:9000 and False
   2023-10-26 19:47:45,062:INFO:pinotdb.sqlalchemy:Updated pinot dialect args from {'dbapi': <module 'pinotdb' from '/usr/local/lib/python3.9/site-packages/pinotdb/__init__.py'>, 'compiler_linting': 1, 'debug': False}: None and False
   2023-10-26 19:47:45,062:INFO:pinotdb.sqlalchemy:Updated pinot dialect args from {'host': 'pinot-broker.pinot.svc.cluster.local', 'port': 8099, 'path': 'query/sql', 'scheme': 'http', 'username': None, 'password': None, 'debug': False}: http://pinot-controller.pinot.svc.cluster.local:9000 and False
   2023-10-26 19:47:45,066:INFO:pinotdb.sqlalchemy:Updated pinot dialect args from {'dbapi': <module 'pinotdb' from '/usr/local/lib/python3.9/site-packages/pinotdb/__init__.py'>, 'compiler_linting': 1, 'debug': False}: None and False
   2023-10-26 19:47:45,066:INFO:pinotdb.sqlalchemy:Updated pinot dialect args from {'host': 'pinot-broker.pinot.svc.cluster.local', 'port': 8099, 'path': 'query/sql', 'scheme': 'http', 'username': None, 'password': None, 'debug': False}: http://pinot-controller.pinot.svc.cluster.local:9000 and False
   2023-10-26 19:47:45,072:DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): pinot-broker.pinot.svc.cluster.local:8099
   2023-10-26 19:47:45,114:DEBUG:urllib3.connectionpool:http://pinot-broker.pinot.svc.cluster.local:8099 "POST /query/sql HTTP/1.1" 200 3744
   2023-10-26 19:47:45,115:DEBUG:pinotdb.db:Got the rows as a type <class 'list'> of size 84
   2023-10-26 19:47:45,116:DEBUG:pinotdb.db:[['2023-06-08 00:00:00.0', 443],
    ['2023-03-22 00:00:00.0', 101],
    ['2023-10-06 00:00:00.0', 82],
    ['2023-06-27 00:00:00.0', 80],
    ['2023-03-09 00:00:00.0', 78],
    ['2023-06-28 00:00:00.0', 78],
    ['2023-03-24 00:00:00.0', 74],
    ['2023-05-02 00:00:00.0', 69],
    ['2023-04-19 00:00:00.0', 69],
    ['2023-03-23 00:00:00.0', 65],
    ['2023-03-28 00:00:00.0', 62],
    ['2023-03-21 00:00:00.0', 61],
    ['2023-04-05 00:00:00.0', 58],
    ['2023-06-26 00:00:00.0', 58],
    ['2023-06-15 00:00:00.0', 58],
    ['2023-02-28 00:00:00.0', 56],
    ['2023-02-27 00:00:00.0', 55],
    ['2023-06-06 00:00:00.0', 53],
    ['2023-03-10 00:00:00.0', 52],
    ['2023-05-09 00:00:00.0', 51],
    ['2023-06-16 00:00:00.0', 50],
    ['2023-04-21 00:00:00.0', 49],
    ['2023-06-14 00:00:00.0', 48],
    ['2023-06-30 00:00:00.0', 48],
    ['2023-04-11 00:00:00.0', 48],
    ['2023-05-10 00:00:00.0', 48],
    ['2023-08-30 00:00:00.0', 47],
    ['2023-03-27 00:00:00.0', 46],
    ['2023-05-03 00:00:00.0', 45],
    ['2023-03-13 00:00:00.0', 44],
    ['2023-03-30 00:00:00.0', 44],
    ['2023-05-04 00:00:00.0', 42],
    ['2023-06-02 00:00:00.0', 42],
    ['2023-09-18 00:00:00.0', 41],
    ['2023-03-16 00:00:00.0', 38],
    ['2023-05-05 00:00:00.0', 38],
    ['2023-06-13 00:00:00.0', 38],
    ['2023-05-12 00:00:00.0', 38],
    ['2023-03-15 00:00:00.0', 36],
    ['2023-04-03 00:00:00.0', 36],
    ['2023-03-01 00:00:00.0', 36],
    ['2023-04-12 00:00:00.0', 36],
    ['2023-03-20 00:00:00.0', 36],
    ['2023-04-18 00:00:00.0', 34],
    ['2023-03-14 00:00:00.0', 34],
    ['2023-04-04 00:00:00.0', 33],
    ['2023-03-07 00:00:00.0', 33],
    ['2023-06-07 00:00:00.0', 31],
    ['2023-03-02 00:00:00.0', 31],
    ['2023-06-21 00:00:00.0', 31],
    ['2023-03-31 00:00:00.0', 30],
    ['2023-04-07 00:00:00.0', 29],
    ['2023-08-29 00:00:00.0', 29],
    ['2023-06-19 00:00:00.0', 28],
    ['2023-05-31 00:00:00.0', 27],
    ['2023-05-11 00:00:00.0', 26],
    ['2023-06-20 00:00:00.0', 26],
    ['2023-04-06 00:00:00.0', 25],
    ['2023-06-05 00:00:00.0', 25],
    ['2023-06-01 00:00:00.0', 24],
    ['2023-09-20 00:00:00.0', 23],
    ['2023-03-17 00:00:00.0', 23],
    ['2023-09-19 00:00:00.0', 22],
    ['2023-07-03 00:00:00.0', 22],
    ['2023-06-22 00:00:00.0', 22],
    ['2023-03-29 00:00:00.0', 21],
    ['2023-04-25 00:00:00.0', 20],
    ['2023-04-28 00:00:00.0', 18],
    ['2023-04-27 00:00:00.0', 17],
    ['2023-04-26 00:00:00.0', 17],
    ['2023-03-08 00:00:00.0', 16],
    ['2023-05-08 00:00:00.0', 13],
    ['2023-04-20 00:00:00.0', 12],
    ['2023-05-01 00:00:00.0', 10],
    ['2023-02-24 00:00:00.0', 8],
    ['2023-04-24 00:00:00.0', 8],
    ['2023-09-12 00:00:00.0', 7],
    ['2023-04-17 00:00:00.0', 7],
    ['2023-06-29 00:00:00.0', 6],
    ['2023-08-31 00:00:00.0', 4],
    ['2023-10-05 00:00:00.0', 2],
    ['2023-09-11 00:00:00.0', 1],
    ['2023-05-30 00:00:00.0', 1],
    ['2023-08-28 00:00:00.0', 1]]
   10.42.9.0 - - [26/Oct/2023:19:47:45 +0000] "POST /api/v1/chart/data?force HTTP/1.1" 200 472 "https://superset.knada.rancher.kosmos.fr/explore/?form_data_key=BjW4HfY8NeJzcurrBzb6L16PZtorS2K4FnVzTPuFXrk6Pfk0YUH1mQOFAuNUK4jc&datasource_type=table&datasource_id=54" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"
   10.42.9.0 - - [26/Oct/2023:19:47:46 +0000] "PUT /api/v1/explore/form_data/BjW4HfY8NeJzcurrBzb6L16PZtorS2K4FnVzTPuFXrk6Pfk0YUH1mQOFAuNUK4jc?tab_id=63 HTTP/1.1" 200 75 "https://superset.knada.rancher.kosmos.fr/explore/?form_data_key=BjW4HfY8NeJzcurrBzb6L16PZtorS2K4FnVzTPuFXrk6Pfk0YUH1mQOFAuNUK4jc&datasource_type=table&datasource_id=54" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"
   10.42.9.0 - - [26/Oct/2023:19:47:46 +0000] "POST /superset/log/?explode=events&dashboard_id=1 HTTP/1.1" 200 1 "https://superset.knada.rancher.kosmos.fr/explore/?form_data_key=BjW4HfY8NeJzcurrBzb6L16PZtorS2K4FnVzTPuFXrk6Pfk0YUH1mQOFAuNUK4jc&datasource_type=table&datasource_id=54" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"
   ``` 
   
   The calculated column : 
   ![image](https://github.com/apache/superset/assets/10060967/f319755d-db1b-42fd-9dc4-bbed6d972085)
   ![image](https://github.com/apache/superset/assets/10060967/ebc4f114-7b76-4f4a-8a84-b99d142d8b62)
   
   (we tried DATETIME and STRING, same result)
   
   There is no error, just nothing is displayed in superset (no result)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "martin-raymond (via GitHub)" <gi...@apache.org>.
martin-raymond commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1782849453

   ![image](https://github.com/apache/superset/assets/10060967/a53f1bc9-5e18-4783-90cf-c76ecda17394)
   
   The query : 
   `SELECT CAST(DATE_TRUNC('day', CAST(cast(dateHeureDebut as timestamp) AS TIMESTAMP)) AS TIMESTAMP),
          COUNT(*)
   FROM "knada-indicateur-visite-v2"."knada-indicateur-visite-v2"
   GROUP BY CAST(DATE_TRUNC('day', CAST(cast(dateHeureDebut as timestamp) AS TIMESTAMP)) AS TIMESTAMP)
   ORDER BY COUNT(*) DESC
   LIMIT 10000;`
   
   which works fine in pinot
   
   @zhaoyongjie that is what you wanted ? 
   and thank you very much for your help by the way !
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "zhaoyongjie (via GitHub)" <gi...@apache.org>.
zhaoyongjie commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-2043657177

   Hi Evan, we are using Superset 3.1 and Pinot stable version on the
   Production environment, the corresponding query works well, so if no more
   issues are reported, might close this one.
   
   
   
   On Mon 8. Apr 2024 at 22:58, Evan Rusackas ***@***.***> wrote:
   
   > It's been quite a while since this thread was revisited, but it's one of
   > the best support threads I've read on here in a long time! @martin-raymond
   > <https://github.com/martin-raymond> are you still up against this, and
   > @zhaoyongjie <https://github.com/zhaoyongjie> do you have any new
   > theories? I'm wondering if this is still an issue in Superset 3.1 / 4.0. If
   > there is a faulty query being generated, maybe you can share what it looks
   > like now?
   >
   > —
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/superset/issues/25749#issuecomment-2043630225>,
   > or unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/AAPMKUVOR5HY4YCT22DXY4TY4MAHLAVCNFSM6AAAAAA6OJMWLCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBTGYZTAMRSGU>
   > .
   > You are receiving this because you were mentioned.Message ID:
   > ***@***.***>
   >
   -- 
   
   Best regards,
   
   Yongjie
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "martin-raymond (via GitHub)" <gi...@apache.org>.
martin-raymond commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1808955001

   i confirm all our charts are now ok. thx very much @zhaoyongjie 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "martin-raymond (via GitHub)" <gi...@apache.org>.
martin-raymond closed issue #25749: refactor(pinot) regression on area chart (DB engine error)
URL: https://github.com/apache/superset/issues/25749


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "zhaoyongjie (via GitHub)" <gi...@apache.org>.
zhaoyongjie commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1782840674

   @martin-raymond Data Explore page, the other words that open the chart in Superset.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "martin-raymond (via GitHub)" <gi...@apache.org>.
martin-raymond commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1802064504

   ![image](https://github.com/apache/superset/assets/10060967/f2d409b0-15a9-462e-a87e-007d68bb692e)
   ![image](https://github.com/apache/superset/assets/10060967/668d8980-6b9d-4afc-b638-38c9ddc9f918)
   ![image](https://github.com/apache/superset/assets/10060967/a612943c-6874-4c66-9f2d-e5d127647b35)
   ![image](https://github.com/apache/superset/assets/10060967/457e339c-b08c-46d3-8db8-a88bc7ff32e9)
   
   @zhaoyongjie 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas closed issue #25749: refactor(pinot) regression on area chart (DB engine error)
URL: https://github.com/apache/superset/issues/25749


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "martin-raymond (via GitHub)" <gi...@apache.org>.
martin-raymond commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1824829162

   @zhaoyongjie the issue is still ongoing.
   
   The workaround is theoretically correct, but in practice, on our production data, the volume is so significant that casting to a timestamp, applied to the temproel field through the time range filter, results in processing times multiplied by 40, making our service unusable.
   
   This is ultimately due to a faulty query in Superset 3.0.1. Therefore, the problem persists.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "martin-raymond (via GitHub)" <gi...@apache.org>.
martin-raymond commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1781338913

   ok @zhaoyongjie . that's not it.  there is no error, but the result is not right. Here is the query executed in pinot : 
   ![image](https://github.com/apache/superset/assets/10060967/8557d5ce-2700-4a27-805b-83bde6138572)
   
   The syntax is right, but the result is not the result expected


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "martin-raymond (via GitHub)" <gi...@apache.org>.
martin-raymond commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1782990469

   the query : 
   ```
   SELECT CAST(DATE_TRUNC('day', CAST(cast(dateHeureDebut as timestamp) AS TIMESTAMP)) AS TIMESTAMP),
          count(*)
   FROM "knada-indicateur-visite-v2"."knada-indicateur-visite-v2"
   GROUP BY CAST(DATE_TRUNC('day', CAST(cast(dateHeureDebut as timestamp) AS TIMESTAMP)) AS TIMESTAMP)
   ORDER BY count(*) DESC
   LIMIT 10000;
   ``` 
   
   the logs : 
   
   ```
   2023-10-27 14:12:29,058:INFO:pinotdb.sqlalchemy:Updated pinot dialect args from {'debug': False}: None and False
   2023-10-27 14:12:29,078:INFO:pinotdb.sqlalchemy:Updated pinot dialect args from {'dbapi': <module 'pinotdb' from '/usr/local/lib/python3.9/site-packages/pinotdb/__init__.py'>, 'compiler_linting': 1, 'debug': False}: None and False
   2023-10-27 14:12:29,078:INFO:pinotdb.sqlalchemy:Updated pinot dialect args from {'host': 'pinot-broker.pinot.svc.cluster.local', 'port': 8099, 'path': 'query/sql', 'scheme': 'http', 'username': None, 'password': None, 'debug': False}: http://pinot-controller.pinot.svc.cluster.local:9000 and False
   2023-10-27 14:12:29,090:INFO:pinotdb.sqlalchemy:Updated pinot dialect args from {'dbapi': <module 'pinotdb' from '/usr/local/lib/python3.9/site-packages/pinotdb/__init__.py'>, 'compiler_linting': 1, 'debug': False}: None and False
   2023-10-27 14:12:29,090:INFO:pinotdb.sqlalchemy:Updated pinot dialect args from {'host': 'pinot-broker.pinot.svc.cluster.local', 'port': 8099, 'path': 'query/sql', 'scheme': 'http', 'username': None, 'password': None, 'debug': False}: http://pinot-controller.pinot.svc.cluster.local:9000 and False
   2023-10-27 14:12:29,093:INFO:pinotdb.sqlalchemy:Updated pinot dialect args from {'dbapi': <module 'pinotdb' from '/usr/local/lib/python3.9/site-packages/pinotdb/__init__.py'>, 'compiler_linting': 1, 'debug': False}: None and False
   2023-10-27 14:12:29,094:INFO:pinotdb.sqlalchemy:Updated pinot dialect args from {'host': 'pinot-broker.pinot.svc.cluster.local', 'port': 8099, 'path': 'query/sql', 'scheme': 'http', 'username': None, 'password': None, 'debug': False}: http://pinot-controller.pinot.svc.cluster.local:9000 and False
   2023-10-27 14:12:29,096:DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): pinot-broker.pinot.svc.cluster.local:8099
   2023-10-27 14:12:29,160:DEBUG:urllib3.connectionpool:http://pinot-broker.pinot.svc.cluster.local:8099 "POST /query/sql HTTP/1.1" 200 None
   2023-10-27 14:12:29,175:DEBUG:pinotdb.db:Got the rows as a type <class 'list'> of size 1000
   2023-10-27 14:12:29,284:DEBUG:pinotdb.db:[['Personnel',
     '2023-05-31 12:03:14.082',
     'Collège Aldébaran (COL - 0000000C)',
     ['Scolarité', "Page d'accueil personnelle"],
     '1115',
     'Ordinateur',
     ['Portail Académique', 'Portail département', 'Site inter etablissement'],
     ['Public'],
     ['Absences - tableau de bord', 'Absences', 'Accueil personnel'],
     'Europe/Paris',
     ['Collège'],
     1685534594082,
     1685538574435],
    ['Enseignant',
     '2023-05-31 12:43:46.39',
     'Cite scolaire Colt et Emanche (LEG - 2486935G, COL - 8098115I)',
     ['Communication'],
     '4817',
     'Ordinateur',
     ['Site inter etablissement',
      'Portail région',
      'Portail Académique',
      'Portail département'],
     ['Public'],
     ['Messagerie'],
     'Europe/Paris',
     ["Lycée d'enseignement général", 'Collège'],
     1685537026390,
     1685538862566],
    ['Autre',
     '2023-05-31 15:23:37.19',
     'Collège Aldébaran (COL - 0000000C)',
     ["Page d'accueil personnelle"],
     '1115',
     'Ordinateur',
     ['Portail Académique', 'Portail département', 'Site inter etablissement'],
     ['Public'],
     ['Accueil personnel'],
     'Europe/Paris',
     ['Collège'],
     1685546617190,
     1685548417556],
    ['Autre',
     '2023-05-31 15:25:02.604',
     'Cite scolaire Colt et Emanche (LEG - 2486935G, COL - 8098115I)',
     ["Page d'accueil personnelle", 'Publication'],
     '4817',
     'Ordinateur',
     ['Site inter etablissement',
      'Portail région',
      'Portail Académique',
      'Portail département'],
     ['Public'],
     ['Accueil personnel', 'Communication ciblée'],
     'Europe/Paris',
     ["Lycée d'enseignement général", 'Collège'],
     1685546702604,
     1685548505940],
    ['Autre',
     '2023-05-31 15:33:26.236',
     'Collège Aldébaran (COL - 0000000C)',
     ['Publication', "Page d'accueil personnelle", 'Services transverses'],
     '1115',
     'Ordinateur',
     ['Portail Académique', 'Portail département', 'Site inter etablissement'],
     ['Public'],
     ['Communication ciblée', 'Accueil personnel', 'Annuaire'],
     'Europe/Paris',
     ['Collège'],
     1685547206236,
     1685550716680], .....
   ....
   ]]
   2023-10-27 14:12:29,551:INFO:pinotdb.sqlalchemy:Updated pinot dialect args from {'debug': False}: None and False
   2023-10-27 14:12:29,561:INFO:pinotdb.sqlalchemy:Updated pinot dialect args from {'dbapi': <module 'pinotdb' from '/usr/local/lib/python3.9/site-packages/pinotdb/__init__.py'>, 'compiler_linting': 1, 'debug': False}: None and False
   2023-10-27 14:12:29,561:INFO:pinotdb.sqlalchemy:Updated pinot dialect args from {'host': 'pinot-broker.pinot.svc.cluster.local', 'port': 8099, 'path': 'query/sql', 'scheme': 'http', 'username': None, 'password': None, 'debug': False}: http://pinot-controller.pinot.svc.cluster.local:9000 and False
   2023-10-27 14:12:29,567:INFO:pinotdb.sqlalchemy:Updated pinot dialect args from {'dbapi': <module 'pinotdb' from '/usr/local/lib/python3.9/site-packages/pinotdb/__init__.py'>, 'compiler_linting': 1, 'debug': False}: None and False
   2023-10-27 14:12:29,568:INFO:pinotdb.sqlalchemy:Updated pinot dialect args from {'host': 'pinot-broker.pinot.svc.cluster.local', 'port': 8099, 'path': 'query/sql', 'scheme': 'http', 'username': None, 'password': None, 'debug': False}: http://pinot-controller.pinot.svc.cluster.local:9000 and False
   2023-10-27 14:12:29,572:INFO:pinotdb.sqlalchemy:Updated pinot dialect args from {'dbapi': <module 'pinotdb' from '/usr/local/lib/python3.9/site-packages/pinotdb/__init__.py'>, 'compiler_linting': 1, 'debug': False}: None and False
   2023-10-27 14:12:29,572:INFO:pinotdb.sqlalchemy:Updated pinot dialect args from {'host': 'pinot-broker.pinot.svc.cluster.local', 'port': 8099, 'path': 'query/sql', 'scheme': 'http', 'username': None, 'password': None, 'debug': False}: http://pinot-controller.pinot.svc.cluster.local:9000 and False
   2023-10-27 14:12:29,576:DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): pinot-broker.pinot.svc.cluster.local:8099
   2023-10-27 14:12:29,593:DEBUG:urllib3.connectionpool:http://pinot-broker.pinot.svc.cluster.local:8099 "POST /query/sql HTTP/1.1" 200 1218
   2023-10-27 14:12:29,594:DEBUG:pinotdb.db:Got the rows as a type <class 'list'> of size 1
   2023-10-27 14:12:29,594:DEBUG:pinotdb.db:[[3474]]
   10.42.9.0 - - [27/Oct/2023:14:12:29 +0000] "POST /datasource/samples?force=false&datasource_type=table&datasource_id=54 HTTP/1.1" 200 21448 "https://superset.knada.rancher.kosmos.fr/explore/?form_data_key=5f5-KbTVK-Eft9BQ2pt9PUHQkwlGyTrWv3FKA5tViwf8ED0BFocjk3VArP32EmRM&dashboard_page_id=bZXKqzeU7&slice_id=9" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"
   10.42.9.0 - - [27/Oct/2023:14:12:34 +0000] "GET /api/v1/me/ HTTP/1.1" 200 150 "https://superset.knada.rancher.kosmos.fr/explore/?form_data_key=5f5-KbTVK-Eft9BQ2pt9PUHQkwlGyTrWv3FKA5tViwf8ED0BFocjk3VArP32EmRM&dashboard_page_id=bZXKqzeU7&slice_id=9" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"
   ``` 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "zhaoyongjie (via GitHub)" <gi...@apache.org>.
zhaoyongjie commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1802134847

   What did version of SQLAlchemy Driver use?  try following command in your superset cluser.
   
   ```bash
   pip list | grep pinot
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "zhaoyongjie (via GitHub)" <gi...@apache.org>.
zhaoyongjie commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1779660643

   @martin-raymond  Could you please post error screenshot or log?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "martin-raymond (via GitHub)" <gi...@apache.org>.
martin-raymond commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1781272942

   ![image](https://github.com/apache/superset/assets/10060967/d53023f1-801f-4053-b787-b7f143ee576c)
   @zhaoyongjie 
   
   you want me to try to add a calculated column in the dataset in superset directly ? the timestamp type is not present


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "martin-raymond (via GitHub)" <gi...@apache.org>.
martin-raymond commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1779573986

   we tried that, but with pinot 1.0.0 and superset 3.0.1, the table does not regenerated properly in superset (error on the field timestamp in the schema)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "martin-raymond (via GitHub)" <gi...@apache.org>.
martin-raymond commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1802144922

   pinotdb : 0.3.8
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "zhaoyongjie (via GitHub)" <gi...@apache.org>.
zhaoyongjie commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1800290386

   @martin-raymond I was wondering we should checkout all the list for this case
   1. please confirm remove "ms" or other "python datetime format" in column config
   2. please confirm the type of temporal column is DATETIME instead of LONG/INTERGER or TEXT
   3. please confirm the x-axis used termporal column
   
   BTW, we have deployed this patch and run Pinot about 3 month, everything works fine.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "zhaoyongjie (via GitHub)" <gi...@apache.org>.
zhaoyongjie commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1781319940

   @martin-raymond 
   I tested this case in my local Pinot environment, which is might be DATETIMECONVERT expression got an exception if the first arugument is float value.
   
   <img width="1088" alt="image" src="https://github.com/apache/superset/assets/2016594/0a171488-bb37-4746-a53e-e944fee4f1ce">
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "zhaoyongjie (via GitHub)" <gi...@apache.org>.
zhaoyongjie commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1781333345

   > ![image](https://user-images.githubusercontent.com/10060967/278380341-d53023f1-801f-4053-b787-b7f143ee576c.png) @zhaoyongjie stardDate is dateHeureDebut
   > 
   > you want me to try to add a calculated column in the dataset in superset directly ? 
   Yes,  from your screenshot the dateHeureDebut is Long.
   
   > the timestamp type is not present
   Not True, try the expression`cast(dateHeureDebut as timestamp)` use it and create a calculated column in Superset. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "zhaoyongjie (via GitHub)" <gi...@apache.org>.
zhaoyongjie commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1781886113

   @martin-raymond column type should be "timestamp" rather than "string", you don't need to fill out the format datetime input box.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "martin-raymond (via GitHub)" <gi...@apache.org>.
martin-raymond commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1801947169

   @zhaoyongjie thx for your answer
   
   yes i confirm all that, i just re-tried everything from scratch, same results


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "martin-raymond (via GitHub)" <gi...@apache.org>.
martin-raymond commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1802125643

   pinot seems fine. what do you want me to check there ? it worked weel until the new request from 3.0.0
   we changed nothing in superset codebase. @zhaoyongjie 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "martin-raymond (via GitHub)" <gi...@apache.org>.
martin-raymond commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1799656064

   Hi @zhaoyongjie and @ege-st do you have any news on the issue ? After discussion with the pinot team, the problem is definitly in superset, and from our perspective, it is a huge issue because it prevents us to be able to use the 3.0.0 for our clients


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "martin-raymond (via GitHub)" <gi...@apache.org>.
martin-raymond commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1781204144

   @zhaoyongjie we discussed with the pinot team and the problem should be in superset and not in pinot or apache calcite


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "martin-raymond (via GitHub)" <gi...@apache.org>.
martin-raymond commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1782836862

   @zhaoyongjie what do you mean by "explore" ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "martin-raymond (via GitHub)" <gi...@apache.org>.
martin-raymond commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1782855217

   ![image](https://github.com/apache/superset/assets/10060967/7d9780e0-dd4a-4f4e-814a-ba41a02d93e5)
   
   The drill to detail shows that we are able to get events, but for some reason the display is bogged


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-2043630225

   It's been quite a while since this thread was revisited, but it's one of the best support threads I've read on here in a long time! @martin-raymond are you still up against this, and @zhaoyongjie do you have any new theories? I'm wondering if this is still an issue in Superset 3.1 / 4.0. If there is a faulty query being generated, maybe you can share what it looks like now?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "zhaoyongjie (via GitHub)" <gi...@apache.org>.
zhaoyongjie commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1802012558

   > @zhaoyongjie thx for your answer
   > 
   > yes i confirm all that, i just re-tried everything from scratch, same results
   
   Could you post screenshot as following
   1) Column config in Dataset Edit model, especially, the config of temporal column setting.
   2) Explore Page


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "zhaoyongjie (via GitHub)" <gi...@apache.org>.
zhaoyongjie commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1781241214

   @martin-raymond 
   Could you post the config of `startDate` column in Superset Dataset ? I guess this column might be a second EPOCH but set it as ms in config.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "martin-raymond (via GitHub)" <gi...@apache.org>.
martin-raymond commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1782802381

   @zhaoyongjie yes we tried TIMESTAMP, STRING, with a format datetime, without a format, same results in the end


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "zhaoyongjie (via GitHub)" <gi...@apache.org>.
zhaoyongjie commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1781223526

   @martin-raymond Do you try my suggestion that removing "ms" tag and create a timestamp column instead of integer column as time column in Superset dataset?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "martin-raymond (via GitHub)" <gi...@apache.org>.
martin-raymond commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1782874808

   Aucun is the french word for "no contribution mode" so it is already cleared up @zhaoyongjie 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] refactor(pinot) regression on area chart (DB engine error) [superset]

Posted by "martin-raymond (via GitHub)" <gi...@apache.org>.
martin-raymond commented on issue #25749:
URL: https://github.com/apache/superset/issues/25749#issuecomment-1779784793

   @zhaoyongjie yes, here it is : 
   
   `2023-10-25 17:13:09,270:INFO:pinotdb.sqlalchemy:Updated pinot dialect args from {'dbapi': <module 'pinotdb' from '/usr/local/lib/python3.9/site-packages/pinotdb/__init__.py'>, 'compiler_linting': 1, 'debug': False}: None and False
   2023-10-25 17:13:09,270:INFO:pinotdb.sqlalchemy:Updated pinot dialect args from {'host': 'pinot-broker.pinot.svc.cluster.local', 'port': 8099, 'path': 'query/sql', 'scheme': 'http', 'username': None, 'password': None, 'debug': False}: http://pinot-controller.pinot.svc.cluster.local:9000/ and False
   2023-10-25 17:13:09,274:DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): pinot-controller.pinot.svc.cluster.local:9000
   2023-10-25 17:13:09,280:DEBUG:urllib3.connectionpool:http://pinot-controller.pinot.svc.cluster.local:9000/ "GET /tables/knada-indicateur-visite-v2/schema HTTP/1.1" 200 1350
   2023-10-25 17:13:09,281:INFO:pinotdb.sqlalchemy:Getting columns for knada-indicateur-visite-v2 from http://pinot-controller.pinot.svc.cluster.local:9000/: {'schemaName': 'knada-indicateur-visite-v2', 'dimensionFieldSpecs': [{'name': 'profil', 'dataType': 'STRING'}, {'name': 'codeSite', 'dataType': 'STRING'}, {'name': 'libelleSite', 'dataType': 'STRING'}, {'name': 'timezone', 'dataType': 'STRING'}, {'name': 'plateforme', 'dataType': 'STRING'}, {'name': 'service', 'dataType': 'STRING', 'singleValueField': False}, {'name': 'categorieService', 'dataType': 'STRING', 'singleValueField': False}, {'name': 'codeReseau', 'dataType': 'STRING', 'singleValueField': False}, {'name': 'reseau', 'dataType': 'STRING', 'singleValueField': False}, {'name': 'type', 'dataType': 'STRING', 'singleValueField': False}, {'name': 'nature', 'dataType': 'STRING', 'singleValueField': False}], 'dateTimeFieldSpecs': [{'name': 'dateHeureDebut', 'dataType': 'LONG', 'format': '1:MILLISECONDS:EPOCH', 'granularity': '
 1:MILLISECONDS'}, {'name': 'dateHeureFin', 'dataType': 'LONG', 'format': '1:MILLISECONDS:EPOCH', 'granularity': '1:MILLISECONDS'}, {'name': 'ts', 'dataType': 'TIMESTAMP', 'format': '1:MILLISECONDS:EPOCH', 'granularity': '1:MILLISECONDS'}]}
   2023-10-25 17:13:09,282:ERROR:flask_appbuilder.api:'timestamp'
   Traceback (most recent call last):
     File "/usr/local/lib/python3.9/site-packages/flask_appbuilder/api/__init__.py", line 110, in wraps
       return f(self, *args, **kwargs)
     File "/app/superset/views/base_api.py", line 127, in wraps
       raise ex
     File "/app/superset/views/base_api.py", line 121, in wraps
       duration, response = time_function(f, self, *args, **kwargs)
     File "/app/superset/utils/core.py", line 1526, in time_function
       response = func(*args, **kwargs)
     File "/app/superset/utils/log.py", line 255, in wrapper
       value = f(*args, **kwargs)
     File "/app/superset/databases/api.py", line 728, in table_metadata
       table_info = get_table_metadata(database, table_name, schema_name)
     File "/app/superset/databases/utils.py", line 67, in get_table_metadata
       columns = database.get_columns(table_name, schema_name)
     File "/app/superset/models/core.py", line 847, in get_columns
       return self.db_engine_spec.get_columns(inspector, table_name, schema)
     File "/app/superset/db_engine_specs/base.py", line 1294, in get_columns
       cast(list[SQLAColumnType], inspector.get_columns(table_name, schema))
     File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 497, in get_columns
       col_defs = self.dialect.get_columns(
     File "/usr/local/lib/python3.9/site-packages/pinotdb/sqlalchemy.py", line 415, in get_columns
       columns = [
     File "/usr/local/lib/python3.9/site-packages/pinotdb/sqlalchemy.py", line 418, in <listcomp>
       "type": get_type(spec["dataType"], spec.get("fieldSize")),
     File "/usr/local/lib/python3.9/site-packages/pinotdb/sqlalchemy.py", line 486, in get_type
       return type_map[data_type.lower()]
   KeyError: 'timestamp'
   10.42.9.0 - - [25/Oct/2023:17:13:09 +0000] "GET /api/v1/database/1/table/knada-indicateur-visite-v2/knada-indicateur-visite-v2/ HTTP/1.1" 500 26 "https://superset.knada.rancher.kosmos.fr/dataset/add/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36"`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org