You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/04/25 08:39:00 UTC
[jira] [Commented] (KYLIN-3347) QueryService Exception when using
calcite function ex : {fn CURRENT_TIMESTAMP(0)}
[ https://issues.apache.org/jira/browse/KYLIN-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451878#comment-16451878 ]
ASF GitHub Bot commented on KYLIN-3347:
---------------------------------------
imsong closed pull request #134: KYLIN-3347 : calcite function QueryService Exception
URL: https://github.com/apache/kylin/pull/134
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/core-metadata/src/main/java/org/apache/kylin/measure/basic/DoubleIngester.java b/core-metadata/src/main/java/org/apache/kylin/measure/basic/DoubleIngester.java
index 90ecb0d583..3707fdcc5d 100644
--- a/core-metadata/src/main/java/org/apache/kylin/measure/basic/DoubleIngester.java
+++ b/core-metadata/src/main/java/org/apache/kylin/measure/basic/DoubleIngester.java
@@ -18,6 +18,7 @@
package org.apache.kylin.measure.basic;
+import java.math.BigDecimal;
import java.util.Map;
import org.apache.kylin.common.util.Dictionary;
@@ -35,7 +36,7 @@ public Double valueOf(String[] values, MeasureDesc measureDesc, Map<TblColRef, D
if (values[0] == null || values[0].length() == 0)
return new Double(0);
else
- return Double.parseDouble(values[0]);
+ return new BigDecimal(values[0]).doubleValue();
}
@Override
diff --git a/core-metadata/src/main/java/org/apache/kylin/measure/basic/LongIngester.java b/core-metadata/src/main/java/org/apache/kylin/measure/basic/LongIngester.java
index 793acf2f78..990adfa373 100644
--- a/core-metadata/src/main/java/org/apache/kylin/measure/basic/LongIngester.java
+++ b/core-metadata/src/main/java/org/apache/kylin/measure/basic/LongIngester.java
@@ -18,6 +18,7 @@
package org.apache.kylin.measure.basic;
+import java.math.BigDecimal;
import java.util.Map;
import org.apache.kylin.common.util.Dictionary;
@@ -35,7 +36,7 @@ public Long valueOf(String[] values, MeasureDesc measureDesc, Map<TblColRef, Dic
if (values[0] == null || values[0].length() == 0)
return new Long(0L);
else
- return Long.valueOf(values[0]);
+ return new BigDecimal(values[0]).longValue();
}
@Override
diff --git a/query/src/main/java/org/apache/kylin/query/util/DefaultQueryTransformer.java b/query/src/main/java/org/apache/kylin/query/util/DefaultQueryTransformer.java
index 671d9dbf35..6b88225693 100644
--- a/query/src/main/java/org/apache/kylin/query/util/DefaultQueryTransformer.java
+++ b/query/src/main/java/org/apache/kylin/query/util/DefaultQueryTransformer.java
@@ -50,6 +50,8 @@
private static final Pattern PIN_SUM_OF_FN_CONVERT = Pattern
.compile(S0 + "SUM" + S0 + "\\(" + S0 + "\\{\\s*fn" + SM + "convert" + S0 + "\\(" + S0 + "([^\\s,]+)" + S0
+ "," + S0 + "SQL_DOUBLE" + S0 + "\\)" + S0 + "\\}" + S0 + "\\)", Pattern.CASE_INSENSITIVE);
+ private static final Pattern PTN_HAVING_FUNCTION = Pattern.compile("\\{fn" + "(.*?)" + "\\}",
+ Pattern.CASE_INSENSITIVE);
@Override
public String transform(String sql, String project, String defaultSchema) {
@@ -138,7 +140,13 @@ public String transform(String sql, String project, String defaultSchema) {
sql = sql.substring(0, m.start(1)) + "'" + value + "'" + sql.substring(m.end(3));
}
-
+ while (true) {
+ m = PTN_HAVING_FUNCTION.matcher(sql);
+ if (!m.find())
+ break;
+ sql = sql.substring(0, m.start()) + m.group(1) + sql.substring(m.end());
+ }
+
return sql;
}
}
diff --git a/query/src/test/java/org/apache/kylin/query/util/DefaultQueryTransformerTest.java b/query/src/test/java/org/apache/kylin/query/util/DefaultQueryTransformerTest.java
index 8ce10174e7..a53e02029f 100644
--- a/query/src/test/java/org/apache/kylin/query/util/DefaultQueryTransformerTest.java
+++ b/query/src/test/java/org/apache/kylin/query/util/DefaultQueryTransformerTest.java
@@ -100,5 +100,9 @@ public void functionEscapeTransform() throws Exception {
String fnConvertSumSql = "select {fn EXTRACT(YEAR from PART_DT)} from KYLIN_SALES";
String correctSql = transformer.transform(fnConvertSumSql, "", "");
assertTrue("select EXTRACT(YEAR from PART_DT) from KYLIN_SALES".equalsIgnoreCase(correctSql));
+
+ fnConvertSumSql = "SELECT {fn CURRENT_TIMESTAMP(0)}";
+ correctSql = transformer.transform(fnConvertSumSql, "", "");
+ assertTrue("SELECT CURRENT_TIMESTAMP".equalsIgnoreCase(correctSql));
}
}
\ No newline at end of file
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
> QueryService Exception when using calcite function ex : {fn CURRENT_TIMESTAMP(0)}
> ---------------------------------------------------------------------------------
>
> Key: KYLIN-3347
> URL: https://issues.apache.org/jira/browse/KYLIN-3347
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
> Affects Versions: v2.3.0, v2.3.1
> Reporter: Hokyung Song
> Priority: Major
>
> Hello. :)
> I'm trying to upgrade kylin version to v2.3.x
> But, when I use cacite function query, it occured error. (v2.2.0 is ok)
> In my opinion, the reason is that regular expression matcher of *DefaultQueryTransformer.java* has changed at v2.3.x
> I will contribute a path on this problem soon.
> thanks
> (ps. I'm using Kylin with Tableau 10)
>
> stack trace as follows (v2.3.0)
> {code:java}
> 2018-04-23 14:22:12,475 INFO [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] service.QueryService:429 : The original query: SELECT {fn CURRENT_TIMESTAMP(0)} AS "COL"
> 2018-04-23 14:22:12,526 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] runtime.CalciteException:59 : org.apache.calcite.sql.validate.SqlValidatorException: Function '{fn CURRENT_TIMESTAMP}' is not defined
> 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] runtime.CalciteException:61 : org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to line 1, column 32: Function '{fn CURRENT_TIMESTAMP}' is not defined
> 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] service.QueryService:536 : Exception while executing query
> java.sql.SQLException: Error while preparing statement [SELECT {fn CURRENT_TIMESTAMP(0)} AS "COL"]
> at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
> at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
> at org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:203)
> at org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:185)
> at org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:86)
> at org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:168)
> at org.apache.kylin.rest.service.QueryService.getPrepareOnlySqlResponse(QueryService.java:994)
> at org.apache.kylin.rest.service.QueryService.execute(QueryService.java:916)
> at org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:661)
> at org.apache.kylin.rest.service.QueryService.query(QueryService.java:188)
> at org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:496)
> at org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:455)
> at org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:390)
> at org.apache.kylin.rest.controller.QueryController.prepareQuery(QueryController.java:99)
> at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
> at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
> at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
> at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
> at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
> at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
> at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
> at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
> at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
> at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215)
> at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)
> at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
> at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
> at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
> Caused by: org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to line 1, column 32: Function '{fn CURRENT_TIMESTAMP}' is not defined
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:804)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:789)
> at org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4393)
> at org.apache.calcite.sql.SqlJdbcFunctionCall.rewriteCall(SqlJdbcFunctionCall.java:440)
> at org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1119)
> at org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1097)
> at org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1126)
> at org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1097)
> at org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:888)
> at org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:605)
> at org.apache.calcite.prepare.CalcitePrepareImpl.parse_(CalcitePrepareImpl.java:297)
> at org.apache.calcite.prepare.CalcitePrepareImpl.parse(CalcitePrepareImpl.java:267)
> at org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:630)
> at org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:617)
> at org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:214)
> at org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:196)
> ... 85 more
> Caused by: org.apache.calcite.sql.validate.SqlValidatorException: Function '{fn CURRENT_TIMESTAMP}' is not defined
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
> at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:572)
> ... 101 more
> 2018-04-23 14:22:12,537 INFO [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] service.QueryService:328 :
> ==========================[QUERY]===============================
> Query Id: f8811967-be03-4fd0-b11f-85c68de0abcb
> SQL: SELECT {fn CURRENT_TIMESTAMP(0)} AS "COL"
> User: ADMIN
> Success: false
> Duration: 0.073
> Project: Sandbox
> Realization Names: []
> Cuboid Ids: []
> Total scan count: 0
> Total scan bytes: 0
> Result row count: 0
> Accept Partial: false
> Is Partial Result: false
> Hit Exception Cache: false
> Storage cache used: false
> Is Query Push-Down: false
> Is Prepare: true
> Trace URL: null
> Message: Error while preparing statement [SELECT {fn CURRENT_TIMESTAMP(0)} AS "COL"]
> ==========================[QUERY]==============================={code}
>
> stack trace as follows (v2.2.0 is ok)
> {code:java}
> 2018-04-23 11:48:07,731 INFO [Query 341498a0-251b-4720-96ba-d99c43effd74-132] service.QueryService:397 : The original query: SELECT {fn CURRENT_TIMESTAMP(0)} AS "COL"
> 2018-04-23 11:48:07,732 INFO [Query 341498a0-251b-4720-96ba-d99c43effd74-132] schema.OLAPSchemaFactory:116 : Schema json:{
> ....
> 2018-04-23 11:48:07,706 INFO [Query 194d5044-48a0-4b2e-96ab-a0a987b23b0a-149] service.QueryService:547 : The corrected query: SELECT CURRENT_TIMESTAMP(0) AS "COL"
> 2018-04-23 11:48:07,710 INFO [Query 194d5044-48a0-4b2e-96ab-a0a987b23b0a-149] service.QueryService:964 : Processed rows for each storageContext:
> 2018-04-23 11:48:07,711 INFO [Query 194d5044-48a0-4b2e-96ab-a0a987b23b0a-149] service.QueryService:428 : Stats of SQL response: isException: false, duration: 13, total scan count 0
> 2018-04-23 11:48:07,711 DEBUG [Query 194d5044-48a0-4b2e-96ab-a0a987b23b0a-149] util.CheckUtil:35 : query is too lightweight with duration: 13 (threshold 2000), scan count: 0 (threshold 10240), scan bytes: 0 (threshold 1048576)
> 2018-04-23 11:48:07,712 DEBUG [Query 194d5044-48a0-4b2e-96ab-a0a987b23b0a-149] security.KylinAclPermissionEvaluator:97 : Checking permission 'ADMINISTRATION' for object 'org.springframework.security.acls.domain.ObjectIdentityImpl[Type: org.apache.kylin.metadata.project.ProjectInstance; Identifier: fdc7c850-115e-4209-a8da-b1fc874ae76c]'
> 2018-04-23 11:48:07,715 DEBUG [Query 194d5044-48a0-4b2e-96ab-a0a987b23b0a-149] security.KylinAclPermissionEvaluator:120 : Returning false - no ACLs apply for this principal
> 2018-04-23 11:48:07,716 DEBUG [Query 194d5044-48a0-4b2e-96ab-a0a987b23b0a-149] security.KylinAclPermissionEvaluator:97 : Checking permission 'MANAGEMENT' for object 'org.springframework.security.acls.domain.ObjectIdentityImpl[Type: org.apache.kylin.metadata.project.ProjectInstance; Identifier: fdc7c850-115e-4209-a8da-b1fc874ae76c]'
> 2018-04-23 11:48:07,718 DEBUG [Query 194d5044-48a0-4b2e-96ab-a0a987b23b0a-149] security.KylinAclPermissionEvaluator:120 : Returning false - no ACLs apply for this principal
> 2018-04-23 11:48:07,719 DEBUG [Query 194d5044-48a0-4b2e-96ab-a0a987b23b0a-149] security.KylinAclPermissionEvaluator:97 : Checking permission 'OPERATION' for object 'org.springframework.security.acls.domain.ObjectIdentityImpl[Type: org.apache.kylin.metadata.project.ProjectInstance; Identifier: fdc7c850-115e-4209-a8da-b1fc874ae76c]'
> 2018-04-23 11:48:07,721 DEBUG [Query 194d5044-48a0-4b2e-96ab-a0a987b23b0a-149] security.KylinAclPermissionEvaluator:120 : Returning false - no ACLs apply for this principal
> 2018-04-23 11:48:07,721 DEBUG [Query 194d5044-48a0-4b2e-96ab-a0a987b23b0a-149] security.KylinAclPermissionEvaluator:97 : Checking permission 'READ' for object 'org.springframework.security.acls.domain.ObjectIdentityImpl[Type: org.apache.kylin.metadata.project.ProjectInstance; Identifier: fdc7c850-115e-4209-a8da-b1fc874ae76c]'
> 2018-04-23 11:48:07,723 DEBUG [Query 194d5044-48a0-4b2e-96ab-a0a987b23b0a-149] security.KylinAclPermissionEvaluator:107 : Access is granted
> 2018-04-23 11:48:07,723 INFO [Query 194d5044-48a0-4b2e-96ab-a0a987b23b0a-149] service.QueryService:313 :
> ==========================[QUERY]===============================
> Query Id: 194d5044-48a0-4b2e-96ab-a0a987b23b0a
> SQL: SELECT {fn CURRENT_TIMESTAMP(0)} AS "COL"
> User: ANALYST
> Success: true
> Duration: 0.013
> Project: DA
> Realization Names: []
> Cuboid Ids: []
> Total scan count: 0
> Total scan bytes: 0
> Result row count: 0
> Accept Partial: false
> Is Partial Result: false
> Hit Exception Cache: false
> Storage cache used: false
> Is Query Push-Down: false
> Message: null
> ==========================[QUERY]==============================={code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)