You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "jeanleen (via GitHub)" <gi...@apache.org> on 2024/07/08 06:34:59 UTC
[I] [Invalid view name] [data quality] Invalid view name when use r [dolphinscheduler]
jeanleen opened a new issue, #16286:
URL: https://github.com/apache/dolphinscheduler/issues/16286
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
I want to make sure that the data proves to be compliant when using the oralce database；then bug below appear
User class threw exception: org.apache.spark.sql.AnalysisException: Invalid view name: imc.oa.pdc_xxxx
at org.apache.spark.sql.errors.QueryCompilationErrors$.invalidViewNameError(QueryCompilationErrors.scala:2283)
at org.apache.spark.sql.Dataset.createTempViewCommand(Dataset.scala:3542)
at org.apache.spark.sql.Dataset.createOrReplaceTempView(Dataset.scala:3495)
at org.apache.dolphinscheduler.data.quality.execution.SparkBatchExecution.registerTempView(SparkBatchExecution.java:67)
at org.apache.dolphinscheduler.data.quality.execution.SparkBatchExecution.registerInputTempView(SparkBatchExecution.java:77)
at org.apache.dolphinscheduler.data.quality.execution.SparkBatchExecution.lambda$execute$0(SparkBatchExecution.java:48)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.apache.dolphinscheduler.data.quality.execution.SparkBatchExecution.execute(SparkBatchExecution.java:48)
at org.apache.dolphinscheduler.data.quality.context.DataQualityContext.execute(DataQualityContext.java:62)
at org.apache.dolphinscheduler.data.quality.DataQualityApplication.main(DataQualityApplication.java:70)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:739)
### What you expected to happen
`public static List<BaseConfig> getReaderConfigList(
Map<String, String> inputParameterValue,
DataQualityTaskExecutionContext dataQualityTaskExecutionContext) throws DataQualityException {
List<BaseConfig> readerConfigList = new ArrayList<>();
// all the rule need the source config
if (StringUtils.isNotEmpty(dataQualityTaskExecutionContext.getSourceConnectorType())) {
BaseConnectionParam sourceDataSource =
(BaseConnectionParam) DataSourceUtils.buildConnectionParams(
DbType.of(dataQualityTaskExecutionContext.getSourceType()),
dataQualityTaskExecutionContext.getSourceConnectionParams());
BaseConfig sourceBaseConfig = new BaseConfig();
sourceBaseConfig.setType(dataQualityTaskExecutionContext.getSourceConnectorType());
Map<String, Object> config = new HashMap<>();
if (sourceDataSource != null) {
config.put(DATABASE, inputParameterValue.get(SRC_DATABASE));
config.put(TABLE, inputParameterValue.get(SRC_TABLE));
config.put(URL, DataSourceUtils.getJdbcUrl(DbType.of(dataQualityTaskExecutionContext.getSourceType()),
sourceDataSource));
config.put(USER, sourceDataSource.getUser());
config.put(PASSWORD, ParserUtils.encode(sourceDataSource.getPassword()));
config.put(DRIVER, DataSourceUtils
.getDatasourceDriver(DbType.of(dataQualityTaskExecutionContext.getSourceType())));
**String outputTable = inputParameterValue.get(SRC_DATABASE) + "_" + inputParameterValue.get(SRC_TABLE);**
config.put(OUTPUT_TABLE, outputTable);
inputParameterValue.put(SRC_TABLE, outputTable);
}
sourceBaseConfig.setConfig(config);
readerConfigList.add(sourceBaseConfig);
}`
### How to reproduce
### Anything else
_No response_
### Version
3.1.x
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Invalid view name] [data quality] Invalid view name when use r [dolphinscheduler]
Posted by "CabbageDu (via GitHub)" <gi...@apache.org>.
CabbageDu commented on issue #16286:
URL: https://github.com/apache/dolphinscheduler/issues/16286#issuecomment-2268162429
I meet the same problem,the database name have '-'
User class threw exception: org.apache.spark.sql.AnalysisException: Invalid view name: db-test123;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Invalid view name] [data quality] Invalid view name when use r [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #16286:
URL: https://github.com/apache/dolphinscheduler/issues/16286#issuecomment-2213436369
Can you provide reproduce steps?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Invalid view name] [data quality] Invalid view name when use r [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16286:
URL: https://github.com/apache/dolphinscheduler/issues/16286#issuecomment-2351246258
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Invalid view name] [data quality] Invalid view name when use r [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #16286: [Invalid view name] [data quality] Invalid view name when use r
URL: https://github.com/apache/dolphinscheduler/issues/16286
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Invalid view name] [data quality] Invalid view name when use r [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16286:
URL: https://github.com/apache/dolphinscheduler/issues/16286#issuecomment-2336493581
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org