You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@linkis.apache.org by GitBox <gi...@apache.org> on 2022/04/21 15:36:50 UTC
[GitHub] [incubator-linkis] CCweixiao opened a new issue, #2022: [Bug] [1.1.2] Ecp material download error （ecp物料下载报错）
CCweixiao opened a new issue, #2022:
URL: https://github.com/apache/incubator-linkis/issues/2022
### Search before asking
- [X] I searched the [issues](https://github.com/apache/incubator-linkis/issues) and found no similar issues.
### Linkis Component
linkis-public-enhancements
### What happened + What you expected to happen
我的操作如下：
1. 我正常安装完linkis，服务正常启动，运行 linkis-cli-hive 执行hql，正常返回结果。
2. 我关闭linkis的所有服务，重新安装linkis，第二次安装，我没有通过install.sh来进行安装，而是手动解压目录，分发linkis各个服务所需的配置文件，重新启动服务，服务正常启动，在此运行 linkis-cli-hive 执行hql，此时引擎启动报错，引擎运行时产生的目录中，lib 和conf 软链目录无内容，public-service 有报错日志：
[linkis-ps-publicservice.log](https://github.com/apache/incubator-linkis/files/8532343/linkis-ps-publicservice.log)
引擎运行时缺失lib 和conf 所以导致引擎启动失败，文件缺失的原因是，ps服务下载物料失败，根据异常栈定位可能报错的源码：
VersionServiceImpl类中搜索
inputStream.skip(startByte - 1);
查看表linkis_ps_bml_resources_version中数据，第一次数据初始化后
<img width="924" alt="image" src="https://user-images.githubusercontent.com/27730211/164494722-3deca543-654b-4007-8ea6-d5c7c6fa3916.png">
第二次数据初始化：
<img width="913" alt="image" src="https://user-images.githubusercontent.com/27730211/164494900-e27b00c4-46b9-447e-b175-c347cab36da7.png">
首先，表中linkis_ps_bml_resources_version的start_byte end_byte 两个字段的值前后对比，不一致，感觉这里有问题
第二次linkis_ps_bml_resources_version中调用的方法应该是：
VersionServiceImpl.updateVersion
```java
@Override
public String updateVersion(String resourceId, String user, MultipartFile file,
Map<String, Object> params) throws Exception {
ResourceHelper resourceHelper = ResourceHelperFactory.getResourceHelper();
InputStream inputStream = file.getInputStream();
final String resourceIdLock = resourceId.intern();
String fileName = new String(file.getOriginalFilename().getBytes("ISO8859-1"), "UTF-8");
//获取资源的path
String newVersion = params.get("newVersion").toString();
String path = versionDao.getResourcePath(resourceId) + "_" + newVersion;
//上传资源前，需要对resourceId这个字符串的intern进行加锁，这样所有需要更新该资源的用户都会同步
//synchronized (resourceIdLock.intern()){
//资源上传到hdfs
StringBuilder stringBuilder = new StringBuilder();
long size = resourceHelper.upload(path, user, inputStream, stringBuilder, OVER_WRITE);
String md5String = stringBuilder.toString();
String clientIp = params.get("clientIp").toString();
//生成新的version
//String lastVersion = versionDao.getNewestVersion(resourceId);
long startByte = 0L;
//更新resource_version表
ResourceVersion resourceVersion = ResourceVersion.createNewResourceVersion(resourceId, path,
md5String, clientIp, size, newVersion, startByte);
versionDao.insertNewVersion(resourceVersion);
//}
return newVersion;
}
```
此处，startByte=0是写死的，我改成startByte=1 之后，再次重复我的测试步骤后，不报错了，引擎正常运行SQL
My operation is as follows:
1. I installed linkis normally, the service starts normally, run linkis-cli-hive to execute hql, and the result is returned normally.
2. I shut down all linkis services, reinstalled linkis, and for the second installation, I did not install through install.sh, but manually decompressed the directory, distributed the configuration files required by each linkis service, restarted the service, and the service was normal Start, run linkis-cli-hive here to execute hql, then the engine starts to report an error, in the directory generated when the engine is running, the lib and conf soft chain directories have no content, and the public-service has an error log:
[linkis-ps-publicservice.log](https://github.com/apache/incubator-linkis/files/8532343/linkis-ps-publicservice.log)
When the engine is running, the lib and conf are missing, so the engine fails to start. The reason for the missing files is that the ps service fails to download materials. According to the exception stack, locate the source code that may report an error:
Search in VersionServiceImpl class: inputStream.skip(startByte - 1);
View the data in the linkis_ps_bml_resources_version table, after the first data initializatio
<img width="924" alt="image" src="https://user-images.githubusercontent.com/27730211/164494722-3deca543-654b-4007-8ea6-d5c7c6fa3916.png">
Second data initialization:
<img width="913" alt="image" src="https://user-images.githubusercontent.com/27730211/164494900-e27b00c4-46b9-447e-b175-c347cab36da7.png">
首先，表中linkis_ps_bml_resources_version的start_byte end_byte 两个字段的值前后对比，不一致，感觉这里有问题
First of all, the values of the two fields start_byte end_byte of linkis_ps_bml_resources_version in the table are compared before and after, and they are inconsistent. I feel that there is a problem here.
VersionServiceImpl.updateVersion
```java
@Override
public String updateVersion(String resourceId, String user, MultipartFile file,
Map<String, Object> params) throws Exception {
ResourceHelper resourceHelper = ResourceHelperFactory.getResourceHelper();
InputStream inputStream = file.getInputStream();
final String resourceIdLock = resourceId.intern();
String fileName = new String(file.getOriginalFilename().getBytes("ISO8859-1"), "UTF-8");
//获取资源的path
String newVersion = params.get("newVersion").toString();
String path = versionDao.getResourcePath(resourceId) + "_" + newVersion;
//上传资源前，需要对resourceId这个字符串的intern进行加锁，这样所有需要更新该资源的用户都会同步
//synchronized (resourceIdLock.intern()){
//资源上传到hdfs
StringBuilder stringBuilder = new StringBuilder();
long size = resourceHelper.upload(path, user, inputStream, stringBuilder, OVER_WRITE);
String md5String = stringBuilder.toString();
String clientIp = params.get("clientIp").toString();
//生成新的version
//String lastVersion = versionDao.getNewestVersion(resourceId);
long startByte = 0L;
//更新resource_version表
ResourceVersion resourceVersion = ResourceVersion.createNewResourceVersion(resourceId, path,
md5String, clientIp, size, newVersion, startByte);
versionDao.insertNewVersion(resourceVersion);
//}
return newVersion;
}
```
Here, startByte=0 is hard-coded. After I changed it to startByte=1, after repeating my test steps again, no error is reported, and the engine runs SQL normally.
### Relevent platform
linux
### Reproduction script
none
### Anything else
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org
For additional commands, e-mail: dev-help@linkis.apache.org
[GitHub] [incubator-linkis] peacewong closed issue #2022: [Bug] [1.1.2] Ecp material download error （ecp物料下载报错）
Posted by GitBox <gi...@apache.org>.
peacewong closed issue #2022: [Bug] [1.1.2] Ecp material download error （ecp物料下载报错）
URL: https://github.com/apache/incubator-linkis/issues/2022
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org