You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by GitBox <gi...@apache.org> on 2022/10/10 15:26:27 UTC
[GitHub] [inlong] GanfengTan opened a new pull request, #6132: [INLONG-6131][Agent] Support file filtering by condition
GanfengTan opened a new pull request, #6132:
URL: https://github.com/apache/inlong/pull/6132
Filter file names by file attributes.
- Fixes #6131
### Motivation
1. Filter files by file properties, for example: logs of k8s.
2. Add rule filter file.
### Modifications
1. Add file utils.
### Verifying this change
*(Please pick either of the following options)*
- [x] This change is a trivial rework/code cleanup without any test coverage.
- [ ] This change is already covered by existing tests, such as:
*(please describe tests)*
- [ ] This change added tests and can be verified as follows:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #6132: [INLONG-6131][Agent] Support file filtering by condition
Posted by GitBox <gi...@apache.org>.
EMsnap commented on code in PR #6132:
URL: https://github.com/apache/inlong/pull/6132#discussion_r991777389
##########
inlong-agent/agent-common/src/main/java/org/apache/inlong/agent/pojo/FileJob.java:
##########
@@ -41,13 +41,16 @@ public class FileJob {
private String envList;
- private List<Map<String, String>> metaFields;
+ // json sting, List<Map<String,string>>
Review Comment:
string
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #6132: [INLONG-6131][Agent] Support file filtering by condition
Posted by GitBox <gi...@apache.org>.
EMsnap commented on code in PR #6132:
URL: https://github.com/apache/inlong/pull/6132#discussion_r991783022
##########
inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/utils/FileDataUtils.java:
##########
@@ -65,4 +81,56 @@ public static boolean isJSON(String json) {
return isJson;
}
+ /**
+ * Filter file by conditions
+ */
+ public static Collection<File> filterFile(Collection<File> allFiles, JobProfile jobConf) {
+ // filter file by labels
+ Collection<File> files = null;
+ try {
+ files = filterByLabels(allFiles, jobConf);
+ } catch (IOException e) {
+ LOGGER.error("filter file error: ", e);
+ }
+ return files;
+ }
+
+ /**
+ * Filter file by labels if standard log for k8s
+ */
+ private static Collection<File> filterByLabels(Collection<File> allFiles, JobProfile jobConf) throws IOException {
+ Map<String, String> labelsMap = MetaDataUtils.getPodLabels(jobConf);
+ if (labelsMap.isEmpty()) {
+ return allFiles;
+ }
+ Collection<File> standardK8sLogFiles = new ArrayList<>();
+ Iterator<File> iterator = allFiles.iterator();
+ KubernetesClient client = PluginUtils.getKubernetesClient();
+ while (iterator.hasNext()) {
+ File file = iterator.next();
+ Map<String, String> logInfo = MetaDataUtils.getLogInfo(file.getName());
+ if (logInfo.isEmpty()) {
+ continue;
+ }
+ PodResource podResource = client.pods().inNamespace(logInfo.get(NAMESPACE))
+ .withName(logInfo.get(POD_NAME));
+ if (Objects.isNull(podResource)) {
+ continue;
+ }
+ Pod pod = podResource.get();
+ Map<String, String> podLabels = pod.getMetadata().getLabels();
+ AtomicBoolean filterLabelStatus = new AtomicBoolean(true);
Review Comment:
suggest extract the filter logic as a method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap merged pull request #6132: [INLONG-6131][Agent] Support file filtering by condition
Posted by GitBox <gi...@apache.org>.
EMsnap merged PR #6132:
URL: https://github.com/apache/inlong/pull/6132
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] GanfengTan commented on a diff in pull request #6132: [INLONG-6131][Agent] Support file filtering by condition
Posted by GitBox <gi...@apache.org>.
GanfengTan commented on code in PR #6132:
URL: https://github.com/apache/inlong/pull/6132#discussion_r991791862
##########
inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/utils/FileDataUtils.java:
##########
@@ -65,4 +81,56 @@ public static boolean isJSON(String json) {
return isJson;
}
+ /**
+ * Filter file by conditions
+ */
+ public static Collection<File> filterFile(Collection<File> allFiles, JobProfile jobConf) {
+ // filter file by labels
+ Collection<File> files = null;
+ try {
+ files = filterByLabels(allFiles, jobConf);
+ } catch (IOException e) {
+ LOGGER.error("filter file error: ", e);
+ }
+ return files;
+ }
+
+ /**
+ * Filter file by labels if standard log for k8s
+ */
+ private static Collection<File> filterByLabels(Collection<File> allFiles, JobProfile jobConf) throws IOException {
+ Map<String, String> labelsMap = MetaDataUtils.getPodLabels(jobConf);
+ if (labelsMap.isEmpty()) {
+ return allFiles;
+ }
+ Collection<File> standardK8sLogFiles = new ArrayList<>();
+ Iterator<File> iterator = allFiles.iterator();
+ KubernetesClient client = PluginUtils.getKubernetesClient();
+ while (iterator.hasNext()) {
+ File file = iterator.next();
+ Map<String, String> logInfo = MetaDataUtils.getLogInfo(file.getName());
+ if (logInfo.isEmpty()) {
+ continue;
+ }
+ PodResource podResource = client.pods().inNamespace(logInfo.get(NAMESPACE))
+ .withName(logInfo.get(POD_NAME));
+ if (Objects.isNull(podResource)) {
+ continue;
+ }
+ Pod pod = podResource.get();
+ Map<String, String> podLabels = pod.getMetadata().getLabels();
+ AtomicBoolean filterLabelStatus = new AtomicBoolean(true);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] GanfengTan commented on a diff in pull request #6132: [INLONG-6131][Agent] Support file filtering by condition
Posted by GitBox <gi...@apache.org>.
GanfengTan commented on code in PR #6132:
URL: https://github.com/apache/inlong/pull/6132#discussion_r991781225
##########
inlong-agent/agent-common/src/main/java/org/apache/inlong/agent/pojo/FileJob.java:
##########
@@ -41,13 +41,16 @@ public class FileJob {
private String envList;
- private List<Map<String, String>> metaFields;
+ // json sting, List<Map<String,string>>
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org