You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by "doleyzi (via GitHub)" <gi...@apache.org> on 2024/06/15 08:26:38 UTC
[PR] [INLONG-10402][Audit] Audit Service supports the hourly Audit data one day ago [inlong]
doleyzi opened a new pull request, #10428:
URL: https://github.com/apache/inlong/pull/10428
- Fixes #10402
### Motivation
Audit Service supports the hourly Audit data one day ago.
### Modifications
Supports querying historical data that does not exist in the cache from the audit storage and automatic partition management.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-10402][Audit] Audit Service supports the hourly Audit data one day ago [inlong]
Posted by "doleyzi (via GitHub)" <gi...@apache.org>.
doleyzi commented on code in PR #10428:
URL: https://github.com/apache/inlong/pull/10428#discussion_r1642345727
##########
inlong-audit/audit-service/src/main/java/org/apache/inlong/audit/sink/JdbcSink.java:
##########
@@ -153,6 +174,46 @@ protected void createDataSource() {
dataSource = new HikariDataSource(config);
}
+ private void managePartitions() {
+ addPartition();
+ deletePartition();
+ }
+
+ private String formatPartitionName(LocalDate date) {
+ return "p" + date.format(DateTimeFormatter.ofPattern("yyyyMMdd"));
Review Comment:
Fixed
##########
inlong-audit/audit-service/src/main/java/org/apache/inlong/audit/sink/JdbcSink.java:
##########
@@ -153,6 +174,46 @@ protected void createDataSource() {
dataSource = new HikariDataSource(config);
}
+ private void managePartitions() {
+ addPartition();
+ deletePartition();
+ }
+
+ private String formatPartitionName(LocalDate date) {
+ return "p" + date.format(DateTimeFormatter.ofPattern("yyyyMMdd"));
+ }
+
+ private void addPartition() {
+ String partitionName = formatPartitionName(LocalDate.now().plusDays(1));
+ String partitionValue = LocalDate.now().plusDays(2).format(DateTimeFormatter.ofPattern("yyyy-MM-dd"));
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-10402][Audit] Audit Service supports the hourly Audit data one day ago [inlong]
Posted by "doleyzi (via GitHub)" <gi...@apache.org>.
doleyzi commented on code in PR #10428:
URL: https://github.com/apache/inlong/pull/10428#discussion_r1642344340
##########
inlong-audit/audit-service/src/main/java/org/apache/inlong/audit/cache/AbstractCache.java:
##########
@@ -96,38 +95,47 @@ public Cache<String, StatData> getCache() {
public List<StatData> getData(String startTime, String endTime, String inlongGroupId,
String inlongStreamId, String auditId, String auditTag) {
List<StatData> result = new LinkedList<>();
- List<String> keyList = buildCacheKeyList(startTime, endTime, inlongGroupId,
+ List<CacheKeyEntity> keyList = buildCacheKeyList(startTime, endTime, inlongGroupId,
inlongStreamId, auditId, auditTag);
- for (String cacheKey : keyList) {
- StatData statData = cache.getIfPresent(cacheKey);
+ for (CacheKeyEntity cacheKey : keyList) {
+ StatData statData = cache.getIfPresent(cacheKey.getCacheKey());
if (null == statData) {
// Compatible with scenarios where the auditTag openapi parameter can be empty.
- statData = cache.getIfPresent(cacheKey + DEFAULT_AUDIT_TAG);
+ statData = cache.getIfPresent(cacheKey.getCacheKey() + DEFAULT_AUDIT_TAG);
}
if (null != statData) {
result.add(statData);
+ } else {
+ statData = fetchDataFromAuditStorage(cacheKey.getStartTime(), cacheKey.getEndTime(), inlongGroupId,
+ inlongStreamId,
+ auditId, auditTag);
+ result.add(statData);
}
+
}
return result;
}
- private List<String> buildCacheKeyList(String startTime, String endTime, String inlongGroupId,
+ private List<CacheKeyEntity> buildCacheKeyList(String startTime, String endTime, String inlongGroupId,
String inlongStreamId, String auditId, String auditTag) {
- List<String> keyList = new LinkedList<>();
+ List<CacheKeyEntity> keyList = new LinkedList<>();
try {
- SimpleDateFormat dateFormat = new SimpleDateFormat(DATE_FORMAT);
- Date startDate = dateFormat.parse(startTime);
- Date endDate = dateFormat.parse(endTime);
- for (int index = 0; index < MAX_CACHE_KEY_SIZE; index++) {
- Calendar calendar = Calendar.getInstance();
- calendar.setTime(startDate);
- calendar.add(Calendar.MINUTE, index * auditCycle.getValue());
- calendar.set(Calendar.SECOND, 0);
- if (calendar.getTime().compareTo(endDate) > 0) {
+ DateTimeFormatter formatter = DateTimeFormatter.ofPattern(DATE_FORMAT);
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-10402][Audit] Audit Service supports the hourly Audit data one day ago [inlong]
Posted by "aloyszhang (via GitHub)" <gi...@apache.org>.
aloyszhang commented on code in PR #10428:
URL: https://github.com/apache/inlong/pull/10428#discussion_r1642099799
##########
inlong-audit/audit-service/src/main/java/org/apache/inlong/audit/sink/JdbcSink.java:
##########
@@ -153,6 +174,46 @@ protected void createDataSource() {
dataSource = new HikariDataSource(config);
}
+ private void managePartitions() {
+ addPartition();
+ deletePartition();
+ }
+
+ private String formatPartitionName(LocalDate date) {
+ return "p" + date.format(DateTimeFormatter.ofPattern("yyyyMMdd"));
+ }
+
+ private void addPartition() {
+ String partitionName = formatPartitionName(LocalDate.now().plusDays(1));
+ String partitionValue = LocalDate.now().plusDays(2).format(DateTimeFormatter.ofPattern("yyyy-MM-dd"));
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-10402][Audit] Audit Service supports the hourly Audit data one day ago [inlong]
Posted by "aloyszhang (via GitHub)" <gi...@apache.org>.
aloyszhang commented on code in PR #10428:
URL: https://github.com/apache/inlong/pull/10428#discussion_r1642097093
##########
inlong-audit/audit-service/src/main/java/org/apache/inlong/audit/cache/AbstractCache.java:
##########
@@ -96,38 +95,47 @@ public Cache<String, StatData> getCache() {
public List<StatData> getData(String startTime, String endTime, String inlongGroupId,
String inlongStreamId, String auditId, String auditTag) {
List<StatData> result = new LinkedList<>();
- List<String> keyList = buildCacheKeyList(startTime, endTime, inlongGroupId,
+ List<CacheKeyEntity> keyList = buildCacheKeyList(startTime, endTime, inlongGroupId,
inlongStreamId, auditId, auditTag);
- for (String cacheKey : keyList) {
- StatData statData = cache.getIfPresent(cacheKey);
+ for (CacheKeyEntity cacheKey : keyList) {
+ StatData statData = cache.getIfPresent(cacheKey.getCacheKey());
if (null == statData) {
// Compatible with scenarios where the auditTag openapi parameter can be empty.
- statData = cache.getIfPresent(cacheKey + DEFAULT_AUDIT_TAG);
+ statData = cache.getIfPresent(cacheKey.getCacheKey() + DEFAULT_AUDIT_TAG);
}
if (null != statData) {
result.add(statData);
+ } else {
+ statData = fetchDataFromAuditStorage(cacheKey.getStartTime(), cacheKey.getEndTime(), inlongGroupId,
+ inlongStreamId,
+ auditId, auditTag);
+ result.add(statData);
}
+
}
return result;
}
- private List<String> buildCacheKeyList(String startTime, String endTime, String inlongGroupId,
+ private List<CacheKeyEntity> buildCacheKeyList(String startTime, String endTime, String inlongGroupId,
String inlongStreamId, String auditId, String auditTag) {
- List<String> keyList = new LinkedList<>();
+ List<CacheKeyEntity> keyList = new LinkedList<>();
try {
- SimpleDateFormat dateFormat = new SimpleDateFormat(DATE_FORMAT);
- Date startDate = dateFormat.parse(startTime);
- Date endDate = dateFormat.parse(endTime);
- for (int index = 0; index < MAX_CACHE_KEY_SIZE; index++) {
- Calendar calendar = Calendar.getInstance();
- calendar.setTime(startDate);
- calendar.add(Calendar.MINUTE, index * auditCycle.getValue());
- calendar.set(Calendar.SECOND, 0);
- if (calendar.getTime().compareTo(endDate) > 0) {
+ DateTimeFormatter formatter = DateTimeFormatter.ofPattern(DATE_FORMAT);
Review Comment:
DateTimeFormatter initialized once is enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-10402][Audit] Audit Service supports the hourly Audit data one day ago [inlong]
Posted by "aloyszhang (via GitHub)" <gi...@apache.org>.
aloyszhang commented on code in PR #10428:
URL: https://github.com/apache/inlong/pull/10428#discussion_r1642099402
##########
inlong-audit/audit-service/src/main/java/org/apache/inlong/audit/sink/JdbcSink.java:
##########
@@ -153,6 +174,46 @@ protected void createDataSource() {
dataSource = new HikariDataSource(config);
}
+ private void managePartitions() {
+ addPartition();
+ deletePartition();
+ }
+
+ private String formatPartitionName(LocalDate date) {
+ return "p" + date.format(DateTimeFormatter.ofPattern("yyyyMMdd"));
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-10402][Audit] Audit Service supports the hourly Audit data one day ago [inlong]
Posted by "dockerzhang (via GitHub)" <gi...@apache.org>.
dockerzhang merged PR #10428:
URL: https://github.com/apache/inlong/pull/10428
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org