You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "winterhazel (via GitHub)" <gi...@apache.org> on 2025/08/13 13:03:46 UTC

[PR] Events listing and deletion optimizations [cloudstack]

winterhazel opened a new pull request, #11443:
URL: https://github.com/apache/cloudstack/pull/11443

   ### Description
   
   The `listEvents` and `deleteEvents` APIs execute some non-optimized queries, which results in very long response times in environments that have many events.
   
   Some optimizations were performed in these two workflows in order to reduce response times. The listing now uses a covering index, and does not perform a unecessary `DISTINCT` anymore. The deletion is now performed in batches. The batch size is defined by the global setting `delete.query.batch.size`. 
   
   ### Types of changes
   
   - [ ] Breaking change (fix or feature that would cause existing functionality to change)
   - [ ] New feature (non-breaking change which adds functionality)
   - [ ] Bug fix (non-breaking change which fixes an issue)
   - [X] Enhancement (improves an existing feature and functionality)
   - [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
   - [ ] build/CI
   - [ ] test (unit or integration test code)
   
   ### Feature/Enhancement Scale or Bug Severity
   
   #### Feature/Enhancement Scale
   
   - [X] Major
   - [ ] Minor
   
   ### How Has This Been Tested?
   
   #### Events listing
   
   In a test environment with 2 million event entries, I called `listEvents` multiple times passing different combinations of `account`, `archived`, `domainid`, `duration`, `enddate`, `entrytime`, `id`, `keyword`, `level`, `projectid`, `resourceid`, `resourcetype`, `startdate`, `startid` and `type`. For almost all the listings, I got a more reasonable response time (between 0.1 and 3 seconds depending on the filters) compared to before (30+ seconds). The only exception is the `keyword` parameter, which is not optimizable with a index because it performs a `column LIKE "%keyword%"` for 3 different columns.
   
   ##### Access control
   
   I called `listEvents` and verified that:
   
   - root admins can list all events;
   - domain admins can only list events from their domain;
   - users can only list their events.
   
   #### Events deletion
   
   1. I assigned a value to `delete.query.batch.size`, enabling batch delete;
   2. I called `deleteEvents` to remove all events before `2025-01-01`. Then, I verified that they were removed successfully;
   3. I assigned a value to `event.purge.delay`, enabling the automatic deletion of events older than 15 days. After the deletion task executed, I verified that events older than 15 days were removed successfully.
   
   ##### Access control
   
   I called `deleteEvents` and verified that:
   
   - root admins can delete any events; 
   - domain admins can only delete events from their domain;
   - users can only delete their events.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-4340115461

   <b>[SF] Trillian test result (tid-15961)</b>
   Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
   Total time taken: 50779 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11443-t15961-kvm-ol8.zip
   Smoke tests completed. 151 look OK, 0 have errors, 0 did not run
   Only failed and skipped tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "winterhazel (via GitHub)" <gi...@apache.org>.
winterhazel commented on code in PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#discussion_r3170296744


##########
engine/schema/src/main/java/com/cloud/event/dao/EventDaoImpl.java:
##########
@@ -33,83 +38,90 @@
 
 @Component
 public class EventDaoImpl extends GenericDaoBase<EventVO, Long> implements EventDao {
-    protected final SearchBuilder<EventVO> CompletedEventSearch;
+
     protected final SearchBuilder<EventVO> ToArchiveOrDeleteEventSearch;
 
     public EventDaoImpl() {
-        CompletedEventSearch = createSearchBuilder();
-        CompletedEventSearch.and("state", CompletedEventSearch.entity().getState(), SearchCriteria.Op.EQ);
-        CompletedEventSearch.and("startId", CompletedEventSearch.entity().getStartId(), SearchCriteria.Op.EQ);
-        CompletedEventSearch.and("archived", CompletedEventSearch.entity().getArchived(), Op.EQ);
-        CompletedEventSearch.done();
-
         ToArchiveOrDeleteEventSearch = createSearchBuilder();
+        ToArchiveOrDeleteEventSearch.select("id", SearchCriteria.Func.NATIVE, ToArchiveOrDeleteEventSearch.entity().getId());
         ToArchiveOrDeleteEventSearch.and("id", ToArchiveOrDeleteEventSearch.entity().getId(), Op.IN);
         ToArchiveOrDeleteEventSearch.and("type", ToArchiveOrDeleteEventSearch.entity().getType(), Op.EQ);
-        ToArchiveOrDeleteEventSearch.and("accountIds", ToArchiveOrDeleteEventSearch.entity().getAccountId(), Op.IN);
+        ToArchiveOrDeleteEventSearch.and("accountId", ToArchiveOrDeleteEventSearch.entity().getAccountId(), Op.EQ);
+        ToArchiveOrDeleteEventSearch.and("domainIds", ToArchiveOrDeleteEventSearch.entity().getDomainId(), Op.IN);
         ToArchiveOrDeleteEventSearch.and("createdDateB", ToArchiveOrDeleteEventSearch.entity().getCreateDate(), Op.BETWEEN);
         ToArchiveOrDeleteEventSearch.and("createdDateL", ToArchiveOrDeleteEventSearch.entity().getCreateDate(), Op.LTEQ);
+        ToArchiveOrDeleteEventSearch.and("createdDateLT", ToArchiveOrDeleteEventSearch.entity().getCreateDate(), Op.LT);
         ToArchiveOrDeleteEventSearch.and("archived", ToArchiveOrDeleteEventSearch.entity().getArchived(), Op.EQ);
         ToArchiveOrDeleteEventSearch.done();
     }
 
-    @Override
-    public List<EventVO> searchAllEvents(SearchCriteria<EventVO> sc, Filter filter) {
-        return listIncludingRemovedBy(sc, filter);
-    }
-
-    @Override
-    public List<EventVO> listOlderEvents(Date oldTime) {
-        if (oldTime == null)
-            return null;
-        SearchCriteria<EventVO> sc = createSearchCriteria();
-        sc.addAnd("createDate", SearchCriteria.Op.LT, oldTime);
-        sc.addAnd("archived", SearchCriteria.Op.EQ, false);
-        return listIncludingRemovedBy(sc, null);
-    }
-
-    @Override
-    public EventVO findCompletedEvent(long startId) {
-        SearchCriteria<EventVO> sc = CompletedEventSearch.create();
-        sc.setParameters("state", State.Completed);
-        sc.setParameters("startId", startId);
-        sc.setParameters("archived", false);
-        return findOneIncludingRemovedBy(sc);
-    }
-
-    @Override
-    public List<EventVO> listToArchiveOrDeleteEvents(List<Long> ids, String type, Date startDate, Date endDate, List<Long> accountIds) {
+    private SearchCriteria<EventVO> createEventSearchCriteria(List<Long> ids, String type, Date startDate, Date endDate,
+                                                              Date limitDate, Long accountId, List<Long> domainIds) {
         SearchCriteria<EventVO> sc = ToArchiveOrDeleteEventSearch.create();
-        if (ids != null) {
-            sc.setParameters("id", ids.toArray(new Object[ids.size()]));
+
+        if (CollectionUtils.isNotEmpty(ids)) {
+            sc.setParameters("id", ids.toArray(new Object[0]));
         }
-        if (type != null) {
-            sc.setParameters("type", type);
+        if (CollectionUtils.isNotEmpty(domainIds)) {
+            sc.setParameters("domainIds", domainIds.toArray(new Object[0]));
         }
         if (startDate != null && endDate != null) {
             sc.setParameters("createdDateB", startDate, endDate);
         } else if (endDate != null) {
             sc.setParameters("createdDateL", endDate);
         }
-        if (accountIds != null && !accountIds.isEmpty()) {
-            sc.setParameters("accountIds", accountIds.toArray(new Object[accountIds.size()]));
-        }
+        sc.setParametersIfNotNull("accountId", accountId);
+        sc.setParametersIfNotNull("createdDateLT", limitDate);
+        sc.setParametersIfNotNull("type", type);
         sc.setParameters("archived", false);
-        return search(sc, null);
+
+        return sc;
     }
 
     @Override
-    public void archiveEvents(List<EventVO> events) {
-        if (events != null && !events.isEmpty()) {
-            TransactionLegacy txn = TransactionLegacy.currentTxn();
-            txn.start();
-            for (EventVO event : events) {
-                event = lockRow(event.getId(), true);
-                event.setArchived(true);
-                update(event.getId(), event);
-                txn.commit();
+    public long archiveEvents(List<Long> ids, String type, Date startDate, Date endDate, Long accountId, List<Long> domainIds,
+                              long limitPerQuery) {
+        SearchCriteria<EventVO> sc = createEventSearchCriteria(ids, type, startDate, endDate, null, accountId, domainIds);
+        Filter filter = null;
+        if (limitPerQuery > 0) {
+            filter = new Filter(limitPerQuery);
+        }
+
+        long archived;
+        long totalArchived = 0L;
+
+        do {
+            List<EventVO> events = search(sc, filter);
+            if (events.isEmpty()) {
+                break;
             }
-            txn.close();
+
+            archived = archiveEventsInternal(events);
+            totalArchived += archived;
+        } while (limitPerQuery > 0 && archived >= limitPerQuery);
+
+        return totalArchived;
+    }
+
+    @DB
+    private long archiveEventsInternal(List<EventVO> events) {
+        final String idsAsString = events.stream()
+                .map(e -> Long.toString(e.getId()))
+                .collect(Collectors.joining(","));
+        final String query = String.format("UPDATE event SET archived=true WHERE id IN (%s)", idsAsString);
+
+        try (TransactionLegacy txn = TransactionLegacy.currentTxn();
+             PreparedStatement pstmt = txn.prepareStatement(query)) {

Review Comment:
   This suggestion does not make sense to me



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "Damans227 (via GitHub)" <gi...@apache.org>.
Damans227 commented on code in PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#discussion_r4272717121


##########
server/src/main/java/com/cloud/server/ManagementServerImpl.java:
##########
@@ -1225,57 +1224,64 @@ protected void checkPortParameters(final String publicPort, final String private
     }
 
     @Override
-    public boolean archiveEvents(final ArchiveEventsCmd cmd) {
-        final Account caller = getCaller();
-        final List<Long> ids = cmd.getIds();
-        boolean result = true;
-        List<Long> permittedAccountIds = new ArrayList<>();
+    public boolean archiveEvents(ArchiveEventsCmd cmd) {
+        List<Long> ids = cmd.getIds();
+        String type = cmd.getType();
+        Date startDate = cmd.getStartDate();
+        Date endDate = cmd.getEndDate();
+
+        validateEventOperationParameters(ids, type, endDate, startDate);
+
+        Long accountId = null;
+        List<Long> domainIds = null;
+        Account caller = getCaller();
+        Account.Type callerType = caller.getType();
 
-        if (_accountService.isNormalUser(caller.getId()) || caller.getType() == Account.Type.PROJECT) {
-            permittedAccountIds.add(caller.getId());
+        if (callerType == Account.Type.NORMAL || callerType == Account.Type.PROJECT) {
+            accountId = caller.getId();
         } else {
-            final DomainVO domain = _domainDao.findById(caller.getDomainId());
-            final List<Long> permittedDomainIds = _domainDao.getDomainChildrenIds(domain.getPath());
-            permittedAccountIds = _accountDao.getAccountIdsForDomains(permittedDomainIds);
+            domainIds = _domainDao.getDomainAndChildrenIds(caller.getDomainId());
         }
 
-        final List<EventVO> events = _eventDao.listToArchiveOrDeleteEvents(ids, cmd.getType(), cmd.getStartDate(), cmd.getEndDate(), permittedAccountIds);
-        final ControlledEntity[] sameOwnerEvents = events.toArray(new ControlledEntity[events.size()]);
-        _accountMgr.checkAccess(CallContext.current().getCallingAccount(), null, false, sameOwnerEvents);
+        long totalArchived = _eventDao.archiveEvents(ids, type, startDate, endDate, accountId, domainIds,
+                ConfigurationManagerImpl.DELETE_QUERY_BATCH_SIZE.value());
 
-        if (ids != null && events.size() < ids.size()) {
-            return false;
-        }
-        _eventDao.archiveEvents(events);
-        return result;
+        return totalArchived > 0;

Review Comment:
   does this still need the change you mentioned earlier? if i pass three ids and only one is mine, it archives that one and says success. before it failed and changed nothing, same for delete below



##########
server/src/main/java/com/cloud/server/ManagementServerImpl.java:
##########
@@ -1225,57 +1224,64 @@ protected void checkPortParameters(final String publicPort, final String private
     }
 
     @Override
-    public boolean archiveEvents(final ArchiveEventsCmd cmd) {
-        final Account caller = getCaller();
-        final List<Long> ids = cmd.getIds();
-        boolean result = true;
-        List<Long> permittedAccountIds = new ArrayList<>();
+    public boolean archiveEvents(ArchiveEventsCmd cmd) {
+        List<Long> ids = cmd.getIds();
+        String type = cmd.getType();
+        Date startDate = cmd.getStartDate();
+        Date endDate = cmd.getEndDate();
+
+        validateEventOperationParameters(ids, type, endDate, startDate);
+
+        Long accountId = null;
+        List<Long> domainIds = null;
+        Account caller = getCaller();
+        Account.Type callerType = caller.getType();
 
-        if (_accountService.isNormalUser(caller.getId()) || caller.getType() == Account.Type.PROJECT) {
-            permittedAccountIds.add(caller.getId());
+        if (callerType == Account.Type.NORMAL || callerType == Account.Type.PROJECT) {
+            accountId = caller.getId();
         } else {
-            final DomainVO domain = _domainDao.findById(caller.getDomainId());
-            final List<Long> permittedDomainIds = _domainDao.getDomainChildrenIds(domain.getPath());
-            permittedAccountIds = _accountDao.getAccountIdsForDomains(permittedDomainIds);
+            domainIds = _domainDao.getDomainAndChildrenIds(caller.getDomainId());

Review Comment:
   can we skip the domain filter for the root admin? right now it puts every domain id in the query, which gets big on clouds with lots of domains



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "winterhazel (via GitHub)" <gi...@apache.org>.
winterhazel commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-3695199549

   @blueorangutan package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-3695243691

   Packaging result [SF]: ✖️ el8 ✖️ el9  ✔️ debian ✖️ suse15. SL-JID 16178


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-3695547038

   Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16183


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-3694922949

   Packaging result [SF]: ✖️ el8 ✖️ el9  ✖️ debian ✖️ suse15. SL-JID 16177


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing and deletion optimizations [cloudstack]

Posted by "shwstppr (via GitHub)" <gi...@apache.org>.
shwstppr commented on code in PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#discussion_r2318744832


##########
server/src/main/java/com/cloud/server/ManagementServerImpl.java:
##########
@@ -1224,31 +1225,34 @@ public boolean archiveEvents(final ArchiveEventsCmd cmd) {
     }
 
     @Override
-    public boolean deleteEvents(final DeleteEventsCmd cmd) {
-        final Account caller = getCaller();
-        final List<Long> ids = cmd.getIds();
-        boolean result = true;
-        List<Long> permittedAccountIds = new ArrayList<>();
+    public boolean deleteEvents(DeleteEventsCmd cmd) {
+        List<Long> ids = cmd.getIds();
+        String type = cmd.getType();
+        Date startDate = cmd.getStartDate();
+        Date endDate = cmd.getEndDate();
 
-        if (_accountMgr.isNormalUser(caller.getId()) || caller.getType() == Account.Type.PROJECT) {
-            permittedAccountIds.add(caller.getId());
-        } else {
-            final DomainVO domain = _domainDao.findById(caller.getDomainId());
-            final List<Long> permittedDomainIds = _domainDao.getDomainChildrenIds(domain.getPath());
-            permittedAccountIds = _accountDao.getAccountIdsForDomains(permittedDomainIds);
+        if (CollectionUtils.isEmpty(ids) && ObjectUtils.allNull(type, endDate)) {
+            throw new InvalidParameterValueException("Either 'ids', 'type' or 'enddate' must be specified.");
+        }
+        if (startDate != null && endDate == null) {
+            throw new InvalidParameterValueException("'startdate' must be specified with 'enddate' parameter.");

Review Comment:
   I've not checked the current behaviour but can't we allow  delete with only one of the params?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "winterhazel (via GitHub)" <gi...@apache.org>.
winterhazel commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-4283027201

   @blueorangutan package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing and deletion optimizations [cloudstack]

Posted by "winterhazel (via GitHub)" <gi...@apache.org>.
winterhazel commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-3694912422

   @blueorangutan package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-4489123397

   tested the touched APIs, not the performance though. LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-4289285898

   @blueorangutan test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-6039155415

   @winterhazel a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "winterhazel (via GitHub) via commits" <co...@cloudstack.apache.org>.
winterhazel commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-6802366975

   @Damans227 @JoaoJandre @vishesh92 I have addressed your comments. Could you re-review? Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-4283373098

   Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17555


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "winterhazel (via GitHub)" <gi...@apache.org>.
winterhazel commented on code in PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#discussion_r3170304149


##########
server/src/main/java/com/cloud/server/ManagementServerImpl.java:
##########
@@ -1225,57 +1224,64 @@ protected void checkPortParameters(final String publicPort, final String private
     }
 
     @Override
-    public boolean archiveEvents(final ArchiveEventsCmd cmd) {
-        final Account caller = getCaller();
-        final List<Long> ids = cmd.getIds();
-        boolean result = true;
-        List<Long> permittedAccountIds = new ArrayList<>();
+    public boolean archiveEvents(ArchiveEventsCmd cmd) {
+        List<Long> ids = cmd.getIds();
+        String type = cmd.getType();
+        Date startDate = cmd.getStartDate();
+        Date endDate = cmd.getEndDate();
+
+        validateEventOperationParameters(ids, type, endDate, startDate);
+
+        Long accountId = null;
+        List<Long> domainIds = null;
+        Account caller = getCaller();
+        Account.Type callerType = caller.getType();
 
-        if (_accountService.isNormalUser(caller.getId()) || caller.getType() == Account.Type.PROJECT) {
-            permittedAccountIds.add(caller.getId());
+        if (callerType == Account.Type.NORMAL || callerType == Account.Type.PROJECT) {
+            accountId = caller.getId();
         } else {
-            final DomainVO domain = _domainDao.findById(caller.getDomainId());
-            final List<Long> permittedDomainIds = _domainDao.getDomainChildrenIds(domain.getPath());
-            permittedAccountIds = _accountDao.getAccountIdsForDomains(permittedDomainIds);
+            domainIds = _domainDao.getDomainAndChildrenIds(caller.getDomainId());
         }
 
-        final List<EventVO> events = _eventDao.listToArchiveOrDeleteEvents(ids, cmd.getType(), cmd.getStartDate(), cmd.getEndDate(), permittedAccountIds);
-        final ControlledEntity[] sameOwnerEvents = events.toArray(new ControlledEntity[events.size()]);
-        _accountMgr.checkAccess(CallContext.current().getCallingAccount(), null, false, sameOwnerEvents);
+        long totalArchived = _eventDao.archiveEvents(ids, type, startDate, endDate, accountId, domainIds,
+                ConfigurationManagerImpl.DELETE_QUERY_BATCH_SIZE.value());
 
-        if (ids != null && events.size() < ids.size()) {
-            return false;
-        }
-        _eventDao.archiveEvents(events);
-        return result;
+        return totalArchived > 0;

Review Comment:
   ~This was the previous behavior. I do not want to change the response of the API in this patch, just optimize its operations.~ Edit: there was a change. I will adjust.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "winterhazel (via GitHub)" <gi...@apache.org>.
winterhazel commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-4355538674

   > * can we expect / catch any runtime exceptions in the command objects (`ArchiveEvents` and `DeleteEvents`)?
   
   @DaanHoogland `InvalidParameterValueException` can be thrown depending on the parameters, and `CloudRuntimeException` if there is a database-related error. I don't think there's any need to catch those in the command though.
   
   > * will further testing (beyond unit testing) be needed?
   
   Some basic manual testing would be nice.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-3697920347

   <b>[SF] Trillian test result (tid-15078)</b>
   Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
   Total time taken: 56017 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11443-t15078-kvm-ol8.zip
   Smoke tests completed. 148 look OK, 2 have errors, 0 did not run
   Only failed and skipped tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_03_deploy_and_scale_kubernetes_cluster | `Failure` | 27.90 | test_kubernetes_clusters.py
   test_02_isolate_network_FW_PF_default_routes_egress_false | `Failure` | 120.40 | test_routers_network_ops.py
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "JoaoJandre (via GitHub)" <gi...@apache.org>.
JoaoJandre commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-4733031545

   @winterhazel could you fix the conflicts here and also take a look at the outstanding comments?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "JoaoJandre (via GitHub)" <gi...@apache.org>.
JoaoJandre commented on code in PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#discussion_r4404651568


##########
engine/schema/src/main/java/com/cloud/upgrade/dao/Upgrade42300to2400.java:
##########
@@ -27,4 +29,25 @@ public String[] getUpgradableVersionRange() {
     public String getUpgradedVersion() {
         return "24.0.0";
     }
+
+    @Override
+    public void performDataMigration(Connection conn) {
+        addIndexes(conn);
+        dropOldIndexes(conn);
+    }
+
+    private void addIndexes(Connection conn) {
+        DbUpgradeUtils.addIndexWithNameIfNeeded(conn, "event", "i_event__archived__created__id",
+                "archived", "created", "id");
+        DbUpgradeUtils.addIndexWithNameIfNeeded(conn, "event", "i_event__account_id__archived__created__id",
+                "account_id", "archived", "created", "id");
+        DbUpgradeUtils.addIndexWithNameIfNeeded(conn, "event", "i_event__domain_id__archived__created__id",
+                "domain_id", "archived", "created", "id");
+        DbUpgradeUtils.addIndexWithNameIfNeeded(conn, "event", "i_event__resource_type__resource_id__archived__created__id",
+                "resource_type", "resource_id", "archived", "created", "id");

Review Comment:
   
   When using innoDB, the primary key is already added to the end of every index: 
   "In InnoDB tables, the primary key is included as a suffix in all other indexes. Therefore, keeping the primary key compact (e.g., using an appropriate integer type) is important for performance and storage efficiency."
   source:
    https://mariadb.com/docs/server/mariadb-quickstart-guides/mariadb-indexes-guide
   ```suggestion
                   "archived", "created", "id");
           DbUpgradeUtils.addIndexWithNameIfNeeded(conn, "event", "i_event__account_id__archived__created",
                   "account_id", "archived", "created");
           DbUpgradeUtils.addIndexWithNameIfNeeded(conn, "event", "i_event__domain_id__archived__created",
                   "domain_id", "archived", "created");
           DbUpgradeUtils.addIndexWithNameIfNeeded(conn, "event", "i_event__resource_type__resource_id__archived__created",
                   "resource_type", "resource_id", "archived", "created");
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "vishesh92 (via GitHub)" <gi...@apache.org>.
vishesh92 commented on code in PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#discussion_r4409164249


##########
api/src/main/java/org/apache/cloudstack/api/command/user/event/ArchiveEventsCmd.java:
##########
@@ -82,6 +81,22 @@ public String getType() {
         return type;
     }
 
+    public void setIds(List<Long> ids) {
+        this.ids = ids;
+    }
+
+    public void setEndDate(Date endDate) {
+        this.endDate = endDate;
+    }
+
+    public void setStartDate(Date startDate) {
+        this.startDate = startDate;
+    }
+
+    public void setType(String type) {
+        this.type = type;
+    }

Review Comment:
   we don't need the setters here



##########
api/src/main/java/org/apache/cloudstack/api/command/user/event/DeleteEventsCmd.java:
##########
@@ -82,6 +81,22 @@ public String getType() {
         return type;
     }
 
+    public void setIds(List<Long> ids) {
+        this.ids = ids;
+    }
+
+    public void setEndDate(Date endDate) {
+        this.endDate = endDate;
+    }
+
+    public void setStartDate(Date startDate) {
+        this.startDate = startDate;
+    }
+
+    public void setType(String type) {
+        this.type = type;
+    }
+

Review Comment:
   we don't need the setters here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "winterhazel (via GitHub) via commits" <co...@cloudstack.apache.org>.
winterhazel commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-6802370957

   @blueorangutan package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#discussion_r3153325352


##########
server/src/main/java/com/cloud/server/ManagementServerImpl.java:
##########
@@ -1225,57 +1224,64 @@ protected void checkPortParameters(final String publicPort, final String private
     }
 
     @Override
-    public boolean archiveEvents(final ArchiveEventsCmd cmd) {
-        final Account caller = getCaller();
-        final List<Long> ids = cmd.getIds();
-        boolean result = true;
-        List<Long> permittedAccountIds = new ArrayList<>();
+    public boolean archiveEvents(ArchiveEventsCmd cmd) {
+        List<Long> ids = cmd.getIds();
+        String type = cmd.getType();
+        Date startDate = cmd.getStartDate();
+        Date endDate = cmd.getEndDate();
+
+        validateEventOperationParameters(ids, type, endDate, startDate);
+
+        Long accountId = null;
+        List<Long> domainIds = null;
+        Account caller = getCaller();
+        Account.Type callerType = caller.getType();
 
-        if (_accountService.isNormalUser(caller.getId()) || caller.getType() == Account.Type.PROJECT) {
-            permittedAccountIds.add(caller.getId());
+        if (callerType == Account.Type.NORMAL || callerType == Account.Type.PROJECT) {
+            accountId = caller.getId();
         } else {
-            final DomainVO domain = _domainDao.findById(caller.getDomainId());
-            final List<Long> permittedDomainIds = _domainDao.getDomainChildrenIds(domain.getPath());
-            permittedAccountIds = _accountDao.getAccountIdsForDomains(permittedDomainIds);
+            domainIds = _domainDao.getDomainAndChildrenIds(caller.getDomainId());
         }
 
-        final List<EventVO> events = _eventDao.listToArchiveOrDeleteEvents(ids, cmd.getType(), cmd.getStartDate(), cmd.getEndDate(), permittedAccountIds);
-        final ControlledEntity[] sameOwnerEvents = events.toArray(new ControlledEntity[events.size()]);
-        _accountMgr.checkAccess(CallContext.current().getCallingAccount(), null, false, sameOwnerEvents);
+        long totalArchived = _eventDao.archiveEvents(ids, type, startDate, endDate, accountId, domainIds,
+                ConfigurationManagerImpl.DELETE_QUERY_BATCH_SIZE.value());
 
-        if (ids != null && events.size() < ids.size()) {
-            return false;
-        }
-        _eventDao.archiveEvents(events);
-        return result;
+        return totalArchived > 0;

Review Comment:
   When `ids` is provided, this method reports success if *any* matching rows were archived. That means a request like `ids=[1,2,3]` can return success even if only a subset was actually archived (e.g., some IDs don’t exist or aren’t in the caller’s scope). If the API is expected to be all-or-nothing for explicit IDs, consider returning failure (or throwing) when `totalArchived != ids.size()` when `ids` is non-empty, or otherwise make the partial-success behavior explicit.



##########
engine/schema/src/main/java/com/cloud/event/dao/EventDaoImpl.java:
##########
@@ -33,83 +38,90 @@
 
 @Component
 public class EventDaoImpl extends GenericDaoBase<EventVO, Long> implements EventDao {
-    protected final SearchBuilder<EventVO> CompletedEventSearch;
+
     protected final SearchBuilder<EventVO> ToArchiveOrDeleteEventSearch;
 
     public EventDaoImpl() {
-        CompletedEventSearch = createSearchBuilder();
-        CompletedEventSearch.and("state", CompletedEventSearch.entity().getState(), SearchCriteria.Op.EQ);
-        CompletedEventSearch.and("startId", CompletedEventSearch.entity().getStartId(), SearchCriteria.Op.EQ);
-        CompletedEventSearch.and("archived", CompletedEventSearch.entity().getArchived(), Op.EQ);
-        CompletedEventSearch.done();
-
         ToArchiveOrDeleteEventSearch = createSearchBuilder();
+        ToArchiveOrDeleteEventSearch.select("id", SearchCriteria.Func.NATIVE, ToArchiveOrDeleteEventSearch.entity().getId());
         ToArchiveOrDeleteEventSearch.and("id", ToArchiveOrDeleteEventSearch.entity().getId(), Op.IN);
         ToArchiveOrDeleteEventSearch.and("type", ToArchiveOrDeleteEventSearch.entity().getType(), Op.EQ);
-        ToArchiveOrDeleteEventSearch.and("accountIds", ToArchiveOrDeleteEventSearch.entity().getAccountId(), Op.IN);
+        ToArchiveOrDeleteEventSearch.and("accountId", ToArchiveOrDeleteEventSearch.entity().getAccountId(), Op.EQ);
+        ToArchiveOrDeleteEventSearch.and("domainIds", ToArchiveOrDeleteEventSearch.entity().getDomainId(), Op.IN);
         ToArchiveOrDeleteEventSearch.and("createdDateB", ToArchiveOrDeleteEventSearch.entity().getCreateDate(), Op.BETWEEN);
         ToArchiveOrDeleteEventSearch.and("createdDateL", ToArchiveOrDeleteEventSearch.entity().getCreateDate(), Op.LTEQ);
+        ToArchiveOrDeleteEventSearch.and("createdDateLT", ToArchiveOrDeleteEventSearch.entity().getCreateDate(), Op.LT);
         ToArchiveOrDeleteEventSearch.and("archived", ToArchiveOrDeleteEventSearch.entity().getArchived(), Op.EQ);
         ToArchiveOrDeleteEventSearch.done();
     }
 
-    @Override
-    public List<EventVO> searchAllEvents(SearchCriteria<EventVO> sc, Filter filter) {
-        return listIncludingRemovedBy(sc, filter);
-    }
-
-    @Override
-    public List<EventVO> listOlderEvents(Date oldTime) {
-        if (oldTime == null)
-            return null;
-        SearchCriteria<EventVO> sc = createSearchCriteria();
-        sc.addAnd("createDate", SearchCriteria.Op.LT, oldTime);
-        sc.addAnd("archived", SearchCriteria.Op.EQ, false);
-        return listIncludingRemovedBy(sc, null);
-    }
-
-    @Override
-    public EventVO findCompletedEvent(long startId) {
-        SearchCriteria<EventVO> sc = CompletedEventSearch.create();
-        sc.setParameters("state", State.Completed);
-        sc.setParameters("startId", startId);
-        sc.setParameters("archived", false);
-        return findOneIncludingRemovedBy(sc);
-    }
-
-    @Override
-    public List<EventVO> listToArchiveOrDeleteEvents(List<Long> ids, String type, Date startDate, Date endDate, List<Long> accountIds) {
+    private SearchCriteria<EventVO> createEventSearchCriteria(List<Long> ids, String type, Date startDate, Date endDate,
+                                                              Date limitDate, Long accountId, List<Long> domainIds) {
         SearchCriteria<EventVO> sc = ToArchiveOrDeleteEventSearch.create();
-        if (ids != null) {
-            sc.setParameters("id", ids.toArray(new Object[ids.size()]));
+
+        if (CollectionUtils.isNotEmpty(ids)) {
+            sc.setParameters("id", ids.toArray(new Object[0]));
         }
-        if (type != null) {
-            sc.setParameters("type", type);
+        if (CollectionUtils.isNotEmpty(domainIds)) {
+            sc.setParameters("domainIds", domainIds.toArray(new Object[0]));
         }
         if (startDate != null && endDate != null) {
             sc.setParameters("createdDateB", startDate, endDate);
         } else if (endDate != null) {
             sc.setParameters("createdDateL", endDate);
         }
-        if (accountIds != null && !accountIds.isEmpty()) {
-            sc.setParameters("accountIds", accountIds.toArray(new Object[accountIds.size()]));
-        }
+        sc.setParametersIfNotNull("accountId", accountId);
+        sc.setParametersIfNotNull("createdDateLT", limitDate);
+        sc.setParametersIfNotNull("type", type);
         sc.setParameters("archived", false);
-        return search(sc, null);
+
+        return sc;
     }
 
     @Override
-    public void archiveEvents(List<EventVO> events) {
-        if (events != null && !events.isEmpty()) {
-            TransactionLegacy txn = TransactionLegacy.currentTxn();
-            txn.start();
-            for (EventVO event : events) {
-                event = lockRow(event.getId(), true);
-                event.setArchived(true);
-                update(event.getId(), event);
-                txn.commit();
+    public long archiveEvents(List<Long> ids, String type, Date startDate, Date endDate, Long accountId, List<Long> domainIds,
+                              long limitPerQuery) {
+        SearchCriteria<EventVO> sc = createEventSearchCriteria(ids, type, startDate, endDate, null, accountId, domainIds);
+        Filter filter = null;
+        if (limitPerQuery > 0) {
+            filter = new Filter(limitPerQuery);
+        }
+
+        long archived;
+        long totalArchived = 0L;
+
+        do {
+            List<EventVO> events = search(sc, filter);
+            if (events.isEmpty()) {
+                break;
             }
-            txn.close();
+
+            archived = archiveEventsInternal(events);
+            totalArchived += archived;
+        } while (limitPerQuery > 0 && archived >= limitPerQuery);

Review Comment:
   `archiveEvents` can load *all* matching events into memory when `limitPerQuery <= 0` (the default for `delete.query.batch.size` is 0), then builds a single `UPDATE ... WHERE id IN (...)` statement containing every ID. On large event tables this risks OOMs, oversized SQL statements, and very long-running queries. Consider always archiving in batches (even when the config is 0), or implement a batched `UPDATE ... LIMIT <batch>` loop similar to `batchExpunge` so the operation remains bounded.



##########
api/src/main/java/org/apache/cloudstack/api/command/user/event/ArchiveEventsCmd.java:
##########
@@ -97,17 +112,12 @@ public long getEntityOwnerId() {
 
     @Override
     public void execute() {
-        if (ids == null && type == null && endDate == null) {
-            throw new InvalidParameterValueException("either ids, type or enddate must be specified");
-        } else if (startDate != null && endDate == null) {
-            throw new InvalidParameterValueException("enddate must be specified with startdate parameter");
-        }
         boolean result = _mgr.archiveEvents(this);
         if (result) {
             SuccessResponse response = new SuccessResponse(getCommandName());
             setResponseObject(response);
-        } else {
-            throw new ServerApiException(ApiErrorCode.INTERNAL_ERROR, "Unable to archive Events, one or more parameters has invalid values");
+            return;
         }
+        throw new ServerApiException(ApiErrorCode.INTERNAL_ERROR, "Unable to archive Events. One or more parameters have invalid values.");
     }

Review Comment:
   `_mgr.archiveEvents` returns `false` when `totalArchived == 0`, which can happen with valid parameters that match zero rows. Throwing an INTERNAL_ERROR that claims invalid parameters is misleading in that case. Consider distinguishing "no matching events" from actual validation errors (which should throw `InvalidParameterValueException`) and adjusting the message accordingly.



##########
engine/schema/src/main/java/com/cloud/event/dao/EventDaoImpl.java:
##########
@@ -33,83 +38,90 @@
 
 @Component
 public class EventDaoImpl extends GenericDaoBase<EventVO, Long> implements EventDao {
-    protected final SearchBuilder<EventVO> CompletedEventSearch;
+
     protected final SearchBuilder<EventVO> ToArchiveOrDeleteEventSearch;
 
     public EventDaoImpl() {
-        CompletedEventSearch = createSearchBuilder();
-        CompletedEventSearch.and("state", CompletedEventSearch.entity().getState(), SearchCriteria.Op.EQ);
-        CompletedEventSearch.and("startId", CompletedEventSearch.entity().getStartId(), SearchCriteria.Op.EQ);
-        CompletedEventSearch.and("archived", CompletedEventSearch.entity().getArchived(), Op.EQ);
-        CompletedEventSearch.done();
-
         ToArchiveOrDeleteEventSearch = createSearchBuilder();
+        ToArchiveOrDeleteEventSearch.select("id", SearchCriteria.Func.NATIVE, ToArchiveOrDeleteEventSearch.entity().getId());
         ToArchiveOrDeleteEventSearch.and("id", ToArchiveOrDeleteEventSearch.entity().getId(), Op.IN);
         ToArchiveOrDeleteEventSearch.and("type", ToArchiveOrDeleteEventSearch.entity().getType(), Op.EQ);
-        ToArchiveOrDeleteEventSearch.and("accountIds", ToArchiveOrDeleteEventSearch.entity().getAccountId(), Op.IN);
+        ToArchiveOrDeleteEventSearch.and("accountId", ToArchiveOrDeleteEventSearch.entity().getAccountId(), Op.EQ);
+        ToArchiveOrDeleteEventSearch.and("domainIds", ToArchiveOrDeleteEventSearch.entity().getDomainId(), Op.IN);
         ToArchiveOrDeleteEventSearch.and("createdDateB", ToArchiveOrDeleteEventSearch.entity().getCreateDate(), Op.BETWEEN);
         ToArchiveOrDeleteEventSearch.and("createdDateL", ToArchiveOrDeleteEventSearch.entity().getCreateDate(), Op.LTEQ);
+        ToArchiveOrDeleteEventSearch.and("createdDateLT", ToArchiveOrDeleteEventSearch.entity().getCreateDate(), Op.LT);
         ToArchiveOrDeleteEventSearch.and("archived", ToArchiveOrDeleteEventSearch.entity().getArchived(), Op.EQ);
         ToArchiveOrDeleteEventSearch.done();
     }
 
-    @Override
-    public List<EventVO> searchAllEvents(SearchCriteria<EventVO> sc, Filter filter) {
-        return listIncludingRemovedBy(sc, filter);
-    }
-
-    @Override
-    public List<EventVO> listOlderEvents(Date oldTime) {
-        if (oldTime == null)
-            return null;
-        SearchCriteria<EventVO> sc = createSearchCriteria();
-        sc.addAnd("createDate", SearchCriteria.Op.LT, oldTime);
-        sc.addAnd("archived", SearchCriteria.Op.EQ, false);
-        return listIncludingRemovedBy(sc, null);
-    }
-
-    @Override
-    public EventVO findCompletedEvent(long startId) {
-        SearchCriteria<EventVO> sc = CompletedEventSearch.create();
-        sc.setParameters("state", State.Completed);
-        sc.setParameters("startId", startId);
-        sc.setParameters("archived", false);
-        return findOneIncludingRemovedBy(sc);
-    }
-
-    @Override
-    public List<EventVO> listToArchiveOrDeleteEvents(List<Long> ids, String type, Date startDate, Date endDate, List<Long> accountIds) {
+    private SearchCriteria<EventVO> createEventSearchCriteria(List<Long> ids, String type, Date startDate, Date endDate,
+                                                              Date limitDate, Long accountId, List<Long> domainIds) {
         SearchCriteria<EventVO> sc = ToArchiveOrDeleteEventSearch.create();
-        if (ids != null) {
-            sc.setParameters("id", ids.toArray(new Object[ids.size()]));
+
+        if (CollectionUtils.isNotEmpty(ids)) {
+            sc.setParameters("id", ids.toArray(new Object[0]));
         }
-        if (type != null) {
-            sc.setParameters("type", type);
+        if (CollectionUtils.isNotEmpty(domainIds)) {
+            sc.setParameters("domainIds", domainIds.toArray(new Object[0]));
         }
         if (startDate != null && endDate != null) {
             sc.setParameters("createdDateB", startDate, endDate);
         } else if (endDate != null) {
             sc.setParameters("createdDateL", endDate);
         }
-        if (accountIds != null && !accountIds.isEmpty()) {
-            sc.setParameters("accountIds", accountIds.toArray(new Object[accountIds.size()]));
-        }
+        sc.setParametersIfNotNull("accountId", accountId);
+        sc.setParametersIfNotNull("createdDateLT", limitDate);
+        sc.setParametersIfNotNull("type", type);
         sc.setParameters("archived", false);
-        return search(sc, null);
+
+        return sc;
     }
 
     @Override
-    public void archiveEvents(List<EventVO> events) {
-        if (events != null && !events.isEmpty()) {
-            TransactionLegacy txn = TransactionLegacy.currentTxn();
-            txn.start();
-            for (EventVO event : events) {
-                event = lockRow(event.getId(), true);
-                event.setArchived(true);
-                update(event.getId(), event);
-                txn.commit();
+    public long archiveEvents(List<Long> ids, String type, Date startDate, Date endDate, Long accountId, List<Long> domainIds,
+                              long limitPerQuery) {
+        SearchCriteria<EventVO> sc = createEventSearchCriteria(ids, type, startDate, endDate, null, accountId, domainIds);
+        Filter filter = null;
+        if (limitPerQuery > 0) {
+            filter = new Filter(limitPerQuery);
+        }
+
+        long archived;
+        long totalArchived = 0L;
+
+        do {
+            List<EventVO> events = search(sc, filter);
+            if (events.isEmpty()) {
+                break;
             }
-            txn.close();
+
+            archived = archiveEventsInternal(events);
+            totalArchived += archived;
+        } while (limitPerQuery > 0 && archived >= limitPerQuery);
+
+        return totalArchived;
+    }
+
+    @DB
+    private long archiveEventsInternal(List<EventVO> events) {
+        final String idsAsString = events.stream()
+                .map(e -> Long.toString(e.getId()))
+                .collect(Collectors.joining(","));
+        final String query = String.format("UPDATE event SET archived=true WHERE id IN (%s)", idsAsString);
+
+        try (TransactionLegacy txn = TransactionLegacy.currentTxn();
+             PreparedStatement pstmt = txn.prepareStatement(query)) {
+            return pstmt.executeUpdate();
+        } catch (SQLException e) {
+            throw new CloudRuntimeException(e);
         }
     }
+
+    @Override
+    public long purgeAll(List<Long> ids, Date startDate, Date endDate, Date limitDate, String type, Long accountId,
+                         List<Long> domainIds, long limitPerQuery) {
+        SearchCriteria<EventVO> sc = createEventSearchCriteria(ids, type, startDate, endDate, limitDate, accountId, domainIds);
+        return batchExpunge(sc, limitPerQuery);
+    }

Review Comment:
   New batching semantics in `archiveEvents(...)`/`purgeAll(...)` are non-trivial (limit handling, iteration/termination, criteria composition) but there are currently no unit tests covering these paths (unlike other DAO batch-expunge usages in `engine/schema/src/test`). Consider adding a focused DAO unit test validating that multiple batches are processed correctly and that `limitPerQuery <= 0` behaves as intended.



##########
server/src/main/java/com/cloud/server/ManagementServerImpl.java:
##########
@@ -1225,57 +1224,64 @@ protected void checkPortParameters(final String publicPort, final String private
     }
 
     @Override
-    public boolean archiveEvents(final ArchiveEventsCmd cmd) {
-        final Account caller = getCaller();
-        final List<Long> ids = cmd.getIds();
-        boolean result = true;
-        List<Long> permittedAccountIds = new ArrayList<>();
+    public boolean archiveEvents(ArchiveEventsCmd cmd) {
+        List<Long> ids = cmd.getIds();
+        String type = cmd.getType();
+        Date startDate = cmd.getStartDate();
+        Date endDate = cmd.getEndDate();
+
+        validateEventOperationParameters(ids, type, endDate, startDate);
+
+        Long accountId = null;
+        List<Long> domainIds = null;
+        Account caller = getCaller();
+        Account.Type callerType = caller.getType();
 
-        if (_accountService.isNormalUser(caller.getId()) || caller.getType() == Account.Type.PROJECT) {
-            permittedAccountIds.add(caller.getId());
+        if (callerType == Account.Type.NORMAL || callerType == Account.Type.PROJECT) {
+            accountId = caller.getId();
         } else {
-            final DomainVO domain = _domainDao.findById(caller.getDomainId());
-            final List<Long> permittedDomainIds = _domainDao.getDomainChildrenIds(domain.getPath());
-            permittedAccountIds = _accountDao.getAccountIdsForDomains(permittedDomainIds);
+            domainIds = _domainDao.getDomainAndChildrenIds(caller.getDomainId());
         }
 
-        final List<EventVO> events = _eventDao.listToArchiveOrDeleteEvents(ids, cmd.getType(), cmd.getStartDate(), cmd.getEndDate(), permittedAccountIds);
-        final ControlledEntity[] sameOwnerEvents = events.toArray(new ControlledEntity[events.size()]);
-        _accountMgr.checkAccess(CallContext.current().getCallingAccount(), null, false, sameOwnerEvents);
+        long totalArchived = _eventDao.archiveEvents(ids, type, startDate, endDate, accountId, domainIds,
+                ConfigurationManagerImpl.DELETE_QUERY_BATCH_SIZE.value());
 
-        if (ids != null && events.size() < ids.size()) {
-            return false;
-        }
-        _eventDao.archiveEvents(events);
-        return result;
+        return totalArchived > 0;
     }
 
     @Override
-    public boolean deleteEvents(final DeleteEventsCmd cmd) {
-        final Account caller = getCaller();
-        final List<Long> ids = cmd.getIds();
-        boolean result = true;
-        List<Long> permittedAccountIds = new ArrayList<>();
+    public boolean deleteEvents(DeleteEventsCmd cmd) {
+        List<Long> ids = cmd.getIds();
+        String type = cmd.getType();
+        Date startDate = cmd.getStartDate();
+        Date endDate = cmd.getEndDate();
 
-        if (_accountMgr.isNormalUser(caller.getId()) || caller.getType() == Account.Type.PROJECT) {
-            permittedAccountIds.add(caller.getId());
+        validateEventOperationParameters(ids, type, endDate, startDate);
+
+        Long accountId = null;
+        List<Long> domainIds = null;
+        Account caller = getCaller();
+        Account.Type callerType = caller.getType();
+
+        if (callerType == Account.Type.NORMAL || callerType == Account.Type.PROJECT) {
+            accountId = caller.getId();
         } else {
-            final DomainVO domain = _domainDao.findById(caller.getDomainId());
-            final List<Long> permittedDomainIds = _domainDao.getDomainChildrenIds(domain.getPath());
-            permittedAccountIds = _accountDao.getAccountIdsForDomains(permittedDomainIds);
+            domainIds = _domainDao.getDomainAndChildrenIds(caller.getDomainId());
         }
 
-        final List<EventVO> events = _eventDao.listToArchiveOrDeleteEvents(ids, cmd.getType(), cmd.getStartDate(), cmd.getEndDate(), permittedAccountIds);
-        final ControlledEntity[] sameOwnerEvents = events.toArray(new ControlledEntity[events.size()]);
-        _accountMgr.checkAccess(CallContext.current().getCallingAccount(), null, false, sameOwnerEvents);
+        long totalRemoved = _eventDao.purgeAll(ids, startDate, endDate, null, type, accountId, domainIds,
+                ConfigurationManagerImpl.DELETE_QUERY_BATCH_SIZE.value());
 
-        if (ids != null && events.size() < ids.size()) {
-            return false;
+        return totalRemoved > 0;
+    }
+
+    protected void validateEventOperationParameters(List<Long> ids, String type, Date endDate, Date startDate) {
+        if (CollectionUtils.isEmpty(ids) && ObjectUtils.allNull(type, endDate)) {
+            throw new InvalidParameterValueException("Either 'ids', 'type' or 'enddate' must be specified.");
         }
-        for (final EventVO event : events) {
-            _eventDao.remove(event.getId());
+        if (startDate != null && endDate == null) {
+            throw new InvalidParameterValueException("'startdate' must be specified with 'enddate' parameter.");

Review Comment:
   `validateEventOperationParameters` throws when `startDate != null && endDate == null`, but the exception message says "'startdate' must be specified with 'enddate'", which is the opposite of what’s wrong in this case. Update the message to indicate that `enddate` is required when `startdate` is provided (and keep parameter naming consistent with the API).
   ```suggestion
               throw new InvalidParameterValueException("'enddate' must be specified when 'startdate' parameter is provided.");
   ```



##########
server/src/main/java/com/cloud/server/ManagementServerImpl.java:
##########
@@ -1225,57 +1224,64 @@ protected void checkPortParameters(final String publicPort, final String private
     }
 
     @Override
-    public boolean archiveEvents(final ArchiveEventsCmd cmd) {
-        final Account caller = getCaller();
-        final List<Long> ids = cmd.getIds();
-        boolean result = true;
-        List<Long> permittedAccountIds = new ArrayList<>();
+    public boolean archiveEvents(ArchiveEventsCmd cmd) {
+        List<Long> ids = cmd.getIds();
+        String type = cmd.getType();
+        Date startDate = cmd.getStartDate();
+        Date endDate = cmd.getEndDate();
+
+        validateEventOperationParameters(ids, type, endDate, startDate);
+
+        Long accountId = null;
+        List<Long> domainIds = null;
+        Account caller = getCaller();
+        Account.Type callerType = caller.getType();
 
-        if (_accountService.isNormalUser(caller.getId()) || caller.getType() == Account.Type.PROJECT) {
-            permittedAccountIds.add(caller.getId());
+        if (callerType == Account.Type.NORMAL || callerType == Account.Type.PROJECT) {
+            accountId = caller.getId();
         } else {
-            final DomainVO domain = _domainDao.findById(caller.getDomainId());
-            final List<Long> permittedDomainIds = _domainDao.getDomainChildrenIds(domain.getPath());
-            permittedAccountIds = _accountDao.getAccountIdsForDomains(permittedDomainIds);
+            domainIds = _domainDao.getDomainAndChildrenIds(caller.getDomainId());
         }
 
-        final List<EventVO> events = _eventDao.listToArchiveOrDeleteEvents(ids, cmd.getType(), cmd.getStartDate(), cmd.getEndDate(), permittedAccountIds);
-        final ControlledEntity[] sameOwnerEvents = events.toArray(new ControlledEntity[events.size()]);
-        _accountMgr.checkAccess(CallContext.current().getCallingAccount(), null, false, sameOwnerEvents);
+        long totalArchived = _eventDao.archiveEvents(ids, type, startDate, endDate, accountId, domainIds,
+                ConfigurationManagerImpl.DELETE_QUERY_BATCH_SIZE.value());
 
-        if (ids != null && events.size() < ids.size()) {
-            return false;
-        }
-        _eventDao.archiveEvents(events);
-        return result;
+        return totalArchived > 0;
     }
 
     @Override
-    public boolean deleteEvents(final DeleteEventsCmd cmd) {
-        final Account caller = getCaller();
-        final List<Long> ids = cmd.getIds();
-        boolean result = true;
-        List<Long> permittedAccountIds = new ArrayList<>();
+    public boolean deleteEvents(DeleteEventsCmd cmd) {
+        List<Long> ids = cmd.getIds();
+        String type = cmd.getType();
+        Date startDate = cmd.getStartDate();
+        Date endDate = cmd.getEndDate();
 
-        if (_accountMgr.isNormalUser(caller.getId()) || caller.getType() == Account.Type.PROJECT) {
-            permittedAccountIds.add(caller.getId());
+        validateEventOperationParameters(ids, type, endDate, startDate);
+
+        Long accountId = null;
+        List<Long> domainIds = null;
+        Account caller = getCaller();
+        Account.Type callerType = caller.getType();
+
+        if (callerType == Account.Type.NORMAL || callerType == Account.Type.PROJECT) {
+            accountId = caller.getId();
         } else {
-            final DomainVO domain = _domainDao.findById(caller.getDomainId());
-            final List<Long> permittedDomainIds = _domainDao.getDomainChildrenIds(domain.getPath());
-            permittedAccountIds = _accountDao.getAccountIdsForDomains(permittedDomainIds);
+            domainIds = _domainDao.getDomainAndChildrenIds(caller.getDomainId());
         }
 
-        final List<EventVO> events = _eventDao.listToArchiveOrDeleteEvents(ids, cmd.getType(), cmd.getStartDate(), cmd.getEndDate(), permittedAccountIds);
-        final ControlledEntity[] sameOwnerEvents = events.toArray(new ControlledEntity[events.size()]);
-        _accountMgr.checkAccess(CallContext.current().getCallingAccount(), null, false, sameOwnerEvents);
+        long totalRemoved = _eventDao.purgeAll(ids, startDate, endDate, null, type, accountId, domainIds,
+                ConfigurationManagerImpl.DELETE_QUERY_BATCH_SIZE.value());
 

Review Comment:
   When `ids` is provided, this method reports success if *any* rows were deleted. This can silently ignore IDs that don’t match the criteria / caller scope while still returning success, which is potentially misleading for callers expecting all requested IDs to be removed. Consider aligning the return semantics with the requested operation (e.g., fail when `ids` is non-empty and `totalRemoved != ids.size()`, or explicitly document/return partial results).
   ```suggestion
   
           if (CollectionUtils.isNotEmpty(ids)) {
               return totalRemoved == ids.size();
           }
   ```



##########
engine/schema/src/main/java/com/cloud/event/dao/EventDaoImpl.java:
##########
@@ -33,83 +38,90 @@
 
 @Component
 public class EventDaoImpl extends GenericDaoBase<EventVO, Long> implements EventDao {
-    protected final SearchBuilder<EventVO> CompletedEventSearch;
+
     protected final SearchBuilder<EventVO> ToArchiveOrDeleteEventSearch;
 
     public EventDaoImpl() {
-        CompletedEventSearch = createSearchBuilder();
-        CompletedEventSearch.and("state", CompletedEventSearch.entity().getState(), SearchCriteria.Op.EQ);
-        CompletedEventSearch.and("startId", CompletedEventSearch.entity().getStartId(), SearchCriteria.Op.EQ);
-        CompletedEventSearch.and("archived", CompletedEventSearch.entity().getArchived(), Op.EQ);
-        CompletedEventSearch.done();
-
         ToArchiveOrDeleteEventSearch = createSearchBuilder();
+        ToArchiveOrDeleteEventSearch.select("id", SearchCriteria.Func.NATIVE, ToArchiveOrDeleteEventSearch.entity().getId());
         ToArchiveOrDeleteEventSearch.and("id", ToArchiveOrDeleteEventSearch.entity().getId(), Op.IN);
         ToArchiveOrDeleteEventSearch.and("type", ToArchiveOrDeleteEventSearch.entity().getType(), Op.EQ);
-        ToArchiveOrDeleteEventSearch.and("accountIds", ToArchiveOrDeleteEventSearch.entity().getAccountId(), Op.IN);
+        ToArchiveOrDeleteEventSearch.and("accountId", ToArchiveOrDeleteEventSearch.entity().getAccountId(), Op.EQ);
+        ToArchiveOrDeleteEventSearch.and("domainIds", ToArchiveOrDeleteEventSearch.entity().getDomainId(), Op.IN);
         ToArchiveOrDeleteEventSearch.and("createdDateB", ToArchiveOrDeleteEventSearch.entity().getCreateDate(), Op.BETWEEN);
         ToArchiveOrDeleteEventSearch.and("createdDateL", ToArchiveOrDeleteEventSearch.entity().getCreateDate(), Op.LTEQ);
+        ToArchiveOrDeleteEventSearch.and("createdDateLT", ToArchiveOrDeleteEventSearch.entity().getCreateDate(), Op.LT);
         ToArchiveOrDeleteEventSearch.and("archived", ToArchiveOrDeleteEventSearch.entity().getArchived(), Op.EQ);
         ToArchiveOrDeleteEventSearch.done();
     }
 
-    @Override
-    public List<EventVO> searchAllEvents(SearchCriteria<EventVO> sc, Filter filter) {
-        return listIncludingRemovedBy(sc, filter);
-    }
-
-    @Override
-    public List<EventVO> listOlderEvents(Date oldTime) {
-        if (oldTime == null)
-            return null;
-        SearchCriteria<EventVO> sc = createSearchCriteria();
-        sc.addAnd("createDate", SearchCriteria.Op.LT, oldTime);
-        sc.addAnd("archived", SearchCriteria.Op.EQ, false);
-        return listIncludingRemovedBy(sc, null);
-    }
-
-    @Override
-    public EventVO findCompletedEvent(long startId) {
-        SearchCriteria<EventVO> sc = CompletedEventSearch.create();
-        sc.setParameters("state", State.Completed);
-        sc.setParameters("startId", startId);
-        sc.setParameters("archived", false);
-        return findOneIncludingRemovedBy(sc);
-    }
-
-    @Override
-    public List<EventVO> listToArchiveOrDeleteEvents(List<Long> ids, String type, Date startDate, Date endDate, List<Long> accountIds) {
+    private SearchCriteria<EventVO> createEventSearchCriteria(List<Long> ids, String type, Date startDate, Date endDate,
+                                                              Date limitDate, Long accountId, List<Long> domainIds) {
         SearchCriteria<EventVO> sc = ToArchiveOrDeleteEventSearch.create();
-        if (ids != null) {
-            sc.setParameters("id", ids.toArray(new Object[ids.size()]));
+
+        if (CollectionUtils.isNotEmpty(ids)) {
+            sc.setParameters("id", ids.toArray(new Object[0]));
         }
-        if (type != null) {
-            sc.setParameters("type", type);
+        if (CollectionUtils.isNotEmpty(domainIds)) {
+            sc.setParameters("domainIds", domainIds.toArray(new Object[0]));
         }
         if (startDate != null && endDate != null) {
             sc.setParameters("createdDateB", startDate, endDate);
         } else if (endDate != null) {
             sc.setParameters("createdDateL", endDate);
         }
-        if (accountIds != null && !accountIds.isEmpty()) {
-            sc.setParameters("accountIds", accountIds.toArray(new Object[accountIds.size()]));
-        }
+        sc.setParametersIfNotNull("accountId", accountId);
+        sc.setParametersIfNotNull("createdDateLT", limitDate);
+        sc.setParametersIfNotNull("type", type);
         sc.setParameters("archived", false);
-        return search(sc, null);
+
+        return sc;
     }
 
     @Override
-    public void archiveEvents(List<EventVO> events) {
-        if (events != null && !events.isEmpty()) {
-            TransactionLegacy txn = TransactionLegacy.currentTxn();
-            txn.start();
-            for (EventVO event : events) {
-                event = lockRow(event.getId(), true);
-                event.setArchived(true);
-                update(event.getId(), event);
-                txn.commit();
+    public long archiveEvents(List<Long> ids, String type, Date startDate, Date endDate, Long accountId, List<Long> domainIds,
+                              long limitPerQuery) {
+        SearchCriteria<EventVO> sc = createEventSearchCriteria(ids, type, startDate, endDate, null, accountId, domainIds);
+        Filter filter = null;
+        if (limitPerQuery > 0) {
+            filter = new Filter(limitPerQuery);
+        }
+
+        long archived;
+        long totalArchived = 0L;
+
+        do {
+            List<EventVO> events = search(sc, filter);
+            if (events.isEmpty()) {
+                break;
             }
-            txn.close();
+
+            archived = archiveEventsInternal(events);
+            totalArchived += archived;
+        } while (limitPerQuery > 0 && archived >= limitPerQuery);
+
+        return totalArchived;
+    }
+
+    @DB
+    private long archiveEventsInternal(List<EventVO> events) {
+        final String idsAsString = events.stream()
+                .map(e -> Long.toString(e.getId()))
+                .collect(Collectors.joining(","));
+        final String query = String.format("UPDATE event SET archived=true WHERE id IN (%s)", idsAsString);
+
+        try (TransactionLegacy txn = TransactionLegacy.currentTxn();
+             PreparedStatement pstmt = txn.prepareStatement(query)) {

Review Comment:
   `archiveEventsInternal` builds SQL by string-concatenating the IDs into an `IN (...)` clause. Besides the query-size risk, it bypasses parameter binding and can stress SQL parsing/plan caching. Consider using a parameterized statement (placeholders) or a different batching strategy (e.g., update by criteria with `LIMIT`) to keep statements small and reusable.
   ```suggestion
           if (CollectionUtils.isEmpty(events)) {
               return 0L;
           }
   
           final String placeholders = events.stream()
                   .map(event -> "?")
                   .collect(Collectors.joining(","));
           final String query = String.format("UPDATE event SET archived=true WHERE id IN (%s)", placeholders);
   
           try (TransactionLegacy txn = TransactionLegacy.currentTxn();
                PreparedStatement pstmt = txn.prepareStatement(query)) {
               for (int i = 0; i < events.size(); i++) {
                   pstmt.setLong(i + 1, events.get(i).getId());
               }
   ```



##########
engine/schema/src/main/java/com/cloud/event/dao/EventDaoImpl.java:
##########
@@ -33,83 +38,90 @@
 
 @Component
 public class EventDaoImpl extends GenericDaoBase<EventVO, Long> implements EventDao {
-    protected final SearchBuilder<EventVO> CompletedEventSearch;
+
     protected final SearchBuilder<EventVO> ToArchiveOrDeleteEventSearch;
 
     public EventDaoImpl() {
-        CompletedEventSearch = createSearchBuilder();
-        CompletedEventSearch.and("state", CompletedEventSearch.entity().getState(), SearchCriteria.Op.EQ);
-        CompletedEventSearch.and("startId", CompletedEventSearch.entity().getStartId(), SearchCriteria.Op.EQ);
-        CompletedEventSearch.and("archived", CompletedEventSearch.entity().getArchived(), Op.EQ);
-        CompletedEventSearch.done();
-
         ToArchiveOrDeleteEventSearch = createSearchBuilder();
+        ToArchiveOrDeleteEventSearch.select("id", SearchCriteria.Func.NATIVE, ToArchiveOrDeleteEventSearch.entity().getId());
         ToArchiveOrDeleteEventSearch.and("id", ToArchiveOrDeleteEventSearch.entity().getId(), Op.IN);
         ToArchiveOrDeleteEventSearch.and("type", ToArchiveOrDeleteEventSearch.entity().getType(), Op.EQ);
-        ToArchiveOrDeleteEventSearch.and("accountIds", ToArchiveOrDeleteEventSearch.entity().getAccountId(), Op.IN);
+        ToArchiveOrDeleteEventSearch.and("accountId", ToArchiveOrDeleteEventSearch.entity().getAccountId(), Op.EQ);
+        ToArchiveOrDeleteEventSearch.and("domainIds", ToArchiveOrDeleteEventSearch.entity().getDomainId(), Op.IN);
         ToArchiveOrDeleteEventSearch.and("createdDateB", ToArchiveOrDeleteEventSearch.entity().getCreateDate(), Op.BETWEEN);
         ToArchiveOrDeleteEventSearch.and("createdDateL", ToArchiveOrDeleteEventSearch.entity().getCreateDate(), Op.LTEQ);
+        ToArchiveOrDeleteEventSearch.and("createdDateLT", ToArchiveOrDeleteEventSearch.entity().getCreateDate(), Op.LT);
         ToArchiveOrDeleteEventSearch.and("archived", ToArchiveOrDeleteEventSearch.entity().getArchived(), Op.EQ);
         ToArchiveOrDeleteEventSearch.done();
     }
 
-    @Override
-    public List<EventVO> searchAllEvents(SearchCriteria<EventVO> sc, Filter filter) {
-        return listIncludingRemovedBy(sc, filter);
-    }
-
-    @Override
-    public List<EventVO> listOlderEvents(Date oldTime) {
-        if (oldTime == null)
-            return null;
-        SearchCriteria<EventVO> sc = createSearchCriteria();
-        sc.addAnd("createDate", SearchCriteria.Op.LT, oldTime);
-        sc.addAnd("archived", SearchCriteria.Op.EQ, false);
-        return listIncludingRemovedBy(sc, null);
-    }
-
-    @Override
-    public EventVO findCompletedEvent(long startId) {
-        SearchCriteria<EventVO> sc = CompletedEventSearch.create();
-        sc.setParameters("state", State.Completed);
-        sc.setParameters("startId", startId);
-        sc.setParameters("archived", false);
-        return findOneIncludingRemovedBy(sc);
-    }
-
-    @Override
-    public List<EventVO> listToArchiveOrDeleteEvents(List<Long> ids, String type, Date startDate, Date endDate, List<Long> accountIds) {
+    private SearchCriteria<EventVO> createEventSearchCriteria(List<Long> ids, String type, Date startDate, Date endDate,
+                                                              Date limitDate, Long accountId, List<Long> domainIds) {
         SearchCriteria<EventVO> sc = ToArchiveOrDeleteEventSearch.create();
-        if (ids != null) {
-            sc.setParameters("id", ids.toArray(new Object[ids.size()]));
+
+        if (CollectionUtils.isNotEmpty(ids)) {
+            sc.setParameters("id", ids.toArray(new Object[0]));
         }
-        if (type != null) {
-            sc.setParameters("type", type);
+        if (CollectionUtils.isNotEmpty(domainIds)) {
+            sc.setParameters("domainIds", domainIds.toArray(new Object[0]));
         }
         if (startDate != null && endDate != null) {
             sc.setParameters("createdDateB", startDate, endDate);
         } else if (endDate != null) {
             sc.setParameters("createdDateL", endDate);
         }
-        if (accountIds != null && !accountIds.isEmpty()) {
-            sc.setParameters("accountIds", accountIds.toArray(new Object[accountIds.size()]));
-        }
+        sc.setParametersIfNotNull("accountId", accountId);
+        sc.setParametersIfNotNull("createdDateLT", limitDate);
+        sc.setParametersIfNotNull("type", type);
         sc.setParameters("archived", false);
-        return search(sc, null);
+
+        return sc;
     }
 
     @Override
-    public void archiveEvents(List<EventVO> events) {
-        if (events != null && !events.isEmpty()) {
-            TransactionLegacy txn = TransactionLegacy.currentTxn();
-            txn.start();
-            for (EventVO event : events) {
-                event = lockRow(event.getId(), true);
-                event.setArchived(true);
-                update(event.getId(), event);
-                txn.commit();
+    public long archiveEvents(List<Long> ids, String type, Date startDate, Date endDate, Long accountId, List<Long> domainIds,
+                              long limitPerQuery) {
+        SearchCriteria<EventVO> sc = createEventSearchCriteria(ids, type, startDate, endDate, null, accountId, domainIds);
+        Filter filter = null;
+        if (limitPerQuery > 0) {
+            filter = new Filter(limitPerQuery);
+        }
+
+        long archived;
+        long totalArchived = 0L;
+
+        do {
+            List<EventVO> events = search(sc, filter);
+            if (events.isEmpty()) {
+                break;
             }
-            txn.close();
+
+            archived = archiveEventsInternal(events);
+            totalArchived += archived;
+        } while (limitPerQuery > 0 && archived >= limitPerQuery);

Review Comment:
   The loop termination condition uses the number of rows updated (`archived`) to decide whether to fetch the next batch. JDBC drivers/DBs can report fewer affected rows than selected rows (e.g., if some rows are updated concurrently or if an update sets the column to its current value), which can prematurely stop batching while more unarchived events still match the criteria. Consider basing the loop continuation on the batch size returned by `search` (e.g., `events.size() == limitPerQuery`) instead of the update count, or otherwise ensure the termination condition can’t end early.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-4234904233

   This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing and deletion optimizations [cloudstack]

Posted by "winterhazel (via GitHub)" <gi...@apache.org>.
winterhazel commented on code in PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#discussion_r2318863692


##########
server/src/main/java/com/cloud/server/ManagementServerImpl.java:
##########
@@ -1224,31 +1225,34 @@ public boolean archiveEvents(final ArchiveEventsCmd cmd) {
     }
 
     @Override
-    public boolean deleteEvents(final DeleteEventsCmd cmd) {
-        final Account caller = getCaller();
-        final List<Long> ids = cmd.getIds();
-        boolean result = true;
-        List<Long> permittedAccountIds = new ArrayList<>();
+    public boolean deleteEvents(DeleteEventsCmd cmd) {
+        List<Long> ids = cmd.getIds();
+        String type = cmd.getType();
+        Date startDate = cmd.getStartDate();
+        Date endDate = cmd.getEndDate();
 
-        if (_accountMgr.isNormalUser(caller.getId()) || caller.getType() == Account.Type.PROJECT) {
-            permittedAccountIds.add(caller.getId());
-        } else {
-            final DomainVO domain = _domainDao.findById(caller.getDomainId());
-            final List<Long> permittedDomainIds = _domainDao.getDomainChildrenIds(domain.getPath());
-            permittedAccountIds = _accountDao.getAccountIdsForDomains(permittedDomainIds);
+        if (CollectionUtils.isEmpty(ids) && ObjectUtils.allNull(type, endDate)) {
+            throw new InvalidParameterValueException("Either 'ids', 'type' or 'enddate' must be specified.");
+        }
+        if (startDate != null && endDate == null) {
+            throw new InvalidParameterValueException("'startdate' must be specified with 'enddate' parameter.");

Review Comment:
   This is the current behavior. I've just moved the code from `DeleteEventsCmd` to here.
   
   I think that this can be allowed, but I do not want to modify how the parameters of the API behave in this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-3816796243

   This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-3695200466

   @winterhazel a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-3695785762

   @DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "nvazquez (via GitHub)" <gi...@apache.org>.
nvazquez commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-5269536083

   Hi @winterhazel can you please address the review comments and conflicts on this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing and deletion optimizations [cloudstack]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-3183867153

   ## [Codecov](https://app.codecov.io/gh/apache/cloudstack/pull/11443?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 4.00%. Comparing base ([`f5b4858`](https://app.codecov.io/gh/apache/cloudstack/commit/f5b4858012b41993f97b3a762397b5b7e6e0de3e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`cea607d`](https://app.codecov.io/gh/apache/cloudstack/commit/cea607d5993b0884cc49485df782a0144ccefd2d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   > :exclamation:  There is a different number of reports uploaded between BASE (f5b4858) and HEAD (cea607d). Click for more details.
   > 
   > <details><summary>HEAD has 1 upload less than BASE</summary>
   >
   >| Flag | BASE (f5b4858) | HEAD (cea607d) |
   >|------|------|------|
   >|unittests|1|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@              Coverage Diff              @@
   ##               4.20   #11443       +/-   ##
   =============================================
   - Coverage     16.17%    4.00%   -12.17%     
   =============================================
     Files          5656      402     -5254     
     Lines        497990    32632   -465358     
     Branches      60399     5796    -54603     
   =============================================
   - Hits          80529     1308    -79221     
   + Misses       408499    31174   -377325     
   + Partials       8962      150     -8812     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/cloudstack/pull/11443/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [uitests](https://app.codecov.io/gh/apache/cloudstack/pull/11443/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `4.00% <ø> (ø)` | |
   | [unittests](https://app.codecov.io/gh/apache/cloudstack/pull/11443/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/cloudstack/pull/11443?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing and deletion optimizations [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-3183845043

   @winterhazel a [SL] Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-4358366181

   @DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "winterhazel (via GitHub)" <gi...@apache.org>.
winterhazel commented on code in PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#discussion_r3170268705


##########
engine/schema/src/main/java/com/cloud/event/dao/EventDaoImpl.java:
##########
@@ -33,83 +38,90 @@
 
 @Component
 public class EventDaoImpl extends GenericDaoBase<EventVO, Long> implements EventDao {
-    protected final SearchBuilder<EventVO> CompletedEventSearch;
+
     protected final SearchBuilder<EventVO> ToArchiveOrDeleteEventSearch;
 
     public EventDaoImpl() {
-        CompletedEventSearch = createSearchBuilder();
-        CompletedEventSearch.and("state", CompletedEventSearch.entity().getState(), SearchCriteria.Op.EQ);
-        CompletedEventSearch.and("startId", CompletedEventSearch.entity().getStartId(), SearchCriteria.Op.EQ);
-        CompletedEventSearch.and("archived", CompletedEventSearch.entity().getArchived(), Op.EQ);
-        CompletedEventSearch.done();
-
         ToArchiveOrDeleteEventSearch = createSearchBuilder();
+        ToArchiveOrDeleteEventSearch.select("id", SearchCriteria.Func.NATIVE, ToArchiveOrDeleteEventSearch.entity().getId());
         ToArchiveOrDeleteEventSearch.and("id", ToArchiveOrDeleteEventSearch.entity().getId(), Op.IN);
         ToArchiveOrDeleteEventSearch.and("type", ToArchiveOrDeleteEventSearch.entity().getType(), Op.EQ);
-        ToArchiveOrDeleteEventSearch.and("accountIds", ToArchiveOrDeleteEventSearch.entity().getAccountId(), Op.IN);
+        ToArchiveOrDeleteEventSearch.and("accountId", ToArchiveOrDeleteEventSearch.entity().getAccountId(), Op.EQ);
+        ToArchiveOrDeleteEventSearch.and("domainIds", ToArchiveOrDeleteEventSearch.entity().getDomainId(), Op.IN);
         ToArchiveOrDeleteEventSearch.and("createdDateB", ToArchiveOrDeleteEventSearch.entity().getCreateDate(), Op.BETWEEN);
         ToArchiveOrDeleteEventSearch.and("createdDateL", ToArchiveOrDeleteEventSearch.entity().getCreateDate(), Op.LTEQ);
+        ToArchiveOrDeleteEventSearch.and("createdDateLT", ToArchiveOrDeleteEventSearch.entity().getCreateDate(), Op.LT);
         ToArchiveOrDeleteEventSearch.and("archived", ToArchiveOrDeleteEventSearch.entity().getArchived(), Op.EQ);
         ToArchiveOrDeleteEventSearch.done();
     }
 
-    @Override
-    public List<EventVO> searchAllEvents(SearchCriteria<EventVO> sc, Filter filter) {
-        return listIncludingRemovedBy(sc, filter);
-    }
-
-    @Override
-    public List<EventVO> listOlderEvents(Date oldTime) {
-        if (oldTime == null)
-            return null;
-        SearchCriteria<EventVO> sc = createSearchCriteria();
-        sc.addAnd("createDate", SearchCriteria.Op.LT, oldTime);
-        sc.addAnd("archived", SearchCriteria.Op.EQ, false);
-        return listIncludingRemovedBy(sc, null);
-    }
-
-    @Override
-    public EventVO findCompletedEvent(long startId) {
-        SearchCriteria<EventVO> sc = CompletedEventSearch.create();
-        sc.setParameters("state", State.Completed);
-        sc.setParameters("startId", startId);
-        sc.setParameters("archived", false);
-        return findOneIncludingRemovedBy(sc);
-    }
-
-    @Override
-    public List<EventVO> listToArchiveOrDeleteEvents(List<Long> ids, String type, Date startDate, Date endDate, List<Long> accountIds) {
+    private SearchCriteria<EventVO> createEventSearchCriteria(List<Long> ids, String type, Date startDate, Date endDate,
+                                                              Date limitDate, Long accountId, List<Long> domainIds) {
         SearchCriteria<EventVO> sc = ToArchiveOrDeleteEventSearch.create();
-        if (ids != null) {
-            sc.setParameters("id", ids.toArray(new Object[ids.size()]));
+
+        if (CollectionUtils.isNotEmpty(ids)) {
+            sc.setParameters("id", ids.toArray(new Object[0]));
         }
-        if (type != null) {
-            sc.setParameters("type", type);
+        if (CollectionUtils.isNotEmpty(domainIds)) {
+            sc.setParameters("domainIds", domainIds.toArray(new Object[0]));
         }
         if (startDate != null && endDate != null) {
             sc.setParameters("createdDateB", startDate, endDate);
         } else if (endDate != null) {
             sc.setParameters("createdDateL", endDate);
         }
-        if (accountIds != null && !accountIds.isEmpty()) {
-            sc.setParameters("accountIds", accountIds.toArray(new Object[accountIds.size()]));
-        }
+        sc.setParametersIfNotNull("accountId", accountId);
+        sc.setParametersIfNotNull("createdDateLT", limitDate);
+        sc.setParametersIfNotNull("type", type);
         sc.setParameters("archived", false);
-        return search(sc, null);
+
+        return sc;
     }
 
     @Override
-    public void archiveEvents(List<EventVO> events) {
-        if (events != null && !events.isEmpty()) {
-            TransactionLegacy txn = TransactionLegacy.currentTxn();
-            txn.start();
-            for (EventVO event : events) {
-                event = lockRow(event.getId(), true);
-                event.setArchived(true);
-                update(event.getId(), event);
-                txn.commit();
+    public long archiveEvents(List<Long> ids, String type, Date startDate, Date endDate, Long accountId, List<Long> domainIds,
+                              long limitPerQuery) {
+        SearchCriteria<EventVO> sc = createEventSearchCriteria(ids, type, startDate, endDate, null, accountId, domainIds);
+        Filter filter = null;
+        if (limitPerQuery > 0) {
+            filter = new Filter(limitPerQuery);
+        }
+
+        long archived;
+        long totalArchived = 0L;
+
+        do {
+            List<EventVO> events = search(sc, filter);
+            if (events.isEmpty()) {
+                break;
             }
-            txn.close();
+
+            archived = archiveEventsInternal(events);
+            totalArchived += archived;
+        } while (limitPerQuery > 0 && archived >= limitPerQuery);

Review Comment:
   Ignoring the configuration and doing it in batches anyways will make having a configuration weird. I think that we can just enable batch operations by default instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-4361727863

   <b>[SF] Trillian test result (tid-15998)</b>
   Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
   Total time taken: 46856 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11443-t15998-kvm-ol8.zip
   Smoke tests completed. 151 look OK, 0 have errors, 0 did not run
   Only failed and skipped tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-3817387969

   Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16622


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "winterhazel (via GitHub)" <gi...@apache.org>.
winterhazel commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-4954362504

   Moving to 4.24, as I will not be able to work on this PR for 4.23.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "blueorangutan (via GitHub) via commits" <co...@cloudstack.apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-6802402966

   @winterhazel a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing and deletion optimizations [cloudstack]

Posted by "winterhazel (via GitHub)" <gi...@apache.org>.
winterhazel commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-3183838091

   @blueorangutan package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-3817131768

   @winterhazel a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "winterhazel (via GitHub)" <gi...@apache.org>.
winterhazel commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-3817124485

   @blueorangutan package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-4283037534

   @winterhazel a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-3695783705

   @blueorangutan test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "vishesh92 (via GitHub)" <gi...@apache.org>.
vishesh92 commented on code in PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#discussion_r4409238986


##########
engine/schema/src/main/java/com/cloud/upgrade/dao/Upgrade42300to2400.java:
##########
@@ -27,4 +29,25 @@ public String[] getUpgradableVersionRange() {
     public String getUpgradedVersion() {
         return "24.0.0";
     }
+
+    @Override
+    public void performDataMigration(Connection conn) {
+        addIndexes(conn);
+        dropOldIndexes(conn);
+    }
+
+    private void addIndexes(Connection conn) {
+        DbUpgradeUtils.addIndexWithNameIfNeeded(conn, "event", "i_event__archived__created__id",
+                "archived", "created", "id");
+        DbUpgradeUtils.addIndexWithNameIfNeeded(conn, "event", "i_event__account_id__archived__created__id",
+                "account_id", "archived", "created", "id");
+        DbUpgradeUtils.addIndexWithNameIfNeeded(conn, "event", "i_event__domain_id__archived__created__id",
+                "domain_id", "archived", "created", "id");
+        DbUpgradeUtils.addIndexWithNameIfNeeded(conn, "event", "i_event__resource_type__resource_id__archived__created__id",
+                "resource_type", "resource_id", "archived", "created", "id");
+    }
+
+    private void dropOldIndexes(Connection conn) {

Review Comment:
   we should also drop `archived` & `i_event__account_id` indexes. Since we are creating composite indexes starting with the same columns.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "DaanHoogland (via GitHub) via commits" <co...@cloudstack.apache.org>.
DaanHoogland commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-6819187281

   @blueorangutan test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "DaanHoogland (via GitHub) via commits" <co...@cloudstack.apache.org>.
DaanHoogland commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-6819261091

   @RosiKyu , can you test this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing and deletion optimizations [cloudstack]

Posted by "winterhazel (via GitHub)" <gi...@apache.org>.
winterhazel commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-3694912275

   > @winterhazel could we extend this to cover archive events as well?
   
   I did some refactoring and extended the event archival to support batch operations as well. The batch size uses `delete.query.batch.size` for now, but I think we can rename this setting to a more generic one in another PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-4358362808

   @blueorangutan test keepEnv


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-4297486019

   <b>[SF] Trillian test result (tid-15931)</b>
   Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
   Total time taken: 86606 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11443-t15931-kvm-ol8.zip
   Smoke tests completed. 141 look OK, 10 have errors, 0 did not run
   Only failed and skipped tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_10_attachAndDetach_iso | `Failure` | 1514.15 | test_vm_life_cycle.py
   ContextSuite context=TestMigrateVMStrictTags>:setup | `Error` | 0.00 | test_vm_strict_host_tags.py
   ContextSuite context=TestRestoreVMStrictTags>:setup | `Error` | 0.00 | test_vm_strict_host_tags.py
   ContextSuite context=TestScaleVMStrictTags>:setup | `Error` | 0.00 | test_vm_strict_host_tags.py
   ContextSuite context=TestVMDeploymentPlannerStrictTags>:setup | `Error` | 0.00 | test_vm_strict_host_tags.py
   test_04_deploy_vnf_appliance | `Error` | 316.73 | test_vnf_templates.py
   test_06_download_detached_volume | `Error` | 308.40 | test_volumes.py
   ContextSuite context=TestVPCConserveModeRules>:setup | `Error` | 0.00 | test_vpc_conserve_mode.py
   ContextSuite context=TestIpv6Vpc>:setup | `Error` | 0.00 | test_vpc_ipv6.py
   ContextSuite context=TestVPCRedundancy>:setup | `Error` | 0.00 | test_vpc_redundant.py
   ContextSuite context=TestVPCNics>:setup | `Error` | 0.00 | test_vpc_router_nics.py
   ContextSuite context=TestRVPCSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVpcRemoteAccessVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVpcSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   test_disable_oobm_ha_state_ineligible | `Error` | 1517.78 | test_hostha_kvm.py
   test_hostha_enable_ha_when_host_in_maintenance | `Error` | 304.02 | test_hostha_kvm.py
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-6040484325

   Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19447


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "winterhazel (via GitHub)" <gi...@apache.org>.
winterhazel commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-6039138639

   @blueorangutan package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "blueorangutan (via GitHub) via commits" <co...@cloudstack.apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-6804408942

   Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 20001


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "blueorangutan (via GitHub) via commits" <co...@cloudstack.apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-6843072819

   <b>[SF] Trillian test result (tid-17557)</b>
   Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
   Total time taken: 63355 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11443-t17557-kvm-ol8.zip
   Smoke tests completed. 157 look OK, 0 have errors, 0 did not run
   Only failed and skipped tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "blueorangutan (via GitHub) via commits" <co...@cloudstack.apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-6819218511

   @DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "blueorangutan (via GitHub) via commits" <co...@cloudstack.apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-6860079568

   <b>[SF] Trillian test result (tid-17574)</b>
   Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
   Total time taken: 63817 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11443-t17574-kvm-ol8.zip
   Smoke tests completed. 156 look OK, 1 have errors, 0 did not run
   Only failed and skipped tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_ssl_offloading_project_vpc | `Failure` | 744.57 | test_ssl_offloading.py
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing and deletion optimizations [cloudstack]

Posted by "winterhazel (via GitHub)" <gi...@apache.org>.
winterhazel commented on code in PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#discussion_r2278977491


##########
api/src/main/java/org/apache/cloudstack/api/command/user/event/DeleteEventsCmd.java:
##########
@@ -97,17 +96,12 @@ public long getEntityOwnerId() {
 
     @Override
     public void execute() {
-        if (ids == null && type == null && endDate == null) {
-            throw new InvalidParameterValueException("either ids, type or enddate must be specified");
-        } else if (startDate != null && endDate == null) {
-            throw new InvalidParameterValueException("enddate must be specified with startdate parameter");
-        }
         boolean result = _mgr.deleteEvents(this);
         if (result) {
             SuccessResponse response = new SuccessResponse(getCommandName());
             setResponseObject(response);
         } else {
-            throw new ServerApiException(ApiErrorCode.INTERNAL_ERROR, "Unable to delete Events, one or more parameters has invalid values");
+            throw new ServerApiException(ApiErrorCode.INTERNAL_ERROR, "Unable to delete any events. One or more parameters have invalid values.");

Review Comment:
   It was already like this, but I'm ok with changing the message



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing and deletion optimizations [cloudstack]

Posted by "winterhazel (via GitHub)" <gi...@apache.org>.
winterhazel commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-3191489096

   @blueorangutan package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing and deletion optimizations [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-3282180094

   Moving to 4.22 milestone as it has some DB changes
   cc @winterhazel @harikrishna-patnala 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing and deletion optimizations [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-3191637296

   Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 14637


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing and deletion optimizations [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-3694913012

   @winterhazel a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-4428151231

   This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "winterhazel (via GitHub)" <gi...@apache.org>.
winterhazel commented on code in PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#discussion_r4207371130


##########
engine/schema/src/main/java/com/cloud/upgrade/dao/Upgrade42210to42300.java:
##########
@@ -89,4 +90,10 @@ protected void updateJsInterpretationEnabledFields(Connection conn, String encry
             logger.warn("Error while decrypting configuration 'js.interpretation.enabled'. The configuration may already be decrypted.");
         }
     }
+
+    private void addIndexes(Connection conn) {
+        DbUpgradeUtils.addIndexWithNameIfNeeded(conn, "event", "i_event__multiple_columns_for_generic_search",
+                "account_id", "domain_id", "archived",  "display", "resource_type", "resource_id", "start_id", "type", "level",
+                "created", "id");

Review Comment:
   > do we need the index to cover all these columns?
   
   @JoaoJandre not really. Initially, I created a single index to speed up all queries `listEvents` could possibly make. I made some some new tests, and it seems that creating a few smaller ones instead of a big one already does the job. I'll change the index.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-4289296451

   @DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "erikbocks (via GitHub)" <gi...@apache.org>.
erikbocks commented on code in PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#discussion_r3498500617


##########
server/src/main/java/com/cloud/server/ManagementServerImpl.java:
##########
@@ -1225,57 +1224,64 @@ protected void checkPortParameters(final String publicPort, final String private
     }
 
     @Override
-    public boolean archiveEvents(final ArchiveEventsCmd cmd) {
-        final Account caller = getCaller();
-        final List<Long> ids = cmd.getIds();
-        boolean result = true;
-        List<Long> permittedAccountIds = new ArrayList<>();
+    public boolean archiveEvents(ArchiveEventsCmd cmd) {
+        List<Long> ids = cmd.getIds();
+        String type = cmd.getType();
+        Date startDate = cmd.getStartDate();
+        Date endDate = cmd.getEndDate();
+
+        validateEventOperationParameters(ids, type, endDate, startDate);
+
+        Long accountId = null;
+        List<Long> domainIds = null;
+        Account caller = getCaller();
+        Account.Type callerType = caller.getType();
 
-        if (_accountService.isNormalUser(caller.getId()) || caller.getType() == Account.Type.PROJECT) {
-            permittedAccountIds.add(caller.getId());
+        if (callerType == Account.Type.NORMAL || callerType == Account.Type.PROJECT) {
+            accountId = caller.getId();
         } else {
-            final DomainVO domain = _domainDao.findById(caller.getDomainId());
-            final List<Long> permittedDomainIds = _domainDao.getDomainChildrenIds(domain.getPath());
-            permittedAccountIds = _accountDao.getAccountIdsForDomains(permittedDomainIds);
+            domainIds = _domainDao.getDomainAndChildrenIds(caller.getDomainId());
         }
 
-        final List<EventVO> events = _eventDao.listToArchiveOrDeleteEvents(ids, cmd.getType(), cmd.getStartDate(), cmd.getEndDate(), permittedAccountIds);
-        final ControlledEntity[] sameOwnerEvents = events.toArray(new ControlledEntity[events.size()]);
-        _accountMgr.checkAccess(CallContext.current().getCallingAccount(), null, false, sameOwnerEvents);
+        long totalArchived = _eventDao.archiveEvents(ids, type, startDate, endDate, accountId, domainIds,
+                ConfigurationManagerImpl.DELETE_QUERY_BATCH_SIZE.value());

Review Comment:
   ```suggestion
           ConfigurationManagerImpl.DELETE_QUERY_BATCH_SIZE.value());
   ```



##########
server/src/main/java/com/cloud/server/ManagementServerImpl.java:
##########
@@ -1225,57 +1224,64 @@ protected void checkPortParameters(final String publicPort, final String private
     }
 
     @Override
-    public boolean archiveEvents(final ArchiveEventsCmd cmd) {
-        final Account caller = getCaller();
-        final List<Long> ids = cmd.getIds();
-        boolean result = true;
-        List<Long> permittedAccountIds = new ArrayList<>();
+    public boolean archiveEvents(ArchiveEventsCmd cmd) {
+        List<Long> ids = cmd.getIds();
+        String type = cmd.getType();
+        Date startDate = cmd.getStartDate();
+        Date endDate = cmd.getEndDate();
+
+        validateEventOperationParameters(ids, type, endDate, startDate);
+
+        Long accountId = null;
+        List<Long> domainIds = null;
+        Account caller = getCaller();
+        Account.Type callerType = caller.getType();
 
-        if (_accountService.isNormalUser(caller.getId()) || caller.getType() == Account.Type.PROJECT) {
-            permittedAccountIds.add(caller.getId());
+        if (callerType == Account.Type.NORMAL || callerType == Account.Type.PROJECT) {
+            accountId = caller.getId();
         } else {
-            final DomainVO domain = _domainDao.findById(caller.getDomainId());
-            final List<Long> permittedDomainIds = _domainDao.getDomainChildrenIds(domain.getPath());
-            permittedAccountIds = _accountDao.getAccountIdsForDomains(permittedDomainIds);
+            domainIds = _domainDao.getDomainAndChildrenIds(caller.getDomainId());
         }
 
-        final List<EventVO> events = _eventDao.listToArchiveOrDeleteEvents(ids, cmd.getType(), cmd.getStartDate(), cmd.getEndDate(), permittedAccountIds);
-        final ControlledEntity[] sameOwnerEvents = events.toArray(new ControlledEntity[events.size()]);
-        _accountMgr.checkAccess(CallContext.current().getCallingAccount(), null, false, sameOwnerEvents);
+        long totalArchived = _eventDao.archiveEvents(ids, type, startDate, endDate, accountId, domainIds,
+                ConfigurationManagerImpl.DELETE_QUERY_BATCH_SIZE.value());
 
-        if (ids != null && events.size() < ids.size()) {
-            return false;
-        }
-        _eventDao.archiveEvents(events);
-        return result;
+        return totalArchived > 0;
     }
 
     @Override
-    public boolean deleteEvents(final DeleteEventsCmd cmd) {
-        final Account caller = getCaller();
-        final List<Long> ids = cmd.getIds();
-        boolean result = true;
-        List<Long> permittedAccountIds = new ArrayList<>();
+    public boolean deleteEvents(DeleteEventsCmd cmd) {
+        List<Long> ids = cmd.getIds();
+        String type = cmd.getType();
+        Date startDate = cmd.getStartDate();
+        Date endDate = cmd.getEndDate();
 
-        if (_accountMgr.isNormalUser(caller.getId()) || caller.getType() == Account.Type.PROJECT) {
-            permittedAccountIds.add(caller.getId());
+        validateEventOperationParameters(ids, type, endDate, startDate);
+
+        Long accountId = null;
+        List<Long> domainIds = null;
+        Account caller = getCaller();
+        Account.Type callerType = caller.getType();
+
+        if (callerType == Account.Type.NORMAL || callerType == Account.Type.PROJECT) {
+            accountId = caller.getId();
         } else {
-            final DomainVO domain = _domainDao.findById(caller.getDomainId());
-            final List<Long> permittedDomainIds = _domainDao.getDomainChildrenIds(domain.getPath());
-            permittedAccountIds = _accountDao.getAccountIdsForDomains(permittedDomainIds);
+            domainIds = _domainDao.getDomainAndChildrenIds(caller.getDomainId());
         }
 
-        final List<EventVO> events = _eventDao.listToArchiveOrDeleteEvents(ids, cmd.getType(), cmd.getStartDate(), cmd.getEndDate(), permittedAccountIds);
-        final ControlledEntity[] sameOwnerEvents = events.toArray(new ControlledEntity[events.size()]);
-        _accountMgr.checkAccess(CallContext.current().getCallingAccount(), null, false, sameOwnerEvents);
+        long totalRemoved = _eventDao.purgeAll(ids, startDate, endDate, null, type, accountId, domainIds,
+                ConfigurationManagerImpl.DELETE_QUERY_BATCH_SIZE.value());

Review Comment:
   ```suggestion
           ConfigurationManagerImpl.DELETE_QUERY_BATCH_SIZE.value());
   ```



##########
engine/schema/src/main/java/com/cloud/upgrade/dao/Upgrade42210to42300.java:
##########
@@ -89,4 +90,10 @@ protected void updateJsInterpretationEnabledFields(Connection conn, String encry
             logger.warn("Error while decrypting configuration 'js.interpretation.enabled'. The configuration may already be decrypted.");
         }
     }
+
+    private void addIndexes(Connection conn) {
+        DbUpgradeUtils.addIndexWithNameIfNeeded(conn, "event", "i_event__multiple_columns_for_generic_search",
+                "account_id", "domain_id", "archived",  "display", "resource_type", "resource_id", "start_id", "type", "level",
+                "created", "id");

Review Comment:
   @winterhazel 
   can you take a look at this comment?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "RosiKyu (via GitHub) via commits" <co...@cloudstack.apache.org>.
RosiKyu merged PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing and deletion optimizations [cloudstack]

Posted by "winterhazel (via GitHub)" <gi...@apache.org>.
winterhazel commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-3273016705

   > @winterhazel could we extend this to cover archive events as well? Also, I’m not aware of any other DB changes slated for 4.20.2. If this is the only one, would it be safer to target 4.22? Not blocking—just a suggestion.
   
   @shwstppr I'll have a look into whether event archiving needs or would benefit from any changes.
   
   About targeting 4.22 instead: the only DB change here is the addition of an index. This can not cause any inconsistency as far as I know, so it should be safe going to 4.20.2.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing and deletion optimizations [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-3184119724

   Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 14617


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing and deletion optimizations [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-3191492936

   @winterhazel a [SL] Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing and deletion optimizations [cloudstack]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#discussion_r2276872903


##########
api/src/main/java/org/apache/cloudstack/api/command/user/event/DeleteEventsCmd.java:
##########
@@ -97,17 +96,12 @@ public long getEntityOwnerId() {
 
     @Override
     public void execute() {
-        if (ids == null && type == null && endDate == null) {
-            throw new InvalidParameterValueException("either ids, type or enddate must be specified");
-        } else if (startDate != null && endDate == null) {
-            throw new InvalidParameterValueException("enddate must be specified with startdate parameter");
-        }
         boolean result = _mgr.deleteEvents(this);
         if (result) {
             SuccessResponse response = new SuccessResponse(getCommandName());
             setResponseObject(response);
         } else {
-            throw new ServerApiException(ApiErrorCode.INTERNAL_ERROR, "Unable to delete Events, one or more parameters has invalid values");
+            throw new ServerApiException(ApiErrorCode.INTERNAL_ERROR, "Unable to delete any events. One or more parameters have invalid values.");

Review Comment:
   The error message suggests parameter validation issues, but the method now returns false when no events are deleted (totalRemoved = 0), which could happen even with valid parameters if no matching events exist. Consider a more accurate message like "No events found matching the specified criteria" or differentiate between validation errors and no matching records.
   ```suggestion
               throw new ServerApiException(ApiErrorCode.INTERNAL_ERROR, "Unable to delete any events. No events found matching the specified criteria or one or more parameters have invalid values.");
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "JoaoJandre (via GitHub)" <gi...@apache.org>.
JoaoJandre commented on code in PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#discussion_r3350468818


##########
engine/schema/src/main/java/com/cloud/upgrade/dao/Upgrade42210to42300.java:
##########
@@ -89,4 +90,10 @@ protected void updateJsInterpretationEnabledFields(Connection conn, String encry
             logger.warn("Error while decrypting configuration 'js.interpretation.enabled'. The configuration may already be decrypted.");
         }
     }
+
+    private void addIndexes(Connection conn) {
+        DbUpgradeUtils.addIndexWithNameIfNeeded(conn, "event", "i_event__multiple_columns_for_generic_search",
+                "account_id", "domain_id", "archived",  "display", "resource_type", "resource_id", "start_id", "type", "level",
+                "created", "id");

Review Comment:
   I'm not sure that the order of the indexes here is the best. Shouldn't every record with a given account_id have the same domain_id? If so, I think we should flip the order of those. Also, I think that the archived column might be the best one to be  the first in the index, as it is probably the most selective. 
   
   Also, do we need the index to cover all these columns? Did you test the data recovery with fewer columns? How does this index compare to an index with fewer columns in terms of performance of insertions and deletions?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "winterhazel (via GitHub)" <gi...@apache.org>.
winterhazel commented on code in PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#discussion_r3170249886


##########
api/src/main/java/org/apache/cloudstack/api/command/user/event/ArchiveEventsCmd.java:
##########
@@ -97,17 +112,12 @@ public long getEntityOwnerId() {
 
     @Override
     public void execute() {
-        if (ids == null && type == null && endDate == null) {
-            throw new InvalidParameterValueException("either ids, type or enddate must be specified");
-        } else if (startDate != null && endDate == null) {
-            throw new InvalidParameterValueException("enddate must be specified with startdate parameter");
-        }
         boolean result = _mgr.archiveEvents(this);
         if (result) {
             SuccessResponse response = new SuccessResponse(getCommandName());
             setResponseObject(response);
-        } else {
-            throw new ServerApiException(ApiErrorCode.INTERNAL_ERROR, "Unable to archive Events, one or more parameters has invalid values");
+            return;
         }
+        throw new ServerApiException(ApiErrorCode.INTERNAL_ERROR, "Unable to archive Events. One or more parameters have invalid values.");
     }

Review Comment:
   This was the previous behavior. I do not want to change the response of the API in this patch, just optimize its operations.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "winterhazel (via GitHub)" <gi...@apache.org>.
winterhazel commented on code in PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#discussion_r3170277293


##########
engine/schema/src/main/java/com/cloud/event/dao/EventDaoImpl.java:
##########
@@ -33,83 +38,90 @@
 
 @Component
 public class EventDaoImpl extends GenericDaoBase<EventVO, Long> implements EventDao {
-    protected final SearchBuilder<EventVO> CompletedEventSearch;
+
     protected final SearchBuilder<EventVO> ToArchiveOrDeleteEventSearch;
 
     public EventDaoImpl() {
-        CompletedEventSearch = createSearchBuilder();
-        CompletedEventSearch.and("state", CompletedEventSearch.entity().getState(), SearchCriteria.Op.EQ);
-        CompletedEventSearch.and("startId", CompletedEventSearch.entity().getStartId(), SearchCriteria.Op.EQ);
-        CompletedEventSearch.and("archived", CompletedEventSearch.entity().getArchived(), Op.EQ);
-        CompletedEventSearch.done();
-
         ToArchiveOrDeleteEventSearch = createSearchBuilder();
+        ToArchiveOrDeleteEventSearch.select("id", SearchCriteria.Func.NATIVE, ToArchiveOrDeleteEventSearch.entity().getId());
         ToArchiveOrDeleteEventSearch.and("id", ToArchiveOrDeleteEventSearch.entity().getId(), Op.IN);
         ToArchiveOrDeleteEventSearch.and("type", ToArchiveOrDeleteEventSearch.entity().getType(), Op.EQ);
-        ToArchiveOrDeleteEventSearch.and("accountIds", ToArchiveOrDeleteEventSearch.entity().getAccountId(), Op.IN);
+        ToArchiveOrDeleteEventSearch.and("accountId", ToArchiveOrDeleteEventSearch.entity().getAccountId(), Op.EQ);
+        ToArchiveOrDeleteEventSearch.and("domainIds", ToArchiveOrDeleteEventSearch.entity().getDomainId(), Op.IN);
         ToArchiveOrDeleteEventSearch.and("createdDateB", ToArchiveOrDeleteEventSearch.entity().getCreateDate(), Op.BETWEEN);
         ToArchiveOrDeleteEventSearch.and("createdDateL", ToArchiveOrDeleteEventSearch.entity().getCreateDate(), Op.LTEQ);
+        ToArchiveOrDeleteEventSearch.and("createdDateLT", ToArchiveOrDeleteEventSearch.entity().getCreateDate(), Op.LT);
         ToArchiveOrDeleteEventSearch.and("archived", ToArchiveOrDeleteEventSearch.entity().getArchived(), Op.EQ);
         ToArchiveOrDeleteEventSearch.done();
     }
 
-    @Override
-    public List<EventVO> searchAllEvents(SearchCriteria<EventVO> sc, Filter filter) {
-        return listIncludingRemovedBy(sc, filter);
-    }
-
-    @Override
-    public List<EventVO> listOlderEvents(Date oldTime) {
-        if (oldTime == null)
-            return null;
-        SearchCriteria<EventVO> sc = createSearchCriteria();
-        sc.addAnd("createDate", SearchCriteria.Op.LT, oldTime);
-        sc.addAnd("archived", SearchCriteria.Op.EQ, false);
-        return listIncludingRemovedBy(sc, null);
-    }
-
-    @Override
-    public EventVO findCompletedEvent(long startId) {
-        SearchCriteria<EventVO> sc = CompletedEventSearch.create();
-        sc.setParameters("state", State.Completed);
-        sc.setParameters("startId", startId);
-        sc.setParameters("archived", false);
-        return findOneIncludingRemovedBy(sc);
-    }
-
-    @Override
-    public List<EventVO> listToArchiveOrDeleteEvents(List<Long> ids, String type, Date startDate, Date endDate, List<Long> accountIds) {
+    private SearchCriteria<EventVO> createEventSearchCriteria(List<Long> ids, String type, Date startDate, Date endDate,
+                                                              Date limitDate, Long accountId, List<Long> domainIds) {
         SearchCriteria<EventVO> sc = ToArchiveOrDeleteEventSearch.create();
-        if (ids != null) {
-            sc.setParameters("id", ids.toArray(new Object[ids.size()]));
+
+        if (CollectionUtils.isNotEmpty(ids)) {
+            sc.setParameters("id", ids.toArray(new Object[0]));
         }
-        if (type != null) {
-            sc.setParameters("type", type);
+        if (CollectionUtils.isNotEmpty(domainIds)) {
+            sc.setParameters("domainIds", domainIds.toArray(new Object[0]));
         }
         if (startDate != null && endDate != null) {
             sc.setParameters("createdDateB", startDate, endDate);
         } else if (endDate != null) {
             sc.setParameters("createdDateL", endDate);
         }
-        if (accountIds != null && !accountIds.isEmpty()) {
-            sc.setParameters("accountIds", accountIds.toArray(new Object[accountIds.size()]));
-        }
+        sc.setParametersIfNotNull("accountId", accountId);
+        sc.setParametersIfNotNull("createdDateLT", limitDate);
+        sc.setParametersIfNotNull("type", type);
         sc.setParameters("archived", false);
-        return search(sc, null);
+
+        return sc;
     }
 
     @Override
-    public void archiveEvents(List<EventVO> events) {
-        if (events != null && !events.isEmpty()) {
-            TransactionLegacy txn = TransactionLegacy.currentTxn();
-            txn.start();
-            for (EventVO event : events) {
-                event = lockRow(event.getId(), true);
-                event.setArchived(true);
-                update(event.getId(), event);
-                txn.commit();
+    public long archiveEvents(List<Long> ids, String type, Date startDate, Date endDate, Long accountId, List<Long> domainIds,
+                              long limitPerQuery) {
+        SearchCriteria<EventVO> sc = createEventSearchCriteria(ids, type, startDate, endDate, null, accountId, domainIds);
+        Filter filter = null;
+        if (limitPerQuery > 0) {
+            filter = new Filter(limitPerQuery);
+        }
+
+        long archived;
+        long totalArchived = 0L;
+
+        do {
+            List<EventVO> events = search(sc, filter);
+            if (events.isEmpty()) {
+                break;
             }
-            txn.close();
+
+            archived = archiveEventsInternal(events);
+            totalArchived += archived;
+        } while (limitPerQuery > 0 && archived >= limitPerQuery);

Review Comment:
   This is not that big of a problem. The API can just be called again if there was a concurrency issue that resulted in an event not ending as archived.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "winterhazel (via GitHub)" <gi...@apache.org>.
winterhazel commented on code in PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#discussion_r3170304149


##########
server/src/main/java/com/cloud/server/ManagementServerImpl.java:
##########
@@ -1225,57 +1224,64 @@ protected void checkPortParameters(final String publicPort, final String private
     }
 
     @Override
-    public boolean archiveEvents(final ArchiveEventsCmd cmd) {
-        final Account caller = getCaller();
-        final List<Long> ids = cmd.getIds();
-        boolean result = true;
-        List<Long> permittedAccountIds = new ArrayList<>();
+    public boolean archiveEvents(ArchiveEventsCmd cmd) {
+        List<Long> ids = cmd.getIds();
+        String type = cmd.getType();
+        Date startDate = cmd.getStartDate();
+        Date endDate = cmd.getEndDate();
+
+        validateEventOperationParameters(ids, type, endDate, startDate);
+
+        Long accountId = null;
+        List<Long> domainIds = null;
+        Account caller = getCaller();
+        Account.Type callerType = caller.getType();
 
-        if (_accountService.isNormalUser(caller.getId()) || caller.getType() == Account.Type.PROJECT) {
-            permittedAccountIds.add(caller.getId());
+        if (callerType == Account.Type.NORMAL || callerType == Account.Type.PROJECT) {
+            accountId = caller.getId();
         } else {
-            final DomainVO domain = _domainDao.findById(caller.getDomainId());
-            final List<Long> permittedDomainIds = _domainDao.getDomainChildrenIds(domain.getPath());
-            permittedAccountIds = _accountDao.getAccountIdsForDomains(permittedDomainIds);
+            domainIds = _domainDao.getDomainAndChildrenIds(caller.getDomainId());
         }
 
-        final List<EventVO> events = _eventDao.listToArchiveOrDeleteEvents(ids, cmd.getType(), cmd.getStartDate(), cmd.getEndDate(), permittedAccountIds);
-        final ControlledEntity[] sameOwnerEvents = events.toArray(new ControlledEntity[events.size()]);
-        _accountMgr.checkAccess(CallContext.current().getCallingAccount(), null, false, sameOwnerEvents);
+        long totalArchived = _eventDao.archiveEvents(ids, type, startDate, endDate, accountId, domainIds,
+                ConfigurationManagerImpl.DELETE_QUERY_BATCH_SIZE.value());
 
-        if (ids != null && events.size() < ids.size()) {
-            return false;
-        }
-        _eventDao.archiveEvents(events);
-        return result;
+        return totalArchived > 0;

Review Comment:
   This was the previous behavior. I do not want to change the response of the API in this patch, just optimize its operations.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "blueorangutan (via GitHub) via commits" <co...@cloudstack.apache.org>.
blueorangutan commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-6837629586

   @RosiKyu a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "RosiKyu (via GitHub) via commits" <co...@cloudstack.apache.org>.
RosiKyu commented on PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#issuecomment-6837617644

   @blueorangutan test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Events listing, deletion and archival optimizations [cloudstack]

Posted by "winterhazel (via GitHub) via commits" <co...@cloudstack.apache.org>.
winterhazel commented on code in PR #11443:
URL: https://github.com/apache/cloudstack/pull/11443#discussion_r4539281325


##########
server/src/main/java/com/cloud/server/ManagementServerImpl.java:
##########
@@ -1225,57 +1224,64 @@ protected void checkPortParameters(final String publicPort, final String private
     }
 
     @Override
-    public boolean archiveEvents(final ArchiveEventsCmd cmd) {
-        final Account caller = getCaller();
-        final List<Long> ids = cmd.getIds();
-        boolean result = true;
-        List<Long> permittedAccountIds = new ArrayList<>();
+    public boolean archiveEvents(ArchiveEventsCmd cmd) {
+        List<Long> ids = cmd.getIds();
+        String type = cmd.getType();
+        Date startDate = cmd.getStartDate();
+        Date endDate = cmd.getEndDate();
+
+        validateEventOperationParameters(ids, type, endDate, startDate);
+
+        Long accountId = null;
+        List<Long> domainIds = null;
+        Account caller = getCaller();
+        Account.Type callerType = caller.getType();
 
-        if (_accountService.isNormalUser(caller.getId()) || caller.getType() == Account.Type.PROJECT) {
-            permittedAccountIds.add(caller.getId());
+        if (callerType == Account.Type.NORMAL || callerType == Account.Type.PROJECT) {
+            accountId = caller.getId();
         } else {
-            final DomainVO domain = _domainDao.findById(caller.getDomainId());
-            final List<Long> permittedDomainIds = _domainDao.getDomainChildrenIds(domain.getPath());
-            permittedAccountIds = _accountDao.getAccountIdsForDomains(permittedDomainIds);
+            domainIds = _domainDao.getDomainAndChildrenIds(caller.getDomainId());
         }
 
-        final List<EventVO> events = _eventDao.listToArchiveOrDeleteEvents(ids, cmd.getType(), cmd.getStartDate(), cmd.getEndDate(), permittedAccountIds);
-        final ControlledEntity[] sameOwnerEvents = events.toArray(new ControlledEntity[events.size()]);
-        _accountMgr.checkAccess(CallContext.current().getCallingAccount(), null, false, sameOwnerEvents);
+        long totalArchived = _eventDao.archiveEvents(ids, type, startDate, endDate, accountId, domainIds,
+                ConfigurationManagerImpl.DELETE_QUERY_BATCH_SIZE.value());
 
-        if (ids != null && events.size() < ids.size()) {
-            return false;
-        }
-        _eventDao.archiveEvents(events);
-        return result;
+        return totalArchived > 0;

Review Comment:
   Yes, I'll revert this behavior change.



##########
server/src/main/java/com/cloud/server/ManagementServerImpl.java:
##########
@@ -1225,57 +1224,64 @@ protected void checkPortParameters(final String publicPort, final String private
     }
 
     @Override
-    public boolean archiveEvents(final ArchiveEventsCmd cmd) {
-        final Account caller = getCaller();
-        final List<Long> ids = cmd.getIds();
-        boolean result = true;
-        List<Long> permittedAccountIds = new ArrayList<>();
+    public boolean archiveEvents(ArchiveEventsCmd cmd) {
+        List<Long> ids = cmd.getIds();
+        String type = cmd.getType();
+        Date startDate = cmd.getStartDate();
+        Date endDate = cmd.getEndDate();
+
+        validateEventOperationParameters(ids, type, endDate, startDate);
+
+        Long accountId = null;
+        List<Long> domainIds = null;
+        Account caller = getCaller();
+        Account.Type callerType = caller.getType();
 
-        if (_accountService.isNormalUser(caller.getId()) || caller.getType() == Account.Type.PROJECT) {
-            permittedAccountIds.add(caller.getId());
+        if (callerType == Account.Type.NORMAL || callerType == Account.Type.PROJECT) {
+            accountId = caller.getId();
         } else {
-            final DomainVO domain = _domainDao.findById(caller.getDomainId());
-            final List<Long> permittedDomainIds = _domainDao.getDomainChildrenIds(domain.getPath());
-            permittedAccountIds = _accountDao.getAccountIdsForDomains(permittedDomainIds);
+            domainIds = _domainDao.getDomainAndChildrenIds(caller.getDomainId());

Review Comment:
   Ok, will do.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org