You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@syncope.apache.org by GitBox <gi...@apache.org> on 2022/07/13 13:22:43 UTC
[GitHub] [syncope] ilgrosso opened a new pull request, #360: Consolidating Provision, Mapping and Items into single JSON column
ilgrosso opened a new pull request, #360:
URL: https://github.com/apache/syncope/pull/360
The purpose of this PR is to explore the possibility to consolidate a set of JPA entities from their own tables into single JSON column.
The idea is that such strategy should improve the general provisioning performance, especially during propagation, as no joins will be required to fetch the required information.
At least some comparison tests shall be performed against current `master` branch and this PR to understand if the proposed change is effective or not.
It could be an idea to do the same with Connector and Resource capabilitites.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@syncope.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [syncope] github-code-scanning[bot] commented on a diff in pull request #360: [SYNCOPE-1689] Consolidate Provision, Mapping and Items into single JSON column
Posted by GitBox <gi...@apache.org>.
github-code-scanning[bot] commented on code in PR #360:
URL: https://github.com/apache/syncope/pull/360#discussion_r921045612
##########
client/idrepo/console/src/main/java/org/apache/syncope/client/console/commons/StatusProvider.java:
##########
@@ -28,14 +28,14 @@
import org.apache.syncope.client.console.wicket.markup.html.form.ActionLink;
import org.apache.syncope.client.ui.commons.status.ConnObjectWrapper;
import org.apache.syncope.common.lib.to.AnyTO;
-import org.apache.syncope.common.lib.to.ConnObjectTO;
+import org.apache.syncope.common.lib.to.ConnObject;
public interface StatusProvider extends Serializable {
- Optional<Pair<ConnObjectTO, ConnObjectTO>> get(
+ Optional<Pair<ConnObject, ConnObject>> get(
String anyTypeKey, String connObjectKeyValue, String resource);
- List<Triple<ConnObjectTO, ConnObjectWrapper, String>> get(AnyTO any, Collection<String> resources);
+ List<Triple<ConnObject, ConnObjectWrapper, String>> get(AnyTO any, Collection<String> resources);
Review Comment:
## Useless parameter
The parameter resources is unused.
[Show more details](https://github.com/apache/syncope/security/code-scanning/1106)
##########
common/idm/lib/src/main/java/org/apache/syncope/common/lib/to/PagedConnObjectResult.java:
##########
@@ -109,7 +109,7 @@
@JacksonXmlElementWrapper(localName = "result")
@JacksonXmlProperty(localName = "item")
- public List<ConnObjectTO> getResult() {
+ public List<ConnObject> getResult() {
Review Comment:
## Exposing internal representation
getResult exposes the internal representation stored in field result. The value may be modified [after this call to getResult](1).
[Show more details](https://github.com/apache/syncope/security/code-scanning/1115)
##########
common/am/lib/src/main/java/org/apache/syncope/common/lib/to/AttrRepoTO.java:
##########
@@ -77,7 +77,7 @@
this.order = order;
}
- public List<ItemTO> getItems() {
+ public List<Item> getItems() {
Review Comment:
## Exposing internal representation
getItems exposes the internal representation stored in field items. The value may be modified [after this call to getItems](1).
getItems exposes the internal representation stored in field items. The value may be modified [after this call to getItems](2).
getItems exposes the internal representation stored in field items. The value may be modified [after this call to getItems](3).
[Show more details](https://github.com/apache/syncope/security/code-scanning/1107)
##########
common/idm/lib/src/main/java/org/apache/syncope/common/lib/to/ResourceTO.java:
##########
@@ -229,21 +229,21 @@
}
@JsonIgnore
- public Optional<ProvisionTO> getProvision(final String anyType) {
+ public Optional<Provision> getProvision(final String anyType) {
return provisions.stream().filter(
provision -> anyType != null && anyType.equals(provision.getAnyType())).
findFirst();
}
- public List<ProvisionTO> getProvisions() {
+ public List<Provision> getProvisions() {
Review Comment:
## Exposing internal representation
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](1).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](2).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](3).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](4).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](5).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](6).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](7).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](8).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](9).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](10).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](11).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](12).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](13).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](14).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](15).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](16).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](17).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](18).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](19).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](20).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](21).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](22).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](23).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](24).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](25).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](26).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](27).
getProvisions exposes the internal representation stored in field provisions. The value may be modified [after this call to getProvisions](28).
getProvisions exposes the internal representation stored in field provisi
[Show more details](https://github.com/apache/syncope/security/code-scanning/1118)
##########
common/idm/lib/src/main/java/org/apache/syncope/common/lib/to/Mapping.java:
##########
@@ -44,44 +41,9 @@
this.connObjectLink = connObjectLink;
}
- @Override
- public ItemTO getConnObjectKeyItem() {
- return getItems().stream().filter(ItemTO::isConnObjectKey).findFirst().orElse(null);
- }
-
- protected boolean addConnObjectKeyItem(final ItemTO connObjectItem) {
- connObjectItem.setMandatoryCondition("true");
- connObjectItem.setConnObjectKey(true);
-
- return this.add(connObjectItem);
- }
-
- @Override
- public boolean setConnObjectKeyItem(final ItemTO connObjectKeyItem) {
- return Optional.ofNullable(connObjectKeyItem)
- .map(this::addConnObjectKeyItem).orElseGet(() -> remove(getConnObjectKeyItem()));
- }
-
- @JacksonXmlElementWrapper(localName = "items")
- @JacksonXmlProperty(localName = "item")
- @Override
- public List<ItemTO> getItems() {
- return items;
- }
-
- @Override
- public boolean add(final ItemTO item) {
- return Optional.ofNullable(item)
- .filter(itemTO -> this.items.contains(itemTO) || this.items.add(itemTO)).isPresent();
- }
-
- public boolean remove(final ItemTO item) {
- return this.items.remove(item);
- }
-
@JacksonXmlElementWrapper(localName = "linkingItems")
@JacksonXmlProperty(localName = "linkingItem")
- public List<ItemTO> getLinkingItems() {
+ public List<Item> getLinkingItems() {
Review Comment:
## Exposing internal representation
getLinkingItems exposes the internal representation stored in field linkingItems. The value may be modified [after this call to getLinkingItems](1).
getLinkingItems exposes the internal representation stored in field linkingItems. The value may be modified [after this call to getLinkingItems](2).
getLinkingItems exposes the internal representation stored in field linkingItems. The value may be modified [after this call to getLinkingItems](3).
[Show more details](https://github.com/apache/syncope/security/code-scanning/1114)
##########
common/am/lib/src/main/java/org/apache/syncope/common/lib/to/AuthModuleTO.java:
##########
@@ -77,7 +77,7 @@
this.order = order;
}
- public List<ItemTO> getItems() {
+ public List<Item> getItems() {
Review Comment:
## Exposing internal representation
getItems exposes the internal representation stored in field items. The value may be modified [after this call to getItems](1).
getItems exposes the internal representation stored in field items. The value may be modified [after this call to getItems](2).
getItems exposes the internal representation stored in field items. The value may be modified [after this call to getItems](3).
getItems exposes the internal representation stored in field items. The value may be modified [after this call to getItems](4).
getItems exposes the internal representation stored in field items. The value may be modified [after this call to getItems](5).
getItems exposes the internal representation stored in field items. The value may be modified [after this call to getItems](6).
[Show more details](https://github.com/apache/syncope/security/code-scanning/1108)
##########
common/idrepo/lib/src/main/java/org/apache/syncope/common/lib/to/ItemContainer.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.syncope.common.lib.to;
+
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlElementWrapper;
+import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlProperty;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import org.apache.commons.lang3.builder.EqualsBuilder;
+import org.apache.commons.lang3.builder.HashCodeBuilder;
+
+public abstract class ItemContainer implements Serializable {
+
+ private static final long serialVersionUID = 3637981417797873343L;
+
+ private final List<Item> items = new ArrayList<>();
+
+ @JsonIgnore
+ public Optional<Item> getConnObjectKeyItem() {
+ return getItems().stream().filter(Item::isConnObjectKey).findFirst();
+ }
+
+ protected boolean addConnObjectKeyItem(final Item connObjectItem) {
+ connObjectItem.setMandatoryCondition("true");
+ connObjectItem.setConnObjectKey(true);
+
+ return add(connObjectItem);
+ }
+
+ public boolean setConnObjectKeyItem(final Item connObjectKeyItem) {
+ return Optional.ofNullable(connObjectKeyItem).
+ map(this::addConnObjectKeyItem).
+ orElseGet(() -> getConnObjectKeyItem().map(items::remove).orElse(false));
+ }
+
+ @JacksonXmlElementWrapper(localName = "items")
+ @JacksonXmlProperty(localName = "item")
+ public List<Item> getItems() {
Review Comment:
## Exposing internal representation
getItems exposes the internal representation stored in field items. The value may be modified [after this call to getItems](1).
getItems exposes the internal representation stored in field items. The value may be modified [after this call to getItems](2).
getItems exposes the internal representation stored in field items. The value may be modified [after this call to getItems](3).
getItems exposes the internal representation stored in field items. The value may be modified [after this call to getItems](4).
getItems exposes the internal representation stored in field items. The value may be modified [after this call to getItems](5).
getItems exposes the internal representation stored in field items. The value may be modified [after this call to getItems](6).
getItems exposes the internal representation stored in field items. The value may be modified [after this call to getItems](7).
getItems exposes the internal representation stored in field items. The value may be modified [after this call to getItems](8).
getItems exposes the internal representation stored in field items. The value may be modified [after this call to getItems](9).
getItems exposes the internal representation stored in field items. The value may be modified [after this call to getItems](10).
getItems exposes the internal representation stored in field items. The value may be modified [after this call to getItems](11).
[Show more details](https://github.com/apache/syncope/security/code-scanning/1113)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@syncope.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [syncope] github-code-scanning[bot] commented on a diff in pull request #360: Consolidating Provision, Mapping and Items into single JSON column
Posted by GitBox <gi...@apache.org>.
github-code-scanning[bot] commented on code in PR #360:
URL: https://github.com/apache/syncope/pull/360#discussion_r920131551
##########
core/persistence-jpa/src/main/java/org/apache/syncope/core/persistence/jpa/PersistenceContext.java:
##########
@@ -611,7 +611,8 @@
final GroupDAO groupDAO,
final PolicyDAO policyDAO,
final VirSchemaDAO virSchemaDAO,
- final RealmDAO realmDAO) {
+ final RealmDAO realmDAO,
+ final EntityCacheDAO entityCacheDAO) {
Review Comment:
## Useless parameter
The parameter entityCacheDAO is unused.
[Show more details](https://github.com/apache/syncope/security/code-scanning/1085)
##########
core/provisioning-java/src/main/java/org/apache/syncope/core/provisioning/java/pushpull/SinglePushJobDelegate.java:
##########
@@ -45,15 +47,16 @@
public class SinglePushJobDelegate extends PushJobDelegate implements SyncopeSinglePushExecutor {
@Autowired
- private ImplementationDAO implementationDAO;
+ protected ImplementationDAO implementationDAO;
- private List<PushActions> before(
- final Provision provision,
+ protected List<PushActions> before(
+ final ExternalResource resource,
+ final ProvisionTO provision,
Review Comment:
## Useless parameter
The parameter provision is unused.
[Show more details](https://github.com/apache/syncope/security/code-scanning/1092)
##########
core/provisioning-java/src/main/java/org/apache/syncope/core/provisioning/java/DefaultMappingManager.java:
##########
@@ -230,10 +242,10 @@
* @param connObjectKey connector object key
* @return the value to be propagated as __NAME__
*/
- protected Name evaluateNAME(final Any<?> any, final Provision provision, final String connObjectKey) {
+ protected Name evaluateNAME(final Any<?> any, final ProvisionTO provision, final String connObjectKey) {
if (StringUtils.isBlank(connObjectKey)) {
// LOG error but avoid to throw exception: leave it to the external resource
- LOG.warn("Missing ConnObjectKey value for {}: ", provision.getResource());
+ LOG.warn("Missing ConnObjectKey value for {}: ", provision);
Review Comment:
## Use of default toString()
Default toString(): ProvisionTO inherits toString() from Object, and so is not suitable for printing.
[Show more details](https://github.com/apache/syncope/security/code-scanning/1096)
##########
core/provisioning-java/src/main/java/org/apache/syncope/core/provisioning/java/DefaultMappingManager.java:
##########
@@ -607,15 +625,16 @@
@SuppressWarnings("unchecked")
@Override
public Pair<AttrSchemaType, List<PlainAttrValue>> getIntValues(
- final Provision provision,
- final Item mapItem,
+ final ExternalResource resource,
+ final ProvisionTO provision,
+ final ItemTO mapItem,
final IntAttrName intAttrName,
final AttrSchemaType schemaType,
final Any<?> any,
final AccountGetter usernameAccountGetter,
final PlainAttrGetter plainAttrGetter) {
- LOG.debug("Get internal values for {} as '{}' on {}", any, mapItem.getIntAttrName(), provision.getResource());
+ LOG.debug("Get internal values for {} as '{}' on {}", any, mapItem.getIntAttrName(), provision);
Review Comment:
## Use of default toString()
Default toString(): ProvisionTO inherits toString() from Object, and so is not suitable for printing.
[Show more details](https://github.com/apache/syncope/security/code-scanning/1101)
##########
core/provisioning-java/src/main/java/org/apache/syncope/core/provisioning/java/pushpull/stream/StreamPushJobDelegate.java:
##########
@@ -147,7 +144,7 @@
try {
ExternalResource resource = externalResource(anyType, columns, propagationActions);
- Provision provision = resource.getProvisions().get(0);
+ ProvisionTO provision = resource.getProvisions().get(0);
Review Comment:
## Unread local variable
Variable 'ProvisionTO provision' is never read.
[Show more details](https://github.com/apache/syncope/security/code-scanning/1093)
##########
core/provisioning-java/src/main/java/org/apache/syncope/core/provisioning/java/DefaultMappingManager.java:
##########
@@ -262,10 +274,10 @@
* @param connObjectKey connector object key
* @return the value to be propagated as __NAME__
*/
- protected Name evaluateNAME(final Realm realm, final OrgUnit orgUnit, final String connObjectKey) {
+ protected Name evaluateNAME(final Realm realm, final OrgUnitTO orgUnit, final String connObjectKey) {
if (StringUtils.isBlank(connObjectKey)) {
// LOG error but avoid to throw exception: leave it to the external resource
- LOG.warn("Missing ConnObjectKey value for {}: ", orgUnit.getResource());
+ LOG.warn("Missing ConnObjectKey value for {}: ", orgUnit);
Review Comment:
## Use of default toString()
Default toString(): OrgUnitTO inherits toString() from Object, and so is not suitable for printing.
[Show more details](https://github.com/apache/syncope/security/code-scanning/1097)
##########
core/provisioning-java/src/main/java/org/apache/syncope/core/provisioning/java/DefaultMappingManager.java:
##########
@@ -884,18 +908,24 @@
@Transactional(readOnly = true)
@Override
- public Optional<String> getConnObjectKeyValue(final Any<?> any, final Provision provision) {
- Optional<? extends MappingItem> connObjectKeyItem = provision.getMapping().getConnObjectKeyItem();
+ public Optional<String> getConnObjectKeyValue(
+ final Any<?> any,
+ final ExternalResource resource,
+ final ProvisionTO provision) {
+
+ Optional<ItemTO> connObjectKeyItem = provision.getMapping().getConnObjectKeyItem();
if (connObjectKeyItem.isEmpty()) {
- LOG.error("Unable to locate conn object key item for " + provision.getMapping().getKey());
+ LOG.error("Unable to locate conn object key item for " + provision.getMapping());
Review Comment:
## Use of default toString()
Default toString(): MappingTO inherits toString() from Object, and so is not suitable for printing.
[Show more details](https://github.com/apache/syncope/security/code-scanning/1102)
##########
core/idm/logic/src/main/java/org/apache/syncope/core/logic/ReconciliationLogic.java:
##########
@@ -483,24 +514,26 @@
final Set<String> moreAttrsToGet,
final PullTaskTO pullTask) {
- Provision provision = getProvision(anyTypeKey, resourceKey);
+ Triple<AnyType, ExternalResource, ProvisionTO> triple = getProvision(anyTypeKey, resourceKey);
- Any<?> any = getAny(provision, anyKey);
-
- if (provision.getMapping().getConnObjectKeyItem().isEmpty()) {
+ if (triple.getRight().getMapping().getConnObjectKeyItem().isEmpty()) {
throw new NotFoundException(
- "ConnObjectKey cannot be determined for mapping " + provision.getMapping().getKey());
+ "ConnObjectKey cannot be determined for mapping " + triple.getRight().getMapping());
Review Comment:
## Use of default toString()
Default toString(): MappingTO inherits toString() from Object, and so is not suitable for printing.
[Show more details](https://github.com/apache/syncope/security/code-scanning/1095)
##########
core/provisioning-java/src/main/java/org/apache/syncope/core/provisioning/java/DefaultMappingManager.java:
##########
@@ -911,22 +941,22 @@
@Transactional(readOnly = true)
@Override
- public Optional<String> getConnObjectKeyValue(final Realm realm, final OrgUnit orgUnit) {
- Optional<? extends OrgUnitItem> connObjectKeyItem = orgUnit.getConnObjectKeyItem();
+ public Optional<String> getConnObjectKeyValue(final Realm realm, final OrgUnitTO orgUnit) {
+ Optional<ItemTO> connObjectKeyItem = orgUnit.getConnObjectKeyItem();
if (connObjectKeyItem.isEmpty()) {
- LOG.error("Unable to locate conn object key item for " + orgUnit.getKey());
+ LOG.error("Unable to locate conn object key item for " + orgUnit);
Review Comment:
## Use of default toString()
Default toString(): OrgUnitTO inherits toString() from Object, and so is not suitable for printing.
[Show more details](https://github.com/apache/syncope/security/code-scanning/1103)
##########
core/provisioning-api/src/main/java/org/apache/syncope/core/provisioning/api/pushpull/PullActions.java:
##########
@@ -44,7 +44,7 @@
* @param orgUnit Realm provisioning information
* @return additional attributes to include in the result from the underlying connector
*/
- default Set<String> moreAttrsToGet(ProvisioningProfile<?, ?> profile, OrgUnit orgUnit) {
+ default Set<String> moreAttrsToGet(ProvisioningProfile<?, ?> profile, OrgUnitTO orgUnit) {
Review Comment:
## Useless parameter
The parameter orgUnit is unused.
[Show more details](https://github.com/apache/syncope/security/code-scanning/1091)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@syncope.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [syncope] ilgrosso commented on pull request #360: Consolidating Provision, Mapping and Items into single JSON column
Posted by GitBox <gi...@apache.org>.
ilgrosso commented on PR #360:
URL: https://github.com/apache/syncope/pull/360#issuecomment-1184116165
Very interesting results from some smoke performance tests.
I've been using [ilgrosso/syncopeRestClient](https://github.com/ilgrosso/syncopeRestClient) to create users with propagation against Syncope instances started via `mvn -Pdebug,all` under `fit/core-reference` directory.
| | `master` | this PR |
|--------------|-----------|------------|
| 1000 users | 1145426 ms (~19 minutes) | 189939 ms (~3 minutes) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@syncope.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [syncope] ilgrosso merged pull request #360: [SYNCOPE-1689] Consolidate Provision, Mapping and Items into single JSON column
Posted by GitBox <gi...@apache.org>.
ilgrosso merged PR #360:
URL: https://github.com/apache/syncope/pull/360
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@syncope.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org