You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "BsoBird (via GitHub)" <gi...@apache.org> on 2024/07/12 05:24:28 UTC
[PR] Core:support redis and http lock-manager [iceberg]
BsoBird opened a new pull request, #10688:
URL: https://github.com/apache/iceberg/pull/10688
Currently, the only lockManager implementation in iceberg-core is InMemoryLockManager. This PR extends two LockManager implementations, one based on the Redis distributed locking lockManager implementation, and another based on the Rest-API distributed locking lockManager implementation.lockManager implementation based on the Rest-API.
In general, most users use redisLockManager is sufficient to cope with most of the scenarios, for redis can not meet the user's requirements, we can let the user to provide a RestApi service to achieve this function. I believe that, for a long time, the two lockManager is enough to deal with almost all the requirements of the user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core:support redis and http lock-manager [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #10688:
URL: https://github.com/apache/iceberg/pull/10688#discussion_r1675598848
##########
core/src/test/java/org/apache/iceberg/lock/HttpLockManagerTest.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.lock;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.util.UUID;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+public class HttpLockManagerTest {
Review Comment:
For this test case, I think we only need to verify the basic functionality is enough, do not need to verify the concurrent operation of the case, because all the behaviours are implemented by the user-provided rest service.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core:support redis and http lock-manager [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #10688:
URL: https://github.com/apache/iceberg/pull/10688#discussion_r1675607925
##########
core/src/main/java/org/apache/iceberg/lock/ServerSideHttpLockManager.java:
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.lock;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.Map;
+import org.apache.http.HttpResponse;
+import org.apache.http.StatusLine;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.protocol.RequestAcceptEncoding;
+import org.apache.http.client.protocol.ResponseContentEncoding;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.http.impl.client.DefaultHttpClient;
+import org.apache.http.params.BasicHttpParams;
+import org.apache.http.params.CoreConnectionPNames;
+import org.apache.http.params.HttpParams;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.LockManagers;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The user is required to provide an API interface with distributed locking functionality as
+ * agreed.
+ */
+public class ServerSideHttpLockManager extends LockManagers.BaseLockManager {
Review Comment:
In this implementation, we only use a very simple GET request and do not consider anything related to authentication.This is because I don't see the need to protect this distributed locking service.After all, it only does locking/unlocking. I don't think it's too difficult for users to provide a rest-api service that meets the requirements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core:support redis and http lock-manager [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on PR #10688:
URL: https://github.com/apache/iceberg/pull/10688#issuecomment-2226738178
@danielcweeks @rdblue
I know what you mean, Sir.
1.About abandoning the catalog implementation that relies on lockManager. I think this is too radical. I agree that it would be better to prioritise moving towards catalogues that do not rely on lockManager, but it would be better to balance this with the use of lockManager (which in itself is not a difficult task).For example, a large number of users still use GlueCatalog, because they want to access both iceberg and non-iceberg tables in Glue.If we migrate the iceberg catalogues to a different store such as database, users will not be able to view the iceberg tables in glue, and they may not be able to accept this.Because, if they have a choice, they should store the iceberg-catalog in a system that doesn't rely on lockManager in the first place.
2.You might think that users could solve this problem by implementing rest-catalog. However, many users do not have the ability to implement rest-catalog.Also, rest-catalog is still evolving, and many of the specifications have not yet been finalised.
3.Regarding the fact that all clients need to be involved in the locking in order for it to work. I think this is a problem that users need to solve themselves. ICEBERG is just a toolbox and it is up to the user to use it well.
4.Regarding the new lockManager, I also think that we should not continue to add more third-party dependencies to the core, so adding redisson is just an experiment.If someone refuses, I'll remove it immediately. Only the httpLockManager remains. This way, we don't add any additional dependencies to iceberg-core.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core:support http lock-manager [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird closed pull request #10688: Core:support http lock-manager
URL: https://github.com/apache/iceberg/pull/10688
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core:support http lock-manager [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #10688:
URL: https://github.com/apache/iceberg/pull/10688#discussion_r1675607925
##########
core/src/main/java/org/apache/iceberg/lock/ServerSideHttpLockManager.java:
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.lock;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.Map;
+import org.apache.http.HttpResponse;
+import org.apache.http.StatusLine;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.protocol.RequestAcceptEncoding;
+import org.apache.http.client.protocol.ResponseContentEncoding;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.http.impl.client.DefaultHttpClient;
+import org.apache.http.params.BasicHttpParams;
+import org.apache.http.params.CoreConnectionPNames;
+import org.apache.http.params.HttpParams;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.LockManagers;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The user is required to provide an API interface with distributed locking functionality as
+ * agreed.
+ */
+public class ServerSideHttpLockManager extends LockManagers.BaseLockManager {
Review Comment:
In this implementation, we only use a very simple GET request and do not consider anything related to authentication.This is because I don't see the need to protect this distributed locking service.After all, it only does locking/unlocking. I don't think it's too difficult for users to provide a rest-api service that meets the requirements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core:support redis and http lock-manager [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #10688:
URL: https://github.com/apache/iceberg/pull/10688#discussion_r1675609730
##########
core/src/main/java/org/apache/iceberg/lock/RedissonLockManager.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.lock;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import org.apache.iceberg.exceptions.BadRequestException;
+import org.apache.iceberg.relocated.com.google.common.base.Strings;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.LockManagers;
+import org.apache.iceberg.util.Tasks;
+import org.redisson.Redisson;
+import org.redisson.api.RLock;
+import org.redisson.api.RedissonClient;
+import org.redisson.config.Config;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class RedissonLockManager extends LockManagers.BaseLockManager {
+
+ public static final String REDISSON_LOCK_CONF = "lock.redisson.conf.yml.base64";
+ public static final String REDISSON_LOCK_SINGLE_NODE_URL = "lock.redisson.conf.url";
+ public static final String REDISSON_LOCK_KEY_PREFIX = "lock.redisson.key.prefix";
+ public static final String REDISSON_LOCK_RETRY_MAX_TIMES = "lock.redisson.retry.max-times";
+ public static final String REDISSON_LOCK_RETRY_MAX_TIMES_DEFAULT = "3";
+ private static final Logger LOG = LoggerFactory.getLogger(RedissonLockManager.class);
+
+ private final Map<String, RLock> lockCache = Maps.newHashMap();
+ private RedissonClient redissonClient;
+ private String prefix = null;
+ private int maxRetryTimes = 10;
+
+ public RedissonLockManager() {}
+
+ public RedissonLockManager(String redisUrl) {
+ try {
+ init(redisUrl, null);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public RedissonLockManager(String redisUrl, String confYmlBase64) {
+ try {
+ init(redisUrl, confYmlBase64);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private synchronized void init(String redissonUrl, String redissonConfYmlBase64)
+ throws IOException {
+ if (redissonClient != null) {
+ return;
+ }
+ if (!Strings.isNullOrEmpty(redissonConfYmlBase64)) {
+ String ymlConfig =
+ new String(Base64.getDecoder().decode(redissonConfYmlBase64), StandardCharsets.UTF_8);
+ Config config = Config.fromYAML(ymlConfig);
Review Comment:
Base64 encrypted strings can be quite large, but I think we can ignore that for now.
Because even if there is a problem, I don't think it's too hard for users to redeploy a single node redis service for iceberg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core:support redis and http lock-manager [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #10688:
URL: https://github.com/apache/iceberg/pull/10688#discussion_r1676690715
##########
build.gradle:
##########
@@ -358,6 +358,7 @@ project(':iceberg-core') {
implementation libs.jackson.databind
implementation libs.caffeine
implementation libs.roaringbitmap
+ implementation libs.redission
Review Comment:
@rdblue So, we only keep ServerSideHttpLockManager and delete RedissonLockManager, is it OK?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core:support redis and http lock-manager [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #10688:
URL: https://github.com/apache/iceberg/pull/10688#discussion_r1675602524
##########
core/src/test/java/org/apache/iceberg/lock/RedisLockManagerTest.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.lock;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import com.github.fppt.jedismock.RedisServer;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+public class RedisLockManagerTest {
+ private RedisServer redisServer = null;
+
+ @BeforeEach
+ public void before() throws IOException {
+ redisServer = new RedisServer(6379);
Review Comment:
We have introduced an embedded redis service, and I can remove the dependencies associated with this embedded redis service if needed.And copy the RedisServer code here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core:support redis and http lock-manager [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #10688:
URL: https://github.com/apache/iceberg/pull/10688#discussion_r1675655377
##########
core/src/main/java/org/apache/iceberg/lock/ServerSideHttpLockManager.java:
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.lock;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.Map;
+import org.apache.http.HttpResponse;
+import org.apache.http.StatusLine;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.protocol.RequestAcceptEncoding;
+import org.apache.http.client.protocol.ResponseContentEncoding;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.http.impl.client.DefaultHttpClient;
+import org.apache.http.params.BasicHttpParams;
+import org.apache.http.params.CoreConnectionPNames;
+import org.apache.http.params.HttpParams;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.LockManagers;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The user is required to provide an API interface with distributed locking functionality as
+ * agreed.
+ */
+public class ServerSideHttpLockManager extends LockManagers.BaseLockManager {
+ private static final Logger LOG = LoggerFactory.getLogger(ServerSideHttpLockManager.class);
+ private String httpUrl = null;
+ private static final String OPERATOR = "operator";
+ private static final String LOCK = "lock";
+ private static final String UNLOCK = "unlock";
+ private static final String ENTITY_ID = "entityId";
+ private static final String OWNER_ID = "ownerId";
+ private static final int REQUEST_SUCCESS = 200;
+ public static final String REQUEST_URL = "lock.http.conf.request.url";
+ private HttpClient httpClient = null;
+
+ public ServerSideHttpLockManager() {}
+
+ public ServerSideHttpLockManager(String requestUrl) {
+ initialize(ImmutableMap.of(REQUEST_URL, requestUrl));
+ }
+
+ @Override
+ public boolean acquire(String entityId, String ownerId) {
+ return process(entityId, ownerId, LOCK);
+ }
+
+ @Override
+ public void initialize(Map<String, String> properties) {
+ super.initialize(properties);
+ init(properties.getOrDefault(REQUEST_URL, null));
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ if (httpClient != null) {
+ httpClient.getConnectionManager().shutdown();
+ }
+ }
+
+ private synchronized void init(String requestUrl) {
+ if (requestUrl == null) {
+ String msg = String.format("[%s] must be set.", REQUEST_URL);
+ throw new IllegalArgumentException(msg);
+ }
+ if (this.httpUrl == null) {
+ this.httpUrl = requestUrl;
+ }
+ if (this.httpClient == null) {
+ DefaultHttpClient defaultHttpClient = new DefaultHttpClient();
+ defaultHttpClient.addRequestInterceptor(new RequestAcceptEncoding());
+ defaultHttpClient.addResponseInterceptor(new ResponseContentEncoding());
+ defaultHttpClient
+ .getParams()
+ .setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, acquireTimeoutMs());
+ defaultHttpClient
+ .getParams()
+ .setParameter(CoreConnectionPNames.SO_TIMEOUT, heartbeatTimeoutMs());
+ this.httpClient = new DefaultHttpClient();
+ }
+ }
+
+ private String encode(String entity) {
+ if (entity == null) {
+ return null;
+ }
+ return new String(
+ Base64.getUrlEncoder().encode(entity.getBytes(StandardCharsets.UTF_8)),
+ StandardCharsets.UTF_8);
+ }
+
+ private boolean process(String entityId, String ownerId, String operator) {
+ try {
+ HttpGet lockRequest = new HttpGet();
+ lockRequest.addHeader("Content-Type", "application/json");
+ HttpParams requestParams = new BasicHttpParams();
+ requestParams.setParameter(OWNER_ID, encode(ownerId));
+ requestParams.setParameter(ENTITY_ID, encode(entityId));
+ requestParams.setParameter(OPERATOR, operator);
+ String requestUrl =
+ new URIBuilder(httpUrl)
+ .setParameter(OWNER_ID, encode(ownerId))
+ .setParameter(ENTITY_ID, encode(entityId))
+ .setParameter(OPERATOR, operator)
+ .build()
+ .toString();
+ lockRequest.setParams(requestParams);
+ lockRequest.setURI(new URI(requestUrl));
+ HttpResponse response = httpClient.execute(lockRequest);
+ StatusLine stateLine = response.getStatusLine();
+ int statsCode = stateLine.getStatusCode();
+ lockRequest.abort();
+ return REQUEST_SUCCESS == statsCode;
Review Comment:
To be honest, I don't think we need to be concerned with the exact cause of the failure.Also, since we don't know what data the user intends to return, parsing the data returned by the user introduces a significant amount of logical complexity. Instead of doing this, it's easier to just detect the HTTP status code. It's not difficult for users to implement the rest-api service in the way we want.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core:support redis and http lock-manager [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on PR #10688:
URL: https://github.com/apache/iceberg/pull/10688#issuecomment-2225235649
tips:
According to the original plan, I originally intended to extend a zookeeper-based lockManager. However, I found that the whole iceberg project's dependency on zookeeper is quite confusing, so I gave up extending the zookeeper-based lockManager.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core:support http lock-manager [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #10688:
URL: https://github.com/apache/iceberg/pull/10688#discussion_r1676850504
##########
core/src/main/java/org/apache/iceberg/lock/ServerSideHttpLockManager.java:
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.lock;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.Map;
+import org.apache.http.HttpResponse;
+import org.apache.http.StatusLine;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.protocol.RequestAcceptEncoding;
+import org.apache.http.client.protocol.ResponseContentEncoding;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.http.impl.client.DefaultHttpClient;
+import org.apache.http.params.CoreConnectionPNames;
+import org.apache.http.util.EntityUtils;
+import org.apache.iceberg.common.DynConstructors;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.LockManagers;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The user is required to provide an API interface with distributed locking functionality as
+ * agreed.
+ */
+public class ServerSideHttpLockManager extends LockManagers.BaseLockManager {
+ private static final Logger LOG = LoggerFactory.getLogger(ServerSideHttpLockManager.class);
+ private String httpUrl = null;
+ private static final String OPERATOR = "operator";
+ private static final String LOCK = "lock";
+ private static final String UNLOCK = "unlock";
+ private static final String ENTITY_ID = "entityId";
+ private static final String OWNER_ID = "ownerId";
+ private static final int REQUEST_SUCCESS = 200;
+ public static final String REQUEST_URL = "lock.http.conf.request.url";
+ public static final String REQUEST_AUTH = "lock.http.conf.request.auth.impl";
+ private HttpClient httpClient = null;
+ private HttpAuthentication httpAuthentication = null;
Review Comment:
Users can add custom auth implementations when initiating tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core:support redis and http lock-manager [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #10688:
URL: https://github.com/apache/iceberg/pull/10688#discussion_r1675598848
##########
core/src/test/java/org/apache/iceberg/lock/HttpLockManagerTest.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.lock;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.util.UUID;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+public class HttpLockManagerTest {
Review Comment:
For this test case, I think we only need to verify the basic functionality is enough, do not need to verify the concurrent operation of the case, because all the behaviours are implemented by the user-provided rest-api service.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core:support redis and http lock-manager [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #10688:
URL: https://github.com/apache/iceberg/pull/10688#discussion_r1676303612
##########
build.gradle:
##########
@@ -358,6 +358,7 @@ project(':iceberg-core') {
implementation libs.jackson.databind
implementation libs.caffeine
implementation libs.roaringbitmap
+ implementation libs.redission
Review Comment:
I don't think that we should move forward with a PR that adds Redis to the classpath. This changes the classpath that will be included in the runtime bundles, so at a minimum, we need to either make it optional or update all of the LICENSE and NOTICE files to note the inclusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core:support redis and http lock-manager [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #10688:
URL: https://github.com/apache/iceberg/pull/10688#discussion_r1675648520
##########
core/src/test/java/org/apache/iceberg/lock/TestHttpServer.java:
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.lock;
+
+import java.io.BufferedReader;
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.PrintStream;
+import java.net.ServerSocket;
+import java.net.Socket;
+import java.net.URI;
+import java.net.URLDecoder;
+import java.util.Map;
+import java.util.Objects;
+import java.util.StringTokenizer;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TestHttpServer implements Runnable, Closeable {
Review Comment:
This is a rest-api-service for testing. We don't need to be very concerned about its logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core:support redis and http lock-manager [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on PR #10688:
URL: https://github.com/apache/iceberg/pull/10688#issuecomment-2226150038
I don't think we should be adding new `LockManager` implementations. The general discussion has been that we want to deprecate catalog implementations that rely on external locking, so adding new functionality is counter to that goal. This approach has some real fundamental issues, the most important being that all clients need to participate in the locking for it to be effective.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core:support redis and http lock-manager [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #10688:
URL: https://github.com/apache/iceberg/pull/10688#discussion_r1676751334
##########
build.gradle:
##########
@@ -358,6 +358,7 @@ project(':iceberg-core') {
implementation libs.jackson.databind
implementation libs.caffeine
implementation libs.roaringbitmap
+ implementation libs.redission
Review Comment:
@rdblue Sir, I deleted all the Redisson-related parts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core:support http lock-manager [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on PR #10688:
URL: https://github.com/apache/iceberg/pull/10688#issuecomment-2226845799
@snazy
Hello.I'm so glad you could get back to me.
I'm just saying out loud the first thing that comes to mind:
1. Is everything done using rest-catalog? In many cases, users have their own server rooms, they are not using cloud products, and they do not intend to expose iceberg directly to untrusted users. In a network-isolated environment, there seems to be no need to introduce a separate rest-catalog to add complexity.
2. For more complex access environments, rest-catalog is effective.I agree.
3. Currently, the use of lockManager is tied to the implementation of a certain catalog.Similar to aws-iceberg, we don't actually force the user to use lockManager. The user knows what he's doing when he chooses a catalog. Basically, catalog's author has solved the concurrency problem and the race condition in the code of the catalog that uses lockManager. After all, he wouldn't need to use lockManager if he didn't solve these problems.
4. Regarding the security of lockManager. This is something that should actually be considered within some kind of extension module.For example, iceberg-aws provides some implementations of lockManager, and the security of such lockManager must be considered by them.iceberg-core shouldn't care about such issues. After all, safety standards are not uniform.The details are varied.
From what I've seen so far, the opinion of many people in the community is that the lock-manager should be extended and maintained by users if they want it, just like the aws-iceberg module. In iceberg-core, the implementation of lock-manager should only stop at InMemLockManager. But isn't that too little? Wouldn't it be nice to simply extend something like http-lock-manager for users?After all, it's small and doesn't introduce any other dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core:support redis and http lock-manager [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #10688:
URL: https://github.com/apache/iceberg/pull/10688#discussion_r1675609730
##########
core/src/main/java/org/apache/iceberg/lock/RedissonLockManager.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.lock;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import org.apache.iceberg.exceptions.BadRequestException;
+import org.apache.iceberg.relocated.com.google.common.base.Strings;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.LockManagers;
+import org.apache.iceberg.util.Tasks;
+import org.redisson.Redisson;
+import org.redisson.api.RLock;
+import org.redisson.api.RedissonClient;
+import org.redisson.config.Config;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class RedissonLockManager extends LockManagers.BaseLockManager {
+
+ public static final String REDISSON_LOCK_CONF = "lock.redisson.conf.yml.base64";
+ public static final String REDISSON_LOCK_SINGLE_NODE_URL = "lock.redisson.conf.url";
+ public static final String REDISSON_LOCK_KEY_PREFIX = "lock.redisson.key.prefix";
+ public static final String REDISSON_LOCK_RETRY_MAX_TIMES = "lock.redisson.retry.max-times";
+ public static final String REDISSON_LOCK_RETRY_MAX_TIMES_DEFAULT = "3";
+ private static final Logger LOG = LoggerFactory.getLogger(RedissonLockManager.class);
+
+ private final Map<String, RLock> lockCache = Maps.newHashMap();
+ private RedissonClient redissonClient;
+ private String prefix = null;
+ private int maxRetryTimes = 10;
+
+ public RedissonLockManager() {}
+
+ public RedissonLockManager(String redisUrl) {
+ try {
+ init(redisUrl, null);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public RedissonLockManager(String redisUrl, String confYmlBase64) {
+ try {
+ init(redisUrl, confYmlBase64);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private synchronized void init(String redissonUrl, String redissonConfYmlBase64)
+ throws IOException {
+ if (redissonClient != null) {
+ return;
+ }
+ if (!Strings.isNullOrEmpty(redissonConfYmlBase64)) {
+ String ymlConfig =
+ new String(Base64.getDecoder().decode(redissonConfYmlBase64), StandardCharsets.UTF_8);
+ Config config = Config.fromYAML(ymlConfig);
Review Comment:
Base64 encrypted strings can be quite large, but I think we can ignore that for now.
Because even if there is a problem, I don't think it's too hard for users to redeploy a redis service for iceberg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core:support redis and http lock-manager [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #10688:
URL: https://github.com/apache/iceberg/pull/10688#discussion_r1675609730
##########
core/src/main/java/org/apache/iceberg/lock/RedissonLockManager.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.lock;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import org.apache.iceberg.exceptions.BadRequestException;
+import org.apache.iceberg.relocated.com.google.common.base.Strings;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.LockManagers;
+import org.apache.iceberg.util.Tasks;
+import org.redisson.Redisson;
+import org.redisson.api.RLock;
+import org.redisson.api.RedissonClient;
+import org.redisson.config.Config;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class RedissonLockManager extends LockManagers.BaseLockManager {
+
+ public static final String REDISSON_LOCK_CONF = "lock.redisson.conf.yml.base64";
+ public static final String REDISSON_LOCK_SINGLE_NODE_URL = "lock.redisson.conf.url";
+ public static final String REDISSON_LOCK_KEY_PREFIX = "lock.redisson.key.prefix";
+ public static final String REDISSON_LOCK_RETRY_MAX_TIMES = "lock.redisson.retry.max-times";
+ public static final String REDISSON_LOCK_RETRY_MAX_TIMES_DEFAULT = "3";
+ private static final Logger LOG = LoggerFactory.getLogger(RedissonLockManager.class);
+
+ private final Map<String, RLock> lockCache = Maps.newHashMap();
+ private RedissonClient redissonClient;
+ private String prefix = null;
+ private int maxRetryTimes = 10;
+
+ public RedissonLockManager() {}
+
+ public RedissonLockManager(String redisUrl) {
+ try {
+ init(redisUrl, null);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public RedissonLockManager(String redisUrl, String confYmlBase64) {
+ try {
+ init(redisUrl, confYmlBase64);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private synchronized void init(String redissonUrl, String redissonConfYmlBase64)
+ throws IOException {
+ if (redissonClient != null) {
+ return;
+ }
+ if (!Strings.isNullOrEmpty(redissonConfYmlBase64)) {
+ String ymlConfig =
+ new String(Base64.getDecoder().decode(redissonConfYmlBase64), StandardCharsets.UTF_8);
+ Config config = Config.fromYAML(ymlConfig);
Review Comment:
Base64 encrypted strings can be quite large, but I think we can ignore that for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core:support http lock-manager [iceberg]
Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy commented on PR #10688:
URL: https://github.com/apache/iceberg/pull/10688#issuecomment-2226834838
It seems there are lot of individual points that should be discussed. @BsoBird do you mind adding a point to the "Discussions" section on the [agenda for the next community sync on July 31st](https://docs.google.com/document/d/1YuGhUdukLP5gGiqCbk0A5_Wifqe2CZWgOd3TbhY3UQg/edit)?
Let me comment on some of the points. TL;DR I think it's not an easy problem to solve.
> balance this with the use of lockManager (which in itself is not a difficult task).
Locking is one of the most difficult problems in distributed computing. (And actually in non-distributed computing, too.) Just very few things that come to my mind (the real list is much longer):
* Evil users lock all your tables (-> authorization)
* Retry/back-off -> piling up waiting requests -> threading issues
* Crashed clients leave stuff locked
* "all the things" about race conditions
> ... a large number of users use GlueCatalog ... want to access both iceberg and non-iceberg tables in Glue.
That's IMO a fair point.
> 2.You might think that users could solve this problem by implementing rest-catalog. However, many users do not have the ability to implement rest-catalog.Also, rest-catalog is still evolving, and many of the specifications have not yet been finalised.
I suspect, this wouldn't be an issue, if Glue would have a rest-catalog?
> 3.Regarding the fact that all clients need to be involved in the locking in order for it to work. I think this is a problem that users need to solve themselves.
Similar to my statement above, users should not be forced into implementing/providing a proper distributed locking mechanism.
Also isolation levels play a role here. Some use cases are probably fine with "dirty reads", some with "read committed" and others require full "serializeable" guarantees.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org