You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@geode.apache.org by GitBox <gi...@apache.org> on 2022/03/01 15:09:00 UTC
[GitHub] [geode] jdeppe-pivotal opened a new pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
jdeppe-pivotal opened a new pull request #7408:
URL: https://github.com/apache/geode/pull/7408
This commit adds a simple eventing framework to be used by blocking
commands as well as keyspace event notification (still to be
implemented). The main components are:
- EventListener: an interface to be immplemented by anything wishing to
receive events. Currently only implemented for blocking commands in
the form of BlockingCommandListener.
- EventDistributor: the component to which listeners are registered and
where events are received and distributed. A single EventDistributor
exists in the system and is associated with each
ExecutionHandlerContext.
- Event: not implemented as a separate class but logically consists of
the command (RedisCommandType) and key (RedisKey).
<!-- Thank you for submitting a contribution to Apache Geode. -->
<!-- In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
-->
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?
- [ ] Has your PR been rebased against the latest commit within the target branch (typically `develop`)?
- [ ] Is your initial contribution a single, squashed commit?
- [ ] Does `gradlew build` run cleanly?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
<!-- Note:
Please ensure that once the PR is submitted, check Concourse for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to dev@geode.apache.org.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r824371488
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/BlockingCommandListener.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListener implements EventListener {
+
+ private final ExecutionHandlerContext context;
+ private final RedisCommandType command;
+ private final List<RedisKey> keys;
+ private final byte[][] commandOptions;
+ private final long timeout;
+ private final long timeAdded;
+ private Runnable cleanupTask;
+
+ /**
+ * Constructor to create an instance of a BlockingCommandListener in response to a blocking
+ * command. When receiving a relevant event, blocking commands simply resubmit the command
+ * into the Netty pipeline.
+ *
+ * @param context the associated ExecutionHandlerContext
+ * @param command the blocking command
+ * @param timeout the timeout for the command to block in milliseconds
+ * @param keys the list of keys the command is interested in
+ * @param commandOptions any additional options (other than the keys and timeout) required to
+ * resubmit the command.
+ */
+ public BlockingCommandListener(ExecutionHandlerContext context, RedisCommandType command,
+ long timeout, List<RedisKey> keys, byte[]... commandOptions) {
+ this.context = context;
+ this.command = command;
+ this.timeout = timeout;
+ this.keys = Collections.unmodifiableList(keys);
+ this.commandOptions = commandOptions;
+ timeAdded = System.currentTimeMillis();
+ }
+
+ @Override
+ public List<RedisKey> keys() {
+ return keys;
+ }
+
+ @Override
+ public EventResponse process(RedisCommandType commandType, RedisKey key) {
+ if (!keys.contains(key)) {
+ return EventResponse.CONTINUE;
+ }
+
+ resubmitCommand();
+ return EventResponse.REMOVE_AND_STOP;
+ }
+
+ @Override
+ public void resubmitCommand() {
+ List<byte[]> byteArgs = new ArrayList<>(keys.size() + 1);
+ byteArgs.add(command.name().getBytes());
+
+ keys.forEach(x -> byteArgs.add(x.toBytes()));
+ byteArgs.addAll(Arrays.asList(commandOptions));
+
+ // Recalculate the timeout since we've already been waiting
+ long adjustedTimeout = timeout;
+ if (adjustedTimeout > 0) {
+ adjustedTimeout = (adjustedTimeout - (System.currentTimeMillis() - timeAdded)) / 1000;
+ adjustedTimeout = Math.max(1, adjustedTimeout);
Review comment:
This should be fixed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r824374651
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/commands/executor/list/BLPopExecutor.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.geode.redis.internal.RedisConstants;
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.executor.CommandExecutor;
+import org.apache.geode.redis.internal.commands.executor.RedisResponse;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.data.RedisList;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BLPopExecutor implements CommandExecutor {
+
+ @Override
+ public RedisResponse executeCommand(Command command, ExecutionHandlerContext context) {
+ List<byte[]> arguments = command.getCommandArguments();
+ int keyCount = arguments.size() - 1;
+ double timeoutMillis;
+ try {
+ timeoutMillis = Coder.bytesToDouble(arguments.get(keyCount)) * 1000;
Review comment:
I think we're OK and, in fact probably need to keep it as a double since the command can be resubmitted and we wouldn't want to reduce the granularity there to only seconds since it would lose precision. If older clients are just using integers our implementation is still going to work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r820881217
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventDistributor.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Timer;
+import java.util.TimerTask;
+
+import org.apache.logging.log4j.Logger;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.cache.partition.PartitionListenerAdapter;
+import org.apache.geode.logging.internal.log4j.api.LogService;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributor extends PartitionListenerAdapter {
+
+ private static final Logger logger = LogService.getLogger();
+ private final Map<RedisKey, List<EventListener>> listeners = new HashMap<>();
+ private final Map<EventListener, TimerTask> timerTasks = new HashMap<>();
Review comment:
Take a look at what I did - I just passed a callback to the listener that will allow it to be cancelled. I'm not sure if that creates additional 'synthetics' that are an unnecessary overhead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r820879800
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventDistributor.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Timer;
+import java.util.TimerTask;
+
+import org.apache.logging.log4j.Logger;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.cache.partition.PartitionListenerAdapter;
+import org.apache.geode.logging.internal.log4j.api.LogService;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributor extends PartitionListenerAdapter {
+
+ private static final Logger logger = LogService.getLogger();
+ private final Map<RedisKey, List<EventListener>> listeners = new HashMap<>();
+ private final Map<EventListener, TimerTask> timerTasks = new HashMap<>();
+ private final Timer timer = new Timer("GeodeForRedisEventTimer", true);
+ private int keysRegistered = 0;
+
+ public synchronized void registerListener(EventListener listener) {
+ for (RedisKey key : listener.keys()) {
+ listeners.computeIfAbsent(key, k -> new ArrayList<>()).add(listener);
+ }
+
+ if (listener.getTimeout() != 0) {
+ scheduleTimeout(listener, listener.getTimeout());
+ }
+
+ keysRegistered += listener.keys().size();
+ }
+
+ public synchronized void fireEvent(RedisCommandType command, RedisKey key) {
+ if (listeners.isEmpty()) {
Review comment:
I've removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r820881930
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventListener.java
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.List;
+
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+/**
+ * Interface intended to be implemented in order to receive Redis events. Specifically this would be
+ * keyspace events or blocking commands. EventListeners are registered with the
+ * {@link EventDistributor}.
+ */
+public interface EventListener {
+
+ /**
+ * Receive and process an event. This method should execute very quickly. The return value
+ * determines additional process steps for the given event.
+ *
+ * @param commandType the command triggering the event
+ * @param key the key triggering the event
+ * @return response determining subsequent processing steps
+ */
+ EventResponse process(RedisCommandType commandType, RedisKey key);
+
+ /**
+ * Return the list of keys this listener is interested in.
+ */
+ List<RedisKey> keys();
+
+ /**
+ * Method to resubmit a command if appropriate. This is only relevant for listeners that process
+ * events for blocking commands. Listeners that handle keyspace event notification will not use
+ * this.
+ */
+ default void resubmitCommand() {};
+
+ /**
+ * Retrieve the timeout for this listener. The default is no timeout.
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r820886832
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventDistributor.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Timer;
+import java.util.TimerTask;
+
+import org.apache.logging.log4j.Logger;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.cache.partition.PartitionListenerAdapter;
+import org.apache.geode.logging.internal.log4j.api.LogService;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributor extends PartitionListenerAdapter {
+
+ private static final Logger logger = LogService.getLogger();
+ private final Map<RedisKey, List<EventListener>> listeners = new HashMap<>();
+ private final Map<EventListener, TimerTask> timerTasks = new HashMap<>();
+ private final Timer timer = new Timer("GeodeForRedisEventTimer", true);
+ private int keysRegistered = 0;
+
+ public synchronized void registerListener(EventListener listener) {
+ for (RedisKey key : listener.keys()) {
+ listeners.computeIfAbsent(key, k -> new ArrayList<>()).add(listener);
+ }
+
+ if (listener.getTimeout() != 0) {
+ scheduleTimeout(listener, listener.getTimeout());
+ }
+
+ keysRegistered += listener.keys().size();
+ }
+
+ public synchronized void fireEvent(RedisCommandType command, RedisKey key) {
Review comment:
Synchronization still needs to happen on both `registerListener` as well as `removeListener`. Since these methods act on two discrete components (adding an entry to a structure, setting up for a timeout) they need to fully complete otherwise there is always a race that produces NullPointerExceptions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r825626951
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventDistributor.java
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.cache.partition.PartitionListenerAdapter;
+import org.apache.geode.logging.internal.executors.LoggingThreadFactory;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributor extends PartitionListenerAdapter {
+
+ private final Map<RedisKey, Queue<EventListener>> listeners = new ConcurrentHashMap<>();
+
+ private final ScheduledThreadPoolExecutor timerExecutor =
+ new ScheduledThreadPoolExecutor(1,
+ new LoggingThreadFactory("GeodeForRedisEventTimer-", true));
+
+ private int keysRegistered = 0;
+
+ public EventDistributor() {
+ timerExecutor.setRemoveOnCancelPolicy(true);
+ }
+
+ public synchronized void registerListener(EventListener listener) {
+ for (RedisKey key : listener.keys()) {
+ listeners.computeIfAbsent(key, k -> new LinkedBlockingQueue<>()).add(listener);
+ }
+
+ if (listener.getTimeout() != 0) {
+ scheduleTimeout(listener, listener.getTimeout());
+ }
+
+ keysRegistered += listener.keys().size();
+ }
+
+ public void fireEvent(RedisCommandType command, RedisKey key) {
+ Queue<EventListener> listenerList = listeners.get(key);
+ if (listenerList == null) {
+ return;
+ }
+
+ for (EventListener listener : listenerList) {
+ if (listener.process(command, key) == EventResponse.REMOVE_AND_STOP) {
+ removeListener(listener);
+ break;
+ }
+ }
+ }
+
+ /**
+ * The total number of keys registered by all listeners (includes duplicates).
+ */
+ @VisibleForTesting
+ public int size() {
+ return keysRegistered;
+ }
+
+ @Override
+ public void afterBucketRemoved(int bucketId, Iterable<?> keys) {
+ Set<EventListener> resubmittingList = new HashSet<>();
+ for (Map.Entry<RedisKey, Queue<EventListener>> entry : listeners.entrySet()) {
+ if (entry.getKey().getBucketId() == bucketId) {
+ resubmittingList.addAll(entry.getValue());
+ }
+ }
+
+ resubmittingList.forEach(x -> {
+ x.resubmitCommand();
+ removeListener(x);
+ });
+ }
+
+ private synchronized void removeListener(EventListener listener) {
+ boolean listenerRemoved = false;
+ for (RedisKey key : listener.keys()) {
+ Queue<EventListener> listenerList = listeners.get(key);
+ if (listenerList == null) {
+ continue;
+ }
+ listenerRemoved |= listenerList.remove(listener);
+ if (listenerList.isEmpty()) {
Review comment:
I did keep the synchronization. Perhaps I'm mistaken, but I don't think these blocking commands are going to have any hot paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r825628870
##########
File path: geode-for-redis/src/integrationTest/java/org/apache/geode/redis/internal/commands/executor/list/AbstractBLPopIntegrationTest.java
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.BIND_ADDRESS;
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.REDIS_CLIENT_TIMEOUT;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.HostAndPort;
+import redis.clients.jedis.JedisCluster;
+import redis.clients.jedis.Protocol;
+
+import org.apache.geode.redis.RedisIntegrationTest;
+import org.apache.geode.redis.internal.RedisConstants;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.junit.rules.ExecutorServiceRule;
+
+public abstract class AbstractBLPopIntegrationTest implements RedisIntegrationTest {
+ private static final String KEY = "key";
+
+ protected JedisCluster jedis;
+
+ public abstract void awaitEventDistributorSize(int size) throws Exception;
+
+ @ClassRule
+ public static ExecutorServiceRule executor = new ExecutorServiceRule();
+
+ @Before
+ public void setUp() {
+ jedis = new JedisCluster(new HostAndPort(BIND_ADDRESS, getPort()), REDIS_CLIENT_TIMEOUT);
+ }
+
+ @After
+ public void tearDown() {
+ flushAll();
+ jedis.close();
+ }
+
+ @Test
+ public void testInvalidArguments_throwErrors() {
+ assertThatThrownBy(() -> jedis.sendCommand("key", Protocol.Command.BLPOP))
+ .hasMessageContaining("ERR wrong number of arguments for 'blpop' command");
+ assertThatThrownBy(() -> jedis.sendCommand("key1", Protocol.Command.BLPOP, "key"))
+ .hasMessageContaining("ERR wrong number of arguments for 'blpop' command");
+ }
+
+ @Test
+ public void testInvalidTimeout_throwsError() {
+ assertThatThrownBy(() -> jedis.sendCommand("key1", Protocol.Command.BLPOP, "key1",
+ "0.A"))
+ .hasMessageContaining(RedisConstants.ERROR_TIMEOUT_INVALID);
+ }
+
+ @Test
+ public void testKeysInDifferentSlots_throwsError() {
+ assertThatThrownBy(() -> jedis.sendCommand("key1", Protocol.Command.BLPOP, "key1",
+ "key2", "0"))
+ .hasMessageContaining(RedisConstants.ERROR_WRONG_SLOT);
+ }
+
Review comment:
Added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r820882349
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventDistributor.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Timer;
+import java.util.TimerTask;
+
+import org.apache.logging.log4j.Logger;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.cache.partition.PartitionListenerAdapter;
+import org.apache.geode.logging.internal.log4j.api.LogService;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributor extends PartitionListenerAdapter {
+
+ private static final Logger logger = LogService.getLogger();
+ private final Map<RedisKey, List<EventListener>> listeners = new HashMap<>();
+ private final Map<EventListener, TimerTask> timerTasks = new HashMap<>();
+ private final Timer timer = new Timer("GeodeForRedisEventTimer", true);
+ private int keysRegistered = 0;
+
+ public synchronized void registerListener(EventListener listener) {
+ for (RedisKey key : listener.keys()) {
+ listeners.computeIfAbsent(key, k -> new ArrayList<>()).add(listener);
+ }
+
+ if (listener.getTimeout() != 0) {
+ scheduleTimeout(listener, listener.getTimeout());
+ }
+
+ keysRegistered += listener.keys().size();
+ }
+
+ public synchronized void fireEvent(RedisCommandType command, RedisKey key) {
+ if (listeners.isEmpty()) {
+ return;
+ }
+
+ List<EventListener> listenerList = listeners.get(key);
+ if (listenerList == null) {
+ return;
+ }
+
+ for (EventListener listener : listenerList) {
+ if (listener.process(command, key) == EventResponse.REMOVE_AND_STOP) {
+ removeListener(listener);
+ break;
+ }
+ }
+ }
+
+ /**
+ * The total number of keys registered by all listeners (includes duplicates).
+ */
+ @VisibleForTesting
+ public int size() {
+ return keysRegistered;
+ }
+
+ @Override
+ public synchronized void afterBucketRemoved(int bucketId, Iterable<?> keys) {
+ Set<EventListener> resubmittingList = new HashSet<>();
+ for (Map.Entry<RedisKey, List<EventListener>> entry : listeners.entrySet()) {
+ if (entry.getKey().getBucketId() == bucketId) {
+ resubmittingList.addAll(entry.getValue());
+ }
+ }
+
+ resubmittingList.forEach(x -> {
+ x.resubmitCommand();
+ removeListener(x);
+ });
+ }
+
+ private synchronized void removeListener(EventListener listener) {
+ boolean listenerRemoved = false;
+ for (RedisKey key : listener.keys()) {
+ List<EventListener> listenerList = listeners.get(key);
+ listenerRemoved |= listenerList.remove(listener);
+ if (listenerList.isEmpty()) {
+ listeners.remove(key);
+ }
+ }
+
+ if (listenerRemoved) {
+ keysRegistered -= listener.keys().size();
+ if (listener.getTimeout() > 0) {
+ timerTasks.remove(listener).cancel();
+ }
+ }
+ }
+
+ private void scheduleTimeout(EventListener listener, long timeout) {
+ TimerTask task = new TimerTask() {
Review comment:
I think this is all taken care of now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r828551150
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/BlockingCommandListener.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.commands.executor.RedisResponse;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListener implements EventListener {
+
+ private final ExecutionHandlerContext context;
+ private final Command command;
+ private final List<RedisKey> keys;
+ private final double timeoutSeconds;
+ private final long timeSubmitted;
+ private final AtomicBoolean active = new AtomicBoolean(true);
+
+ /**
+ * Constructor to create an instance of a BlockingCommandListener in response to a blocking
+ * command. When receiving a relevant event, blocking commands simply resubmit the command
+ * into the Netty pipeline.
+ *
+ * @param context the associated ExecutionHandlerContext
+ * @param command the blocking command associated with this listener
+ * @param keys the list of keys the command is interested in
+ * @param timeoutSeconds the timeout for the command to block in seconds
+ */
+ public BlockingCommandListener(ExecutionHandlerContext context, Command command,
+ List<RedisKey> keys, double timeoutSeconds) {
+ this.context = context;
+ this.command = command;
+ this.timeoutSeconds = timeoutSeconds;
+ this.keys = Collections.unmodifiableList(keys);
+ timeSubmitted = System.nanoTime();
+ }
+
+ @Override
+ public List<RedisKey> keys() {
+ return keys;
+ }
+
+ @Override
+ public EventResponse process(RedisCommandType commandType, RedisKey key) {
+ if (!keys.contains(key)) {
+ return EventResponse.CONTINUE;
+ }
+
+ if (active.compareAndSet(true, false)) {
Review comment:
I think it's a good idea. This also ensures that the timeout does not fully process when a listener is resubmitted. Since the scheduled timeouts are not being cancelled we will have some listeners hanging around until their timeouts fire. Do you think that's an issue? I could add a `cleanup()` call to the listener that is executed when the listener is removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r828551150
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/BlockingCommandListener.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.commands.executor.RedisResponse;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListener implements EventListener {
+
+ private final ExecutionHandlerContext context;
+ private final Command command;
+ private final List<RedisKey> keys;
+ private final double timeoutSeconds;
+ private final long timeSubmitted;
+ private final AtomicBoolean active = new AtomicBoolean(true);
+
+ /**
+ * Constructor to create an instance of a BlockingCommandListener in response to a blocking
+ * command. When receiving a relevant event, blocking commands simply resubmit the command
+ * into the Netty pipeline.
+ *
+ * @param context the associated ExecutionHandlerContext
+ * @param command the blocking command associated with this listener
+ * @param keys the list of keys the command is interested in
+ * @param timeoutSeconds the timeout for the command to block in seconds
+ */
+ public BlockingCommandListener(ExecutionHandlerContext context, Command command,
+ List<RedisKey> keys, double timeoutSeconds) {
+ this.context = context;
+ this.command = command;
+ this.timeoutSeconds = timeoutSeconds;
+ this.keys = Collections.unmodifiableList(keys);
+ timeSubmitted = System.nanoTime();
+ }
+
+ @Override
+ public List<RedisKey> keys() {
+ return keys;
+ }
+
+ @Override
+ public EventResponse process(RedisCommandType commandType, RedisKey key) {
+ if (!keys.contains(key)) {
+ return EventResponse.CONTINUE;
+ }
+
+ if (active.compareAndSet(true, false)) {
Review comment:
I think it's a good idea. This also ensures that the timeout does not fully process when a listener is resubmitted. Since the scheduled timeouts are not being cancelled we will have some listeners hanging around until their timeouts fire. Do you think that's an issue? I could add a `cleanup()` call to the listener that is executed when the listener is removed. But it's probably not necessary at this point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r825629126
##########
File path: geode-for-redis/src/integrationTest/java/org/apache/geode/redis/internal/eventing/EventDistributorTest.java
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.stream.Collectors;
+
+import org.junit.Test;
+
+import org.apache.geode.redis.ConcurrentLoopingThreads;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+
+public class EventDistributorTest {
+
+ public static class TestEventListener implements EventListener {
+ private final List<RedisKey> keys;
+ private int fired = 0;
+ private final long timeout;
+
+ public TestEventListener(RedisKey... keys) {
+ this(0, keys);
+ }
+
+ public TestEventListener(long timeout, RedisKey... keys) {
+ this.keys = Arrays.stream(keys).collect(Collectors.toList());
+ this.timeout = timeout;
+ }
+
+ public int getFired() {
+ return fired;
+ }
+
+ @Override
+ public EventResponse process(RedisCommandType commandType, RedisKey key) {
+ fired += 1;
+ return EventResponse.REMOVE_AND_STOP;
+ }
+
+ @Override
+ public List<RedisKey> keys() {
+ return keys;
+ }
+
+ @Override
+ public void resubmitCommand() {}
+
+ @Override
+ public long getTimeout() {
+ return timeout;
+ }
+
+ @Override
+ public void timeout() {}
+
+ @Override
+ public void setCleanupTask(Runnable r) {}
+
+ @Override
+ public void cleanup() {}
+ }
+
+ @Test
+ public void firingEventRemovesListener() {
+ RedisKey keyA = new RedisKey("a".getBytes());
+ RedisKey keyB = new RedisKey("b".getBytes());
+ EventDistributor distributor = new EventDistributor();
+ TestEventListener listener = new TestEventListener(keyA, keyB);
+ distributor.registerListener(listener);
+
+ distributor.fireEvent(null, keyA);
+ assertThat(listener.getFired()).isEqualTo(1);
+ assertThat(distributor.size()).isEqualTo(0);
+ }
+
+ @Test
+ public void firingEventRemovesFirstListener_whenMultipleExist() {
+ RedisKey keyA = new RedisKey("a".getBytes());
+ RedisKey keyB = new RedisKey("b".getBytes());
+ EventDistributor distributor = new EventDistributor();
+ TestEventListener listener1 = new TestEventListener(keyA, keyB);
+ TestEventListener listener2 = new TestEventListener(keyA, keyB);
+ distributor.registerListener(listener1);
+ distributor.registerListener(listener2);
+
+ assertThat(distributor.size()).isEqualTo(4);
+
+ distributor.fireEvent(null, keyA);
+ assertThat(listener1.getFired()).isEqualTo(1);
+ assertThat(listener2.getFired()).isEqualTo(0);
+ assertThat(distributor.size()).isEqualTo(2);
+
+ distributor.fireEvent(null, keyA);
+ assertThat(listener1.getFired()).isEqualTo(1);
+ assertThat(listener2.getFired()).isEqualTo(1);
+ assertThat(distributor.size()).isEqualTo(0);
+ }
+
+ @Test
+ public void listenerIsRemovedAfterTimeout() {
+ RedisKey keyA = new RedisKey("a".getBytes());
+ EventDistributor distributor = new EventDistributor();
+ TestEventListener listener1 = new TestEventListener(1, keyA);
Review comment:
So I removed the timeout value completely. Turned out to be unnecessary because of other refactoring.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r820877567
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/BlockingCommandListener.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListener implements EventListener {
+
+ private final ExecutionHandlerContext context;
+ private final RedisCommandType command;
+ private final List<RedisKey> keys;
+ private final byte[][] commandOptions;
+ private final long timeout;
+
+ /**
+ * Constructor to create an instance of a BlockingCommandListener in response to a blocking
+ * command. When receiving a relevant event, blocking commands simply resubmit the command
+ * into the Netty pipeline.
+ *
+ * @param context the associated ExecutionHandlerContext
+ * @param command the blocking command
+ * @param timeout the timeout for the command to block
+ * @param keys the list of keys the command is interested in
+ * @param commandOptions any additional options (other than the keys and timeout) required to
+ * resubmit the command.
+ */
+ public BlockingCommandListener(ExecutionHandlerContext context, RedisCommandType command,
+ long timeout, List<RedisKey> keys, byte[]... commandOptions) {
+ this.context = context;
+ this.command = command;
+ this.timeout = timeout;
+ this.keys = Collections.unmodifiableList(keys);
+ this.commandOptions = commandOptions;
+ }
+
+ @Override
+ public List<RedisKey> keys() {
+ return keys;
+ }
+
+ @Override
+ public EventResponse process(RedisCommandType commandType, RedisKey key) {
+ if (!keys.contains(key)) {
+ return EventResponse.CONTIINUE;
+ }
+
+ resubmitCommand();
+ return EventResponse.REMOVE_AND_STOP;
+ }
+
+ @Override
+ public void resubmitCommand() {
+ List<byte[]> byteArgs = new ArrayList<>(keys.size() + 1);
+ byteArgs.add(command.name().getBytes());
+
+ keys.forEach(x -> byteArgs.add(x.toBytes()));
+ byteArgs.addAll(Arrays.asList(commandOptions));
+
+ // The commands we are currently supporting all have the timeout at the end of the argument
+ // list. Some newer Redis 7 commands (BLMPOP and BZMPOP) have the timeout as the first argument
+ // after the command. We'll need to adjust this once those commands are supported.
+ byteArgs.add(Coder.longToBytes(timeout));
+
+ context.resubmitCommand(new Command(command, byteArgs));
Review comment:
I've add the timeout recalculation now, so I think this is still needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r824372929
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventListener.java
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.List;
+
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+/**
+ * Interface intended to be implemented in order to receive Redis events. Specifically this would be
+ * keyspace events or blocking commands. EventListeners are registered with the
+ * {@link EventDistributor}.
+ */
+public interface EventListener {
+
+ /**
+ * Receive and process an event. This method should execute very quickly. The return value
+ * determines additional process steps for the given event.
+ *
+ * @param commandType the command triggering the event
+ * @param key the key triggering the event
+ * @return response determining subsequent processing steps
+ */
+ EventResponse process(RedisCommandType commandType, RedisKey key);
+
+ /**
+ * Return the list of keys this listener is interested in.
+ */
+ List<RedisKey> keys();
+
+ /**
+ * Method to resubmit a command if appropriate. This is only relevant for listeners that process
+ * events for blocking commands. Listeners that handle keyspace event notification will not use
+ * this.
+ */
+ default void resubmitCommand() {};
+
+ /**
+ * Retrieve the timeout for this listener. The default is no timeout.
+ */
+ default long getTimeout() {
+ return 0;
+ }
+
+ /**
+ * Set a runnable that is responsible for any cleanup when this listener is removed.
+ */
+ default void setCleanupTask(Runnable r) {}
Review comment:
Yes you're right. I don't feel like there is a clean interface that could be used for both blocking listeners and future keyspace event listener(s). I think we'll just have to see once we get to implementing that functionality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] DonalEvans commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
DonalEvans commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r825176132
##########
File path: geode-for-redis/src/integrationTest/java/org/apache/geode/redis/internal/commands/executor/list/AbstractBLPopIntegrationTest.java
##########
@@ -92,6 +97,15 @@ public void testBLPopWhenValueExists() {
assertThat(jedis.lpop(KEY)).isEqualTo("value1");
}
+ @Test
+ public void testBLPopWithDoesNotError_whenTimeoutHasExponent() {
Review comment:
Typo in this test name, I think. Should this be "testBLPopDoesNotError_whenTimeoutHasExponent"?
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventListener.java
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.List;
+import java.util.concurrent.ScheduledExecutorService;
+
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+/**
+ * Interface intended to be implemented in order to receive Redis events. Specifically this would be
+ * keyspace events or blocking commands. EventListeners are registered with the
+ * {@link EventDistributor}.
+ */
+public interface EventListener {
+
+ /**
+ * Receive and process an event. This method should execute very quickly. The return value
+ * determines additional process steps for the given event.
+ *
+ * @param commandType the command triggering the event
+ * @param key the key triggering the event
+ * @return response determining subsequent processing steps
+ */
+ EventResponse process(RedisCommandType commandType, RedisKey key);
+
+ /**
+ * Return the list of keys this listener is interested in.
+ */
+ List<RedisKey> keys();
+
+ /**
+ * Method to resubmit a command if appropriate. This is only relevant for listeners that process
+ * events for blocking commands. Listeners that handle keyspace event notification will not use
+ * this.
+ */
+ void resubmitCommand();
+
+ /**
+ * Retrieve the timeout in nanoseconds for this listener
+ */
+ // long getTimeout();
Review comment:
This should probably be removed.
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/BlockingCommandListener.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.commands.executor.RedisResponse;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListener implements EventListener {
+
+ private final ExecutionHandlerContext context;
+ private final Command command;
+ private final List<RedisKey> keys;
+ private final double timeoutSeconds;
+ private final long timeSubmitted;
+ private AtomicBoolean active = new AtomicBoolean(true);
Review comment:
This can be `final`
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventListener.java
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.List;
+import java.util.concurrent.ScheduledExecutorService;
+
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+/**
+ * Interface intended to be implemented in order to receive Redis events. Specifically this would be
+ * keyspace events or blocking commands. EventListeners are registered with the
+ * {@link EventDistributor}.
+ */
+public interface EventListener {
+
+ /**
+ * Receive and process an event. This method should execute very quickly. The return value
+ * determines additional process steps for the given event.
+ *
+ * @param commandType the command triggering the event
+ * @param key the key triggering the event
+ * @return response determining subsequent processing steps
+ */
+ EventResponse process(RedisCommandType commandType, RedisKey key);
+
+ /**
+ * Return the list of keys this listener is interested in.
+ */
+ List<RedisKey> keys();
+
+ /**
+ * Method to resubmit a command if appropriate. This is only relevant for listeners that process
+ * events for blocking commands. Listeners that handle keyspace event notification will not use
+ * this.
+ */
+ void resubmitCommand();
+
+ /**
+ * Retrieve the timeout in nanoseconds for this listener
+ */
+ // long getTimeout();
+
+ void scheduleTimeout(ScheduledExecutorService executor, EventDistributor distributor);
Review comment:
For consistency with other methods here, this should have some javadoc comments.
##########
File path: geode-for-redis/src/integrationTest/java/org/apache/geode/redis/internal/commands/executor/list/AbstractBLPopIntegrationTest.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import static org.apache.geode.redis.RedisCommandArgumentsTestHelper.assertAtLeastNArgs;
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.BIND_ADDRESS;
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.REDIS_CLIENT_TIMEOUT;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.HostAndPort;
+import redis.clients.jedis.JedisCluster;
+import redis.clients.jedis.Protocol;
+
+import org.apache.geode.redis.RedisIntegrationTest;
+import org.apache.geode.redis.internal.RedisConstants;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.junit.rules.ExecutorServiceRule;
+
+public abstract class AbstractBLPopIntegrationTest implements RedisIntegrationTest {
+ private static final String KEY = "key";
+
+ protected JedisCluster jedis;
+
+ public abstract void awaitEventDistributorSize(int size) throws Exception;
+
+ @ClassRule
+ public static ExecutorServiceRule executor = new ExecutorServiceRule();
+
+ @Before
+ public void setUp() {
+ jedis = new JedisCluster(new HostAndPort(BIND_ADDRESS, getPort()), REDIS_CLIENT_TIMEOUT);
+ }
+
+ @After
+ public void tearDown() {
+ flushAll();
+ jedis.close();
+ }
+
+ @Test
+ public void testInvalidArguments_throwErrors() {
+ assertAtLeastNArgs(jedis, Protocol.Command.BLPOP, 2);
+ }
+
+ @Test
+ public void testInvalidTimeout_throwsError() {
+ assertThatThrownBy(() -> jedis.sendCommand("key1", Protocol.Command.BLPOP, "key1",
+ "0.A"))
+ .hasMessage(RedisConstants.ERROR_TIMEOUT_INVALID);
+ }
+
+ @Test
+ public void testKeysInDifferentSlots_throwsError() {
+ assertThatThrownBy(() -> jedis.sendCommand("key1", Protocol.Command.BLPOP, "key1",
+ "key2", "0"))
+ .hasMessage(RedisConstants.ERROR_WRONG_SLOT);
+ }
+
+ @Test
+ public void testNegativeTimeout_throwsError() {
+ assertThatThrownBy(() -> jedis.blpop(-1, "key1"))
+ .hasMessage(RedisConstants.ERROR_NEGATIVE_TIMEOUT);
+ }
+
+ @Test
+ public void testBLPopForNonListKey() {
+ jedis.set("not-a-list", "value");
+
+ assertThatThrownBy(() -> jedis.blpop(0, "not-a-list"))
+ .hasMessage(RedisConstants.ERROR_WRONG_TYPE);
+ }
+
+ @Test
+ public void testBLPopWhenValueExists() {
+ jedis.lpush(KEY, "value1", "value2");
+
+ List<String> result = jedis.blpop(0, KEY);
+
+ assertThat(result).containsExactly(KEY, "value2");
+ assertThat(jedis.lpop(KEY)).isEqualTo("value1");
+ }
+
+ @Test
+ public void testBLPopWithDoesNotError_whenTimeoutHasExponent() {
+ jedis.lpush(KEY, "value1", "value2");
+
+ Object result = jedis.sendCommand(KEY, Protocol.Command.BLPOP, KEY, "1E+3");
+
+ assertThat(result).isNotNull();
+ }
+
+ @Test
+ public void testBLPopWhenValueDoesNotExist() throws Exception {
+ Future<List<String>> future = executor.submit(() -> jedis.blpop(0, KEY));
+
+ awaitEventDistributorSize(1);
+ jedis.lpush(KEY, "value1", "value2");
+
+ assertThat(future.get()).containsExactly(KEY, "value2");
+ assertThat(jedis.lpop(KEY)).isEqualTo("value1");
+ }
+
+ @Test
+ public void testBLPopWhenTimeoutIsExceeded() {
+ int timeout = 10;
+ Future<List<String>> future = executor.submit(() -> jedis.blpop(timeout, KEY));
+ GeodeAwaitility.await().atMost(timeout * 2, TimeUnit.SECONDS)
+ .untilAsserted(() -> assertThat(future.get()).isNull());
+ }
+
+ @Test
+ public void testConcurrentBLPop() throws Exception {
+ int totalElements = 10_000;
+ List<Object> accumulated = Collections.synchronizedList(new ArrayList<>(totalElements + 2));
+ AtomicBoolean running = new AtomicBoolean(true);
+
+ Future<Void> future1 = executor.submit(() -> doBLPop(running, accumulated));
+ Future<Void> future2 = executor.submit(() -> doBLPop(running, accumulated));
+
+ List<String> result = new ArrayList<>();
+ for (int i = 0; i < totalElements; i++) {
+ jedis.lpush(KEY, "value" + i);
+ result.add("value" + i);
+ }
+
+ GeodeAwaitility.await().atMost(Duration.ofSeconds(5))
+ .untilAsserted(() -> assertThat(accumulated.size()).isEqualTo(totalElements));
Review comment:
This await should probably use the default value, since we're not trying to test the speed of commands here, just that they execute correctly. If the machine this test is running on happens to encounter some resource contention, then it's possible this test could fail due to this timeout being small.
##########
File path: geode-for-redis/src/distributedTest/java/org/apache/geode/redis/internal/commands/executor/list/BLPopDUnitTest.java
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.BIND_ADDRESS;
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.REDIS_CLIENT_TIMEOUT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.List;
+import java.util.concurrent.Future;
+
+import org.junit.After;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import redis.clients.jedis.HostAndPort;
+import redis.clients.jedis.JedisCluster;
+
+import org.apache.geode.redis.internal.GeodeRedisService;
+import org.apache.geode.test.dunit.rules.ClusterStartupRule;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+import org.apache.geode.test.junit.rules.ExecutorServiceRule;
+
+public class BLPopDUnitTest {
Review comment:
Would there be value in adding a test for what happens when we submit a BLPOP command, then crash the server the key would be hosted on if it existed, then do an LPUSH?
##########
File path: geode-for-redis/src/integrationTest/java/org/apache/geode/redis/internal/commands/executor/list/AbstractBLPopIntegrationTest.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import static org.apache.geode.redis.RedisCommandArgumentsTestHelper.assertAtLeastNArgs;
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.BIND_ADDRESS;
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.REDIS_CLIENT_TIMEOUT;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.HostAndPort;
+import redis.clients.jedis.JedisCluster;
+import redis.clients.jedis.Protocol;
+
+import org.apache.geode.redis.RedisIntegrationTest;
+import org.apache.geode.redis.internal.RedisConstants;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.junit.rules.ExecutorServiceRule;
+
+public abstract class AbstractBLPopIntegrationTest implements RedisIntegrationTest {
Review comment:
Could we also get a test for the behaviour when we pass multiple keys to BLPOP, but only one of them already exists (but it's not the first one), and a test for the behaviour when we pass multiple keys to BLPOP and multiple of them already exist (we should return the first key in the list of arguments that we passed that exists). The last test should show the issue described in `BLPopExecutor` to do with the list of keys being modified in-place by the locked execute call.
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/commands/executor/list/BLPopExecutor.java
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import static org.apache.geode.redis.internal.RedisConstants.ERROR_NEGATIVE_TIMEOUT;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.geode.redis.internal.RedisConstants;
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.executor.CommandExecutor;
+import org.apache.geode.redis.internal.commands.executor.RedisResponse;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.data.RedisList;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BLPopExecutor implements CommandExecutor {
+
+ @Override
+ public RedisResponse executeCommand(Command command, ExecutionHandlerContext context) {
+ List<byte[]> arguments = command.getCommandArguments();
+ double timeoutSeconds;
+ try {
+ timeoutSeconds = Coder.bytesToDouble(arguments.get(arguments.size() - 1));
+ } catch (NumberFormatException e) {
+ return RedisResponse.error(RedisConstants.ERROR_TIMEOUT_INVALID);
+ }
+
+ if (timeoutSeconds < 0) {
+ return RedisResponse.error(ERROR_NEGATIVE_TIMEOUT);
+ }
+
+ int keyCount = arguments.size() - 1;
+ List<RedisKey> keys = new ArrayList<>(keyCount);
+ for (int i = 0; i < keyCount; i++) {
+ keys.add(new RedisKey(arguments.get(i)));
+ }
+
+ List<byte[]> popped = context.lockedExecute(keys.get(0), keys,
+ () -> RedisList.blpop(context, command, keys, timeoutSeconds));
Review comment:
Passing `keys` to both `blpop()` and `lockedExecute()` could result in incorrect behaviour in `blpop()` since `LockingStripedCoordinator.execute()` (which is called by `lockedExecute()` sorts the list in-place, which will result in the list passed to `blpop()` also being modified. In order to prevent this, a copy of the list needs to be passed to one of the methods.
##########
File path: geode-for-redis/src/integrationTest/java/org/apache/geode/redis/internal/eventing/EventDistributorTest.java
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.stream.Collectors;
+
+import org.junit.Test;
+
+import org.apache.geode.redis.ConcurrentLoopingThreads;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributorTest {
+
+ public static class TestEventListener implements EventListener {
+ private final List<RedisKey> keys;
+ private int fired = 0;
+
+ public TestEventListener(RedisKey... keys) {
+ this.keys = Arrays.stream(keys).collect(Collectors.toList());
+ }
+
+ public int getFired() {
+ return fired;
+ }
+
+ @Override
+ public EventResponse process(RedisCommandType commandType, RedisKey key) {
+ fired += 1;
+ return EventResponse.REMOVE_AND_STOP;
+ }
+
+ @Override
+ public List<RedisKey> keys() {
+ return keys;
+ }
+
+ @Override
+ public void resubmitCommand() {}
+
+ @Override
+ public void scheduleTimeout(ScheduledExecutorService executor, EventDistributor distributor) {}
+ }
+
+ @Test
+ public void firingEventRemovesListener() {
+ RedisKey keyA = new RedisKey("a".getBytes());
+ RedisKey keyB = new RedisKey("b".getBytes());
+ EventDistributor distributor = new EventDistributor();
+ TestEventListener listener = new TestEventListener(keyA, keyB);
+ distributor.registerListener(listener);
+
+ distributor.fireEvent(null, keyA);
+ assertThat(listener.getFired()).isEqualTo(1);
+ assertThat(distributor.getRegisteredKeys()).isEqualTo(0);
+ }
+
+ @Test
+ public void firingEventRemovesFirstListener_whenMultipleExist() {
+ RedisKey keyA = new RedisKey("a".getBytes());
+ RedisKey keyB = new RedisKey("b".getBytes());
+ EventDistributor distributor = new EventDistributor();
+ TestEventListener listener1 = new TestEventListener(keyA, keyB);
+ TestEventListener listener2 = new TestEventListener(keyA, keyB);
+ distributor.registerListener(listener1);
+ distributor.registerListener(listener2);
+
+ assertThat(distributor.getRegisteredKeys()).isEqualTo(4);
+
+ distributor.fireEvent(null, keyA);
+ assertThat(listener1.getFired()).isEqualTo(1);
+ assertThat(listener2.getFired()).isEqualTo(0);
+ assertThat(distributor.getRegisteredKeys()).isEqualTo(2);
+
+ distributor.fireEvent(null, keyA);
+ assertThat(listener1.getFired()).isEqualTo(1);
+ assertThat(listener2.getFired()).isEqualTo(1);
+ assertThat(distributor.getRegisteredKeys()).isEqualTo(0);
+ }
+
+ @Test
+ public void listenerIsRemovedAfterBucketMoves() {
+ RedisKey keyA = new RedisKey("a".getBytes());
+ RedisKey keyB = new RedisKey("b".getBytes());
+ EventDistributor distributor = new EventDistributor();
+ TestEventListener listener1 = new TestEventListener(keyA);
+ distributor.registerListener(listener1);
+ distributor.registerListener(new TestEventListener(keyB));
+
+ assertThat(distributor.getRegisteredKeys()).isEqualTo(2);
+
+ distributor.afterBucketRemoved(keyA.getBucketId(), null);
+
+ assertThat(distributor.getRegisteredKeys()).isEqualTo(1);
+ }
+
+ @Test
+ public void concurrencyOfManyRegistrationsAndBucketMovementForTheSameKeys() {
+ EventDistributor distributor = new EventDistributor();
+ RedisKey keyA = new RedisKey("keyA".getBytes());
+ RedisKey keyB = new RedisKey("keyB".getBytes());
+ RedisKey keyC = new RedisKey("keyC".getBytes());
+
+ // Should not produce any exceptions
+ new ConcurrentLoopingThreads(10_000,
+ i -> distributor.registerListener(new TestEventListener(keyA, keyB, keyC)),
+ i -> distributor.registerListener(new TestEventListener(keyB, keyC, keyA)),
+ i -> distributor.registerListener(new TestEventListener(keyC, keyA, keyB)),
+ i -> distributor.afterBucketRemoved(keyA.getBucketId(), null),
+ i -> distributor.afterBucketRemoved(keyB.getBucketId(), null),
+ i -> distributor.afterBucketRemoved(keyC.getBucketId(), null),
+ i -> distributor.fireEvent(null, keyA),
+ i -> distributor.fireEvent(null, keyB),
+ i -> distributor.fireEvent(null, keyC))
+ .runInLockstep();
+
+ distributor.fireEvent(null, keyA);
+ distributor.fireEvent(null, keyB);
+ distributor.fireEvent(null, keyC);
+ assertThat(distributor.getRegisteredKeys()).isEqualTo(0);
+ }
+
+ @Test
+ public void concurrencyOfManyRegistrationAndRemovalOfSameListener() {
+ EventDistributor distributor = new EventDistributor();
+ RedisKey keyA = new RedisKey("keyA".getBytes());
+ RedisKey keyB = new RedisKey("keyB".getBytes());
+ RedisKey keyC = new RedisKey("keyC".getBytes());
+ AtomicReference<EventListener> listenerRef1 =
+ new AtomicReference<>(new TestEventListener(keyA, keyB, keyC));
+ AtomicReference<EventListener> listenerRef2 =
+ new AtomicReference<>(new TestEventListener(keyB, keyC, keyA));
+ AtomicReference<EventListener> listenerRef3 =
+ new AtomicReference<>(new TestEventListener(keyC, keyA, keyB));
+
+ // Should not produce any exceptions
+ new ConcurrentLoopingThreads(10_000,
+ i -> distributor.registerListener(listenerRef1.get()),
+ i -> distributor.registerListener(listenerRef2.get()),
+ i -> distributor.registerListener(listenerRef3.get()),
+ i -> distributor.removeListener(listenerRef1.get()),
+ i -> distributor.removeListener(listenerRef2.get()),
+ i -> distributor.removeListener(listenerRef3.get()),
+ i -> distributor.fireEvent(null, keyA),
+ i -> distributor.fireEvent(null, keyB),
+ i -> distributor.fireEvent(null, keyC))
+ .runWithAction(() -> {
+ listenerRef1.set(new TestEventListener(keyA, keyB, keyC));
+ listenerRef2.set(new TestEventListener(keyB, keyC, keyA));
+ listenerRef3.set(new TestEventListener(keyC, keyA, keyB));
+ });
+
+ distributor.fireEvent(null, keyA);
+ distributor.fireEvent(null, keyB);
+ distributor.fireEvent(null, keyC);
+ assertThat(distributor.getRegisteredKeys()).isEqualTo(0);
+ }
+
+ @Test
+ public void ensureNotRegisteringListenerOnQueueJustRemoved() {
Review comment:
This test name confuses me a little. Could it be reworded to better explain what the test is doing or intending to show please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r827967329
##########
File path: geode-for-redis/src/distributedTest/java/org/apache/geode/redis/internal/commands/executor/list/BLPopDUnitTest.java
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.BIND_ADDRESS;
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.REDIS_CLIENT_TIMEOUT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.List;
+import java.util.concurrent.Future;
+
+import org.junit.After;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import redis.clients.jedis.HostAndPort;
+import redis.clients.jedis.JedisCluster;
+
+import org.apache.geode.redis.internal.GeodeRedisService;
+import org.apache.geode.test.dunit.rules.ClusterStartupRule;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+import org.apache.geode.test.junit.rules.ExecutorServiceRule;
+
+public class BLPopDUnitTest {
Review comment:
Added, although the test really exercises the Jedis client more than our code. At least we can understand its behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r825628070
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventDistributor.java
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.cache.partition.PartitionListenerAdapter;
+import org.apache.geode.logging.internal.executors.LoggingThreadFactory;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributor extends PartitionListenerAdapter {
+
+ private final Map<RedisKey, Queue<EventListener>> listeners = new ConcurrentHashMap<>();
+
+ private final ScheduledThreadPoolExecutor timerExecutor =
+ new ScheduledThreadPoolExecutor(1,
+ new LoggingThreadFactory("GeodeForRedisEventTimer-", true));
+
+ private int keysRegistered = 0;
+
+ public EventDistributor() {
+ timerExecutor.setRemoveOnCancelPolicy(true);
+ }
+
+ public synchronized void registerListener(EventListener listener) {
+ for (RedisKey key : listener.keys()) {
+ listeners.computeIfAbsent(key, k -> new LinkedBlockingQueue<>()).add(listener);
+ }
+
+ if (listener.getTimeout() != 0) {
+ scheduleTimeout(listener, listener.getTimeout());
+ }
+
+ keysRegistered += listener.keys().size();
+ }
+
+ public void fireEvent(RedisCommandType command, RedisKey key) {
+ Queue<EventListener> listenerList = listeners.get(key);
+ if (listenerList == null) {
+ return;
+ }
+
+ for (EventListener listener : listenerList) {
+ if (listener.process(command, key) == EventResponse.REMOVE_AND_STOP) {
+ removeListener(listener);
+ break;
+ }
+ }
+ }
+
+ /**
+ * The total number of keys registered by all listeners (includes duplicates).
+ */
+ @VisibleForTesting
+ public int size() {
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r819932850
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/BlockingCommandListener.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListener implements EventListener {
+
+ private final ExecutionHandlerContext context;
+ private final RedisCommandType command;
+ private final List<RedisKey> keys;
+ private final byte[][] commandOptions;
+ private final long timeout;
+
+ /**
+ * Constructor to create an instance of a BlockingCommandListener in response to a blocking
+ * command. When receiving a relevant event, blocking commands simply resubmit the command
+ * into the Netty pipeline.
+ *
+ * @param context the associated ExecutionHandlerContext
+ * @param command the blocking command
+ * @param timeout the timeout for the command to block
+ * @param keys the list of keys the command is interested in
+ * @param commandOptions any additional options (other than the keys and timeout) required to
+ * resubmit the command.
+ */
+ public BlockingCommandListener(ExecutionHandlerContext context, RedisCommandType command,
+ long timeout, List<RedisKey> keys, byte[]... commandOptions) {
+ this.context = context;
+ this.command = command;
+ this.timeout = timeout;
+ this.keys = Collections.unmodifiableList(keys);
+ this.commandOptions = commandOptions;
+ }
+
+ @Override
+ public List<RedisKey> keys() {
+ return keys;
+ }
+
+ @Override
+ public EventResponse process(RedisCommandType commandType, RedisKey key) {
+ if (!keys.contains(key)) {
+ return EventResponse.CONTIINUE;
+ }
+
+ resubmitCommand();
+ return EventResponse.REMOVE_AND_STOP;
+ }
+
+ @Override
+ public void resubmitCommand() {
+ List<byte[]> byteArgs = new ArrayList<>(keys.size() + 1);
+ byteArgs.add(command.name().getBytes());
+
+ keys.forEach(x -> byteArgs.add(x.toBytes()));
+ byteArgs.addAll(Arrays.asList(commandOptions));
+
+ // The commands we are currently supporting all have the timeout at the end of the argument
+ // list. Some newer Redis 7 commands (BLMPOP and BZMPOP) have the timeout as the first argument
+ // after the command. We'll need to adjust this once those commands are supported.
+ byteArgs.add(Coder.longToBytes(timeout));
+
+ context.resubmitCommand(new Command(command, byteArgs));
Review comment:
Oh, I forgot to add something here; I think we need to recalculate the timeout value for resubmitted commands. That was why I thought we needed to rebuild the command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r820881659
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventDistributor.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Timer;
+import java.util.TimerTask;
+
+import org.apache.logging.log4j.Logger;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.cache.partition.PartitionListenerAdapter;
+import org.apache.geode.logging.internal.log4j.api.LogService;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributor extends PartitionListenerAdapter {
+
+ private static final Logger logger = LogService.getLogger();
+ private final Map<RedisKey, List<EventListener>> listeners = new HashMap<>();
+ private final Map<EventListener, TimerTask> timerTasks = new HashMap<>();
+ private final Timer timer = new Timer("GeodeForRedisEventTimer", true);
+ private int keysRegistered = 0;
+
+ public synchronized void registerListener(EventListener listener) {
+ for (RedisKey key : listener.keys()) {
+ listeners.computeIfAbsent(key, k -> new ArrayList<>()).add(listener);
+ }
+
+ if (listener.getTimeout() != 0) {
+ scheduleTimeout(listener, listener.getTimeout());
+ }
+
+ keysRegistered += listener.keys().size();
+ }
+
+ public synchronized void fireEvent(RedisCommandType command, RedisKey key) {
Review comment:
I've been able to remove the synchronization here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r825629391
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/data/RedisList.java
##########
@@ -154,6 +160,35 @@ public long lpush(List<byte[]> elementsToAdd, Region<RedisKey, RedisData> region
return popped;
}
+ public static List<byte[]> blpop(ExecutionHandlerContext context, List<RedisKey> keys,
+ double timeoutSeconds) {
+ RegionProvider regionProvider = context.getRegionProvider();
+ for (RedisKey key : keys) {
+ RedisList list = regionProvider.getTypedRedisData(REDIS_LIST, key, false);
+ if (!list.isNull()) {
+ byte[] poppedValue = list.lpop(context.getRegion(), key);
+
+ // return the key and value
+ List<byte[]> result = new ArrayList<>(2);
+ result.add(key.toBytes());
+ result.add(poppedValue);
+ return result;
+ }
+ }
+
+ List<byte[]> commandArgs = new ArrayList<>(keys.size() + 1);
+ commandArgs.add(RedisCommandType.BLPOP.name().getBytes());
+ keys.forEach(x -> commandArgs.add(x.toBytes()));
+ // This is a placeholder for the timeout. If the command is resubmitted it will be updated with
+ // an adjusted timeout value.
+ commandArgs.add("0".getBytes());
Review comment:
`'0'` is a `char` and not a `byte[]`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r824372570
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/netty/ExecutionHandlerContext.java
##########
@@ -232,7 +236,14 @@ public void channelInactive(ChannelHandlerContext ctx) {
}
}
- private void executeCommand(Command command) throws Exception {
+ public void resubmitCommand(Command command) {
+ ChannelHandlerContext ctx =
+ channel.pipeline().context(ByteToCommandDecoder.class.getSimpleName());
+ ctx.fireChannelRead(command);
Review comment:
99% sure :). I think the only time it might not be the case is when the current thread is also the same thread that is associated with the original blocking command's ExecutionHandlerContext.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r824375956
##########
File path: geode-for-redis/src/integrationTest/java/org/apache/geode/redis/internal/commands/executor/list/AbstractBLPopIntegrationTest.java
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.BIND_ADDRESS;
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.REDIS_CLIENT_TIMEOUT;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.HostAndPort;
+import redis.clients.jedis.JedisCluster;
+import redis.clients.jedis.Protocol;
+
+import org.apache.geode.redis.RedisIntegrationTest;
+import org.apache.geode.redis.internal.RedisConstants;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.junit.rules.ExecutorServiceRule;
+
+public abstract class AbstractBLPopIntegrationTest implements RedisIntegrationTest {
+ private static final String KEY = "key";
+
+ protected JedisCluster jedis;
+
+ public abstract void awaitEventDistributorSize(int size) throws Exception;
+
+ @ClassRule
+ public static ExecutorServiceRule executor = new ExecutorServiceRule();
+
+ @Before
+ public void setUp() {
+ jedis = new JedisCluster(new HostAndPort(BIND_ADDRESS, getPort()), REDIS_CLIENT_TIMEOUT);
+ }
+
+ @After
+ public void tearDown() {
+ flushAll();
+ jedis.close();
+ }
+
+ @Test
+ public void testInvalidArguments_throwErrors() {
+ assertThatThrownBy(() -> jedis.sendCommand("key", Protocol.Command.BLPOP))
+ .hasMessageContaining("ERR wrong number of arguments for 'blpop' command");
+ assertThatThrownBy(() -> jedis.sendCommand("key1", Protocol.Command.BLPOP, "key"))
+ .hasMessageContaining("ERR wrong number of arguments for 'blpop' command");
+ }
+
+ @Test
+ public void testInvalidTimeout_throwsError() {
Review comment:
Added.
##########
File path: geode-for-redis/src/integrationTest/java/org/apache/geode/redis/internal/commands/executor/list/AbstractBLPopIntegrationTest.java
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.BIND_ADDRESS;
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.REDIS_CLIENT_TIMEOUT;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.HostAndPort;
+import redis.clients.jedis.JedisCluster;
+import redis.clients.jedis.Protocol;
+
+import org.apache.geode.redis.RedisIntegrationTest;
+import org.apache.geode.redis.internal.RedisConstants;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.junit.rules.ExecutorServiceRule;
+
+public abstract class AbstractBLPopIntegrationTest implements RedisIntegrationTest {
+ private static final String KEY = "key";
+
+ protected JedisCluster jedis;
+
+ public abstract void awaitEventDistributorSize(int size) throws Exception;
+
+ @ClassRule
+ public static ExecutorServiceRule executor = new ExecutorServiceRule();
+
+ @Before
+ public void setUp() {
+ jedis = new JedisCluster(new HostAndPort(BIND_ADDRESS, getPort()), REDIS_CLIENT_TIMEOUT);
+ }
+
+ @After
+ public void tearDown() {
+ flushAll();
+ jedis.close();
+ }
+
+ @Test
+ public void testInvalidArguments_throwErrors() {
+ assertThatThrownBy(() -> jedis.sendCommand("key", Protocol.Command.BLPOP))
+ .hasMessageContaining("ERR wrong number of arguments for 'blpop' command");
+ assertThatThrownBy(() -> jedis.sendCommand("key1", Protocol.Command.BLPOP, "key"))
+ .hasMessageContaining("ERR wrong number of arguments for 'blpop' command");
+ }
+
+ @Test
+ public void testInvalidTimeout_throwsError() {
+ assertThatThrownBy(() -> jedis.sendCommand("key1", Protocol.Command.BLPOP, "key1",
+ "0.A"))
+ .hasMessageContaining(RedisConstants.ERROR_TIMEOUT_INVALID);
Review comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r824375367
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/commands/executor/list/BLPopExecutor.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.geode.redis.internal.RedisConstants;
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.executor.CommandExecutor;
+import org.apache.geode.redis.internal.commands.executor.RedisResponse;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.data.RedisList;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BLPopExecutor implements CommandExecutor {
+
+ @Override
+ public RedisResponse executeCommand(Command command, ExecutionHandlerContext context) {
+ List<byte[]> arguments = command.getCommandArguments();
+ int keyCount = arguments.size() - 1;
+ double timeoutMillis;
+ try {
+ timeoutMillis = Coder.bytesToDouble(arguments.get(keyCount)) * 1000;
+ } catch (NumberFormatException e) {
+ return RedisResponse.error(RedisConstants.ERROR_TIMEOUT_INVALID);
+ }
Review comment:
Good catch. That should be fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r827964214
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventListener.java
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.List;
+import java.util.concurrent.ScheduledExecutorService;
+
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+/**
+ * Interface intended to be implemented in order to receive Redis events. Specifically this would be
+ * keyspace events or blocking commands. EventListeners are registered with the
+ * {@link EventDistributor}.
+ */
+public interface EventListener {
+
+ /**
+ * Receive and process an event. This method should execute very quickly. The return value
+ * determines additional process steps for the given event.
+ *
+ * @param commandType the command triggering the event
+ * @param key the key triggering the event
+ * @return response determining subsequent processing steps
+ */
+ EventResponse process(RedisCommandType commandType, RedisKey key);
+
+ /**
+ * Return the list of keys this listener is interested in.
+ */
+ List<RedisKey> keys();
+
+ /**
+ * Method to resubmit a command if appropriate. This is only relevant for listeners that process
+ * events for blocking commands. Listeners that handle keyspace event notification will not use
+ * this.
+ */
+ void resubmitCommand();
+
+ /**
+ * Retrieve the timeout in nanoseconds for this listener
+ */
+ // long getTimeout();
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r828549463
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/BlockingCommandListener.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.commands.executor.RedisResponse;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListener implements EventListener {
+
+ private final ExecutionHandlerContext context;
+ private final Command command;
+ private final List<RedisKey> keys;
+ private final double timeoutSeconds;
+ private final long timeSubmitted;
+ private final AtomicBoolean active = new AtomicBoolean(true);
+
+ /**
+ * Constructor to create an instance of a BlockingCommandListener in response to a blocking
+ * command. When receiving a relevant event, blocking commands simply resubmit the command
+ * into the Netty pipeline.
+ *
+ * @param context the associated ExecutionHandlerContext
+ * @param command the blocking command associated with this listener
+ * @param keys the list of keys the command is interested in
+ * @param timeoutSeconds the timeout for the command to block in seconds
+ */
+ public BlockingCommandListener(ExecutionHandlerContext context, Command command,
+ List<RedisKey> keys, double timeoutSeconds) {
+ this.context = context;
+ this.command = command;
+ this.timeoutSeconds = timeoutSeconds;
+ this.keys = Collections.unmodifiableList(keys);
+ timeSubmitted = System.nanoTime();
+ }
+
+ @Override
+ public List<RedisKey> keys() {
+ return keys;
+ }
+
+ @Override
+ public EventResponse process(RedisCommandType commandType, RedisKey key) {
+ if (!keys.contains(key)) {
+ return EventResponse.CONTINUE;
+ }
+
+ if (active.compareAndSet(true, false)) {
+ resubmitCommand();
+ }
+ return EventResponse.REMOVE_AND_STOP;
+ }
+
+ @Override
+ public void resubmitCommand() {
+ // Recalculate the timeout since we've already been waiting
+ double adjustedTimeoutSeconds = 0;
+ if (timeoutSeconds > 0.0D) {
+ long timeoutNanos = (long) (timeoutSeconds * 1e9);
+ long adjustedTimeoutNanos = timeoutNanos - (System.nanoTime() - timeSubmitted);
+ adjustedTimeoutNanos = Math.max(1, adjustedTimeoutNanos);
+ adjustedTimeoutSeconds = ((double) adjustedTimeoutNanos) / 1e9;
+ }
+
+ // The commands we are currently supporting all have the timeout at the end of the argument
+ // list. Some newer Redis 7 commands (BLMPOP and BZMPOP) have the timeout as the first argument
+ // after the command. We'll need to adjust this once those commands are supported.
+ List<byte[]> commandArguments = command.getCommandArguments();
+ commandArguments.set(commandArguments.size() - 1, Coder.doubleToBytes(adjustedTimeoutSeconds));
+
+ context.resubmitCommand(command);
+ }
+
+ @Override
+ public void scheduleTimeout(ScheduledExecutorService executor, EventDistributor distributor) {
+ if (timeoutSeconds == 0) {
Review comment:
No harm in doing so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] dschneider-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
dschneider-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r828311176
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/BlockingCommandListener.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.commands.executor.RedisResponse;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListener implements EventListener {
+
+ private final ExecutionHandlerContext context;
+ private final Command command;
+ private final List<RedisKey> keys;
+ private final double timeoutSeconds;
+ private final long timeSubmitted;
+ private final AtomicBoolean active = new AtomicBoolean(true);
+
+ /**
+ * Constructor to create an instance of a BlockingCommandListener in response to a blocking
+ * command. When receiving a relevant event, blocking commands simply resubmit the command
+ * into the Netty pipeline.
+ *
+ * @param context the associated ExecutionHandlerContext
+ * @param command the blocking command associated with this listener
+ * @param keys the list of keys the command is interested in
+ * @param timeoutSeconds the timeout for the command to block in seconds
+ */
+ public BlockingCommandListener(ExecutionHandlerContext context, Command command,
+ List<RedisKey> keys, double timeoutSeconds) {
+ this.context = context;
+ this.command = command;
+ this.timeoutSeconds = timeoutSeconds;
+ this.keys = Collections.unmodifiableList(keys);
+ timeSubmitted = System.nanoTime();
+ }
+
+ @Override
+ public List<RedisKey> keys() {
+ return keys;
+ }
+
+ @Override
+ public EventResponse process(RedisCommandType commandType, RedisKey key) {
+ if (!keys.contains(key)) {
+ return EventResponse.CONTINUE;
+ }
+
+ if (active.compareAndSet(true, false)) {
Review comment:
Should the active.compareAndSet be in resubmitCommand instead of in this call? If we have bucket move it calls resubmitCommand but if that listener is no longer active do we still want to resubmit it? I think we only want to resubmit once which will create a new listener.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on pull request #7408:
URL: https://github.com/apache/geode/pull/7408#issuecomment-1069766895
@dschneider-pivotal I also made the change of using `JavaWorkarounds.computeIfAbsent`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r820879474
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventDistributor.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Timer;
+import java.util.TimerTask;
+
+import org.apache.logging.log4j.Logger;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.cache.partition.PartitionListenerAdapter;
+import org.apache.geode.logging.internal.log4j.api.LogService;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributor extends PartitionListenerAdapter {
+
+ private static final Logger logger = LogService.getLogger();
+ private final Map<RedisKey, List<EventListener>> listeners = new HashMap<>();
+ private final Map<EventListener, TimerTask> timerTasks = new HashMap<>();
+ private final Timer timer = new Timer("GeodeForRedisEventTimer", true);
+ private int keysRegistered = 0;
+
+ public synchronized void registerListener(EventListener listener) {
+ for (RedisKey key : listener.keys()) {
+ listeners.computeIfAbsent(key, k -> new ArrayList<>()).add(listener);
+ }
+
+ if (listener.getTimeout() != 0) {
+ scheduleTimeout(listener, listener.getTimeout());
+ }
+
+ keysRegistered += listener.keys().size();
+ }
+
+ public synchronized void fireEvent(RedisCommandType command, RedisKey key) {
+ if (listeners.isEmpty()) {
+ return;
+ }
+
+ List<EventListener> listenerList = listeners.get(key);
+ if (listenerList == null) {
+ return;
+ }
+
+ for (EventListener listener : listenerList) {
+ if (listener.process(command, key) == EventResponse.REMOVE_AND_STOP) {
+ removeListener(listener);
+ break;
+ }
+ }
+ }
+
+ /**
+ * The total number of keys registered by all listeners (includes duplicates).
+ */
+ @VisibleForTesting
+ public int size() {
+ return keysRegistered;
+ }
+
+ @Override
+ public synchronized void afterBucketRemoved(int bucketId, Iterable<?> keys) {
+ Set<EventListener> resubmittingList = new HashSet<>();
+ for (Map.Entry<RedisKey, List<EventListener>> entry : listeners.entrySet()) {
+ if (entry.getKey().getBucketId() == bucketId) {
+ resubmittingList.addAll(entry.getValue());
+ }
+ }
+
+ resubmittingList.forEach(x -> {
Review comment:
I think it's not that bad since the resubmission, just queues the command up again for netty to process. So the actual work should happen on a different netty thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r820876840
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/BlockingCommandListener.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListener implements EventListener {
+
+ private final ExecutionHandlerContext context;
+ private final RedisCommandType command;
+ private final List<RedisKey> keys;
+ private final byte[][] commandOptions;
+ private final long timeout;
+
+ /**
+ * Constructor to create an instance of a BlockingCommandListener in response to a blocking
+ * command. When receiving a relevant event, blocking commands simply resubmit the command
+ * into the Netty pipeline.
+ *
+ * @param context the associated ExecutionHandlerContext
+ * @param command the blocking command
+ * @param timeout the timeout for the command to block
+ * @param keys the list of keys the command is interested in
+ * @param commandOptions any additional options (other than the keys and timeout) required to
+ * resubmit the command.
+ */
+ public BlockingCommandListener(ExecutionHandlerContext context, RedisCommandType command,
+ long timeout, List<RedisKey> keys, byte[]... commandOptions) {
+ this.context = context;
+ this.command = command;
+ this.timeout = timeout;
+ this.keys = Collections.unmodifiableList(keys);
+ this.commandOptions = commandOptions;
+ }
+
+ @Override
+ public List<RedisKey> keys() {
+ return keys;
+ }
+
+ @Override
+ public EventResponse process(RedisCommandType commandType, RedisKey key) {
+ if (!keys.contains(key)) {
+ return EventResponse.CONTIINUE;
Review comment:
Fixed
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/netty/ExecutionHandlerContext.java
##########
@@ -132,15 +135,17 @@ public ExecutionHandlerContext(Channel channel,
this.serverPort = serverPort;
this.member = member;
this.securityService = securityService;
+ this.eventDistributor = eventDistributor;
scanCursor = new BigInteger("0");
- channelId = channel.id();
redisStats.addClient();
channel.closeFuture().addListener(future -> logout());
}
- public ChannelFuture writeToChannel(RedisResponse response) {
- return client.writeToChannel(response);
+ public void writeToChannel(RedisResponse response) {
+ if (response != null) {
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r824370655
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/data/RedisList.java
##########
@@ -154,6 +159,29 @@ public long lpush(List<byte[]> elementsToAdd, Region<RedisKey, RedisData> region
return popped;
}
+ public static List<byte[]> blpop(ExecutionHandlerContext context, List<RedisKey> keys,
+ int timeout) {
+ Region<RedisKey, RedisData> region = context.getRegion();
+ for (RedisKey key : keys) {
+ if (region.containsKey(key)) {
Review comment:
Fixed. I just used `getTypedRedisData` and called `isNull()` on the value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] dschneider-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
dschneider-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r819838911
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/BlockingCommandListener.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListener implements EventListener {
+
+ private final ExecutionHandlerContext context;
+ private final RedisCommandType command;
+ private final List<RedisKey> keys;
+ private final byte[][] commandOptions;
+ private final long timeout;
+
+ /**
+ * Constructor to create an instance of a BlockingCommandListener in response to a blocking
+ * command. When receiving a relevant event, blocking commands simply resubmit the command
+ * into the Netty pipeline.
+ *
+ * @param context the associated ExecutionHandlerContext
+ * @param command the blocking command
+ * @param timeout the timeout for the command to block
+ * @param keys the list of keys the command is interested in
+ * @param commandOptions any additional options (other than the keys and timeout) required to
+ * resubmit the command.
+ */
+ public BlockingCommandListener(ExecutionHandlerContext context, RedisCommandType command,
+ long timeout, List<RedisKey> keys, byte[]... commandOptions) {
+ this.context = context;
+ this.command = command;
+ this.timeout = timeout;
+ this.keys = Collections.unmodifiableList(keys);
+ this.commandOptions = commandOptions;
+ }
+
+ @Override
+ public List<RedisKey> keys() {
+ return keys;
+ }
+
+ @Override
+ public EventResponse process(RedisCommandType commandType, RedisKey key) {
+ if (!keys.contains(key)) {
+ return EventResponse.CONTIINUE;
+ }
+
+ resubmitCommand();
+ return EventResponse.REMOVE_AND_STOP;
+ }
+
+ @Override
+ public void resubmitCommand() {
+ List<byte[]> byteArgs = new ArrayList<>(keys.size() + 1);
+ byteArgs.add(command.name().getBytes());
+
+ keys.forEach(x -> byteArgs.add(x.toBytes()));
+ byteArgs.addAll(Arrays.asList(commandOptions));
+
+ // The commands we are currently supporting all have the timeout at the end of the argument
+ // list. Some newer Redis 7 commands (BLMPOP and BZMPOP) have the timeout as the first argument
+ // after the command. We'll need to adjust this once those commands are supported.
+ byteArgs.add(Coder.longToBytes(timeout));
+
+ context.resubmitCommand(new Command(command, byteArgs));
Review comment:
Could instead have Command.resubmit(context)? Command is immutable so can't we just reuse it instead of creating a new one? And Command knows how to find its RedisCommandType and use it to call commandExecutor.executeCommand. No need to call checkParameters again. It seems like this would let you get rid of commandOptions on this class and no need to rebuild argument list in this method. It would also get rid of the redis 7 issue you comment on.
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventDistributor.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Timer;
+import java.util.TimerTask;
+
+import org.apache.logging.log4j.Logger;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.cache.partition.PartitionListenerAdapter;
+import org.apache.geode.logging.internal.log4j.api.LogService;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributor extends PartitionListenerAdapter {
+
+ private static final Logger logger = LogService.getLogger();
+ private final Map<RedisKey, List<EventListener>> listeners = new HashMap<>();
+ private final Map<EventListener, TimerTask> timerTasks = new HashMap<>();
+ private final Timer timer = new Timer("GeodeForRedisEventTimer", true);
+ private int keysRegistered = 0;
+
+ public synchronized void registerListener(EventListener listener) {
+ for (RedisKey key : listener.keys()) {
+ listeners.computeIfAbsent(key, k -> new ArrayList<>()).add(listener);
+ }
+
+ if (listener.getTimeout() != 0) {
+ scheduleTimeout(listener, listener.getTimeout());
+ }
+
+ keysRegistered += listener.keys().size();
+ }
+
+ public synchronized void fireEvent(RedisCommandType command, RedisKey key) {
+ if (listeners.isEmpty()) {
+ return;
+ }
+
+ List<EventListener> listenerList = listeners.get(key);
+ if (listenerList == null) {
+ return;
+ }
+
+ for (EventListener listener : listenerList) {
+ if (listener.process(command, key) == EventResponse.REMOVE_AND_STOP) {
+ removeListener(listener);
+ break;
+ }
+ }
+ }
+
+ /**
+ * The total number of keys registered by all listeners (includes duplicates).
+ */
+ @VisibleForTesting
+ public int size() {
+ return keysRegistered;
+ }
+
+ @Override
+ public synchronized void afterBucketRemoved(int bucketId, Iterable<?> keys) {
+ Set<EventListener> resubmittingList = new HashSet<>();
+ for (Map.Entry<RedisKey, List<EventListener>> entry : listeners.entrySet()) {
+ if (entry.getKey().getBucketId() == bucketId) {
+ resubmittingList.addAll(entry.getValue());
+ }
+ }
+
+ resubmittingList.forEach(x -> {
+ x.resubmitCommand();
+ removeListener(x);
+ });
+ }
+
+ private synchronized void removeListener(EventListener listener) {
+ boolean listenerRemoved = false;
+ for (RedisKey key : listener.keys()) {
+ List<EventListener> listenerList = listeners.get(key);
+ listenerRemoved |= listenerList.remove(listener);
+ if (listenerList.isEmpty()) {
+ listeners.remove(key);
+ }
+ }
+
+ if (listenerRemoved) {
+ keysRegistered -= listener.keys().size();
+ if (listener.getTimeout() > 0) {
+ timerTasks.remove(listener).cancel();
Review comment:
This "cancel()" call could very likely lead to what looks like a memory leak. Timer does not actually remove the listener from its internal queue when you call cancel(). Instead it just marks it as cancelled. Then, if it ever get to the point in time when it would have expired, it will find it and see that it was cancelled. So if users have a long timeout, because the are very patient, then this will hold on to all those listeners and everything they reference for that long period of time even though we called cancel() quickly.
You have two easy options:
1. keep using Timer but keep track of how many times you have called cancel and when that hits a certain number (how ever many zombie listeners you want to allow) call Timer.purge().
2. use ScheduledThreadPoolExecutor instead of timer call call setRemoveOnCancelPolicy(true). This cause the remove to be done right when you call cancel. The remove has to search the queue for the listener so with a big queue this could be expensive. But this has the advantage of one thread not getting stuck with doing a possibly expensive purge call. ScheduledThreadPoolExecutor is more modern and has some possibly useful features that Timer does not. See the javadocs Timer class comment. If suggests using ScheduledThreadPoolExecutor instead of it.
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventDistributor.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Timer;
+import java.util.TimerTask;
+
+import org.apache.logging.log4j.Logger;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.cache.partition.PartitionListenerAdapter;
+import org.apache.geode.logging.internal.log4j.api.LogService;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributor extends PartitionListenerAdapter {
+
+ private static final Logger logger = LogService.getLogger();
+ private final Map<RedisKey, List<EventListener>> listeners = new HashMap<>();
+ private final Map<EventListener, TimerTask> timerTasks = new HashMap<>();
Review comment:
Another option to consider is to expand EventListener to allow its TimerTask to be added to it. Then you just have one map "listeners" instead of two.
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/netty/ExecutionHandlerContext.java
##########
@@ -132,15 +135,17 @@ public ExecutionHandlerContext(Channel channel,
this.serverPort = serverPort;
this.member = member;
this.securityService = securityService;
+ this.eventDistributor = eventDistributor;
scanCursor = new BigInteger("0");
- channelId = channel.id();
redisStats.addClient();
channel.closeFuture().addListener(future -> logout());
}
- public ChannelFuture writeToChannel(RedisResponse response) {
- return client.writeToChannel(response);
+ public void writeToChannel(RedisResponse response) {
+ if (response != null) {
Review comment:
Would it be better to have a special RedisResponse.BLOCKED instead of using null?
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventListener.java
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.List;
+
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+/**
+ * Interface intended to be implemented in order to receive Redis events. Specifically this would be
+ * keyspace events or blocking commands. EventListeners are registered with the
+ * {@link EventDistributor}.
+ */
+public interface EventListener {
+
+ /**
+ * Receive and process an event. This method should execute very quickly. The return value
+ * determines additional process steps for the given event.
+ *
+ * @param commandType the command triggering the event
+ * @param key the key triggering the event
+ * @return response determining subsequent processing steps
+ */
+ EventResponse process(RedisCommandType commandType, RedisKey key);
+
+ /**
+ * Return the list of keys this listener is interested in.
+ */
+ List<RedisKey> keys();
+
+ /**
+ * Method to resubmit a command if appropriate. This is only relevant for listeners that process
+ * events for blocking commands. Listeners that handle keyspace event notification will not use
+ * this.
+ */
+ default void resubmitCommand() {};
+
+ /**
+ * Retrieve the timeout for this listener. The default is no timeout.
Review comment:
describe the time unitss
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/BlockingCommandListener.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListener implements EventListener {
+
+ private final ExecutionHandlerContext context;
+ private final RedisCommandType command;
+ private final List<RedisKey> keys;
+ private final byte[][] commandOptions;
+ private final long timeout;
+
+ /**
+ * Constructor to create an instance of a BlockingCommandListener in response to a blocking
+ * command. When receiving a relevant event, blocking commands simply resubmit the command
+ * into the Netty pipeline.
+ *
+ * @param context the associated ExecutionHandlerContext
+ * @param command the blocking command
+ * @param timeout the timeout for the command to block
+ * @param keys the list of keys the command is interested in
+ * @param commandOptions any additional options (other than the keys and timeout) required to
+ * resubmit the command.
+ */
+ public BlockingCommandListener(ExecutionHandlerContext context, RedisCommandType command,
+ long timeout, List<RedisKey> keys, byte[]... commandOptions) {
+ this.context = context;
+ this.command = command;
+ this.timeout = timeout;
+ this.keys = Collections.unmodifiableList(keys);
+ this.commandOptions = commandOptions;
+ }
+
+ @Override
+ public List<RedisKey> keys() {
+ return keys;
+ }
+
+ @Override
+ public EventResponse process(RedisCommandType commandType, RedisKey key) {
+ if (!keys.contains(key)) {
+ return EventResponse.CONTIINUE;
Review comment:
typo "II"
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventDistributor.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Timer;
+import java.util.TimerTask;
+
+import org.apache.logging.log4j.Logger;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.cache.partition.PartitionListenerAdapter;
+import org.apache.geode.logging.internal.log4j.api.LogService;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributor extends PartitionListenerAdapter {
+
+ private static final Logger logger = LogService.getLogger();
+ private final Map<RedisKey, List<EventListener>> listeners = new HashMap<>();
+ private final Map<EventListener, TimerTask> timerTasks = new HashMap<>();
+ private final Timer timer = new Timer("GeodeForRedisEventTimer", true);
+ private int keysRegistered = 0;
+
+ public synchronized void registerListener(EventListener listener) {
+ for (RedisKey key : listener.keys()) {
+ listeners.computeIfAbsent(key, k -> new ArrayList<>()).add(listener);
+ }
+
+ if (listener.getTimeout() != 0) {
+ scheduleTimeout(listener, listener.getTimeout());
+ }
+
+ keysRegistered += listener.keys().size();
+ }
+
+ public synchronized void fireEvent(RedisCommandType command, RedisKey key) {
+ if (listeners.isEmpty()) {
+ return;
+ }
+
+ List<EventListener> listenerList = listeners.get(key);
+ if (listenerList == null) {
+ return;
+ }
+
+ for (EventListener listener : listenerList) {
+ if (listener.process(command, key) == EventResponse.REMOVE_AND_STOP) {
+ removeListener(listener);
+ break;
+ }
+ }
+ }
+
+ /**
+ * The total number of keys registered by all listeners (includes duplicates).
+ */
+ @VisibleForTesting
+ public int size() {
+ return keysRegistered;
+ }
+
+ @Override
+ public synchronized void afterBucketRemoved(int bucketId, Iterable<?> keys) {
+ Set<EventListener> resubmittingList = new HashSet<>();
+ for (Map.Entry<RedisKey, List<EventListener>> entry : listeners.entrySet()) {
+ if (entry.getKey().getBucketId() == bucketId) {
+ resubmittingList.addAll(entry.getValue());
+ }
+ }
+
+ resubmittingList.forEach(x -> {
Review comment:
This could take a while (i.e. processing the resubmittingList) since it has to send an error response to each client with a blocked command. So should it be done in a background thread instead of holding up whatever thread called afterBucketMoved? It definitely seems wrong to do this part while synchronized on "this".
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventDistributor.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Timer;
+import java.util.TimerTask;
+
+import org.apache.logging.log4j.Logger;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.cache.partition.PartitionListenerAdapter;
+import org.apache.geode.logging.internal.log4j.api.LogService;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributor extends PartitionListenerAdapter {
+
+ private static final Logger logger = LogService.getLogger();
+ private final Map<RedisKey, List<EventListener>> listeners = new HashMap<>();
+ private final Map<EventListener, TimerTask> timerTasks = new HashMap<>();
+ private final Timer timer = new Timer("GeodeForRedisEventTimer", true);
+ private int keysRegistered = 0;
+
+ public synchronized void registerListener(EventListener listener) {
+ for (RedisKey key : listener.keys()) {
+ listeners.computeIfAbsent(key, k -> new ArrayList<>()).add(listener);
+ }
+
+ if (listener.getTimeout() != 0) {
+ scheduleTimeout(listener, listener.getTimeout());
+ }
+
+ keysRegistered += listener.keys().size();
+ }
+
+ public synchronized void fireEvent(RedisCommandType command, RedisKey key) {
+ if (listeners.isEmpty()) {
Review comment:
if listeners is empty will calling listeners.get(key) be that expensive? It seems like you could probably get rid of the isEmpty check here
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventDistributor.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Timer;
+import java.util.TimerTask;
+
+import org.apache.logging.log4j.Logger;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.cache.partition.PartitionListenerAdapter;
+import org.apache.geode.logging.internal.log4j.api.LogService;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributor extends PartitionListenerAdapter {
+
+ private static final Logger logger = LogService.getLogger();
+ private final Map<RedisKey, List<EventListener>> listeners = new HashMap<>();
+ private final Map<EventListener, TimerTask> timerTasks = new HashMap<>();
+ private final Timer timer = new Timer("GeodeForRedisEventTimer", true);
+ private int keysRegistered = 0;
+
+ public synchronized void registerListener(EventListener listener) {
+ for (RedisKey key : listener.keys()) {
+ listeners.computeIfAbsent(key, k -> new ArrayList<>()).add(listener);
+ }
+
+ if (listener.getTimeout() != 0) {
+ scheduleTimeout(listener, listener.getTimeout());
+ }
+
+ keysRegistered += listener.keys().size();
+ }
+
+ public synchronized void fireEvent(RedisCommandType command, RedisKey key) {
+ if (listeners.isEmpty()) {
+ return;
+ }
+
+ List<EventListener> listenerList = listeners.get(key);
+ if (listenerList == null) {
+ return;
+ }
+
+ for (EventListener listener : listenerList) {
+ if (listener.process(command, key) == EventResponse.REMOVE_AND_STOP) {
+ removeListener(listener);
+ break;
+ }
+ }
+ }
+
+ /**
+ * The total number of keys registered by all listeners (includes duplicates).
+ */
+ @VisibleForTesting
+ public int size() {
+ return keysRegistered;
+ }
+
+ @Override
+ public synchronized void afterBucketRemoved(int bucketId, Iterable<?> keys) {
+ Set<EventListener> resubmittingList = new HashSet<>();
+ for (Map.Entry<RedisKey, List<EventListener>> entry : listeners.entrySet()) {
+ if (entry.getKey().getBucketId() == bucketId) {
+ resubmittingList.addAll(entry.getValue());
+ }
+ }
+
+ resubmittingList.forEach(x -> {
+ x.resubmitCommand();
+ removeListener(x);
+ });
+ }
+
+ private synchronized void removeListener(EventListener listener) {
+ boolean listenerRemoved = false;
+ for (RedisKey key : listener.keys()) {
+ List<EventListener> listenerList = listeners.get(key);
+ listenerRemoved |= listenerList.remove(listener);
+ if (listenerList.isEmpty()) {
+ listeners.remove(key);
+ }
+ }
+
+ if (listenerRemoved) {
+ keysRegistered -= listener.keys().size();
+ if (listener.getTimeout() > 0) {
+ timerTasks.remove(listener).cancel();
+ }
+ }
+ }
+
+ private void scheduleTimeout(EventListener listener, long timeout) {
+ TimerTask task = new TimerTask() {
Review comment:
Another nice thing about using ScheduledThreadPoolExecutor is you can just have the EventListener implement Runnable. You will need to store the Future returned from calling submit somewhere so that you can call cancel on that Future if the timeout is hit.
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventDistributor.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Timer;
+import java.util.TimerTask;
+
+import org.apache.logging.log4j.Logger;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.cache.partition.PartitionListenerAdapter;
+import org.apache.geode.logging.internal.log4j.api.LogService;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributor extends PartitionListenerAdapter {
+
+ private static final Logger logger = LogService.getLogger();
+ private final Map<RedisKey, List<EventListener>> listeners = new HashMap<>();
+ private final Map<EventListener, TimerTask> timerTasks = new HashMap<>();
+ private final Timer timer = new Timer("GeodeForRedisEventTimer", true);
+ private int keysRegistered = 0;
+
+ public synchronized void registerListener(EventListener listener) {
+ for (RedisKey key : listener.keys()) {
+ listeners.computeIfAbsent(key, k -> new ArrayList<>()).add(listener);
+ }
+
+ if (listener.getTimeout() != 0) {
+ scheduleTimeout(listener, listener.getTimeout());
+ }
+
+ keysRegistered += listener.keys().size();
+ }
+
+ public synchronized void fireEvent(RedisCommandType command, RedisKey key) {
Review comment:
Having this method synchronized seems like it could hurt performance of conncurrent ops on different keys. They would all fight over this lock.
It seems like your top level "listeners" map could be a ConcurrentHashMap and the nested ArrayList could be one of the Concurrent queue classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r824375289
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/commands/executor/list/BLPopExecutor.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.geode.redis.internal.RedisConstants;
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.executor.CommandExecutor;
+import org.apache.geode.redis.internal.commands.executor.RedisResponse;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.data.RedisList;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BLPopExecutor implements CommandExecutor {
+
+ @Override
+ public RedisResponse executeCommand(Command command, ExecutionHandlerContext context) {
+ List<byte[]> arguments = command.getCommandArguments();
+ int keyCount = arguments.size() - 1;
+ double timeoutMillis;
+ try {
+ timeoutMillis = Coder.bytesToDouble(arguments.get(keyCount)) * 1000;
+ } catch (NumberFormatException e) {
+ return RedisResponse.error(RedisConstants.ERROR_TIMEOUT_INVALID);
+ }
+
+ List<RedisKey> keys = new ArrayList<>(keyCount);
+ for (int i = 0; i < keyCount; i++) {
+ keys.add(new RedisKey(arguments.get(i)));
+ }
+
+ List<byte[]> popped = context.lockedExecute(keys.get(0), keys,
+ () -> RedisList.blpop(context, keys, (int) timeoutMillis));
Review comment:
Hopefully I've fixed all the timeout issues now. I'm keeping timeout as a double throughout, except internally to `BlockingCommandListener` where it is converted to nanoseconds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r824373125
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventResponse.java
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+/**
+ * Response returned by {@link EventListener#process(RedisCommandType, RedisKey)}
+ */
+public enum EventResponse {
Review comment:
I've changed the comments a bit. Again, this may change once we implement keyspace events.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] dschneider-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
dschneider-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r823022806
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/commands/executor/list/BLPopExecutor.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.geode.redis.internal.RedisConstants;
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.executor.CommandExecutor;
+import org.apache.geode.redis.internal.commands.executor.RedisResponse;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.data.RedisList;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BLPopExecutor implements CommandExecutor {
+
+ @Override
+ public RedisResponse executeCommand(Command command, ExecutionHandlerContext context) {
+ List<byte[]> arguments = command.getCommandArguments();
+ int keyCount = arguments.size() - 1;
+ double timeoutMillis;
+ try {
+ timeoutMillis = Coder.bytesToDouble(arguments.get(keyCount)) * 1000;
+ } catch (NumberFormatException e) {
+ return RedisResponse.error(RedisConstants.ERROR_TIMEOUT_INVALID);
+ }
+
+ List<RedisKey> keys = new ArrayList<>(keyCount);
+ for (int i = 0; i < keyCount; i++) {
+ keys.add(new RedisKey(arguments.get(i)));
+ }
+
+ List<byte[]> popped = context.lockedExecute(keys.get(0), keys,
+ () -> RedisList.blpop(context, keys, (int) timeoutMillis));
Review comment:
my understanding is that the timeout comes to us in double whose units is seconds. But couldn't then be a fraction of a second (for example 0.005) seconds? If so why do we cast to an "int" here? Also why do we convert it to millis? Shouldn't we do nanos in case they wanted to wait for less than a millisecond? Do we need to be careful here to not allow a non-zero timeout to become zero (which waits forever)? Maybe this should all be done at a lower level (the conversion of time units).
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/data/RedisList.java
##########
@@ -154,6 +159,29 @@ public long lpush(List<byte[]> elementsToAdd, Region<RedisKey, RedisData> region
return popped;
}
+ public static List<byte[]> blpop(ExecutionHandlerContext context, List<RedisKey> keys,
+ int timeout) {
+ Region<RedisKey, RedisData> region = context.getRegion();
+ for (RedisKey key : keys) {
+ if (region.containsKey(key)) {
Review comment:
instead of interacting with the region twice (containsKey and getTypedRedisData) could you just add a flavor of getTypedRedisData (maybe getExistingTypedRedisData) that just returns null if it does not exist?
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/netty/ExecutionHandlerContext.java
##########
@@ -232,7 +236,14 @@ public void channelInactive(ChannelHandlerContext ctx) {
}
}
- private void executeCommand(Command command) throws Exception {
+ public void resubmitCommand(Command command) {
+ ChannelHandlerContext ctx =
+ channel.pipeline().context(ByteToCommandDecoder.class.getSimpleName());
+ ctx.fireChannelRead(command);
Review comment:
do you know for sure that fireChannelRead is async? That is, it just hands command off to some queue and does not end up processing command in this thread.
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/BlockingCommandListener.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListener implements EventListener {
+
+ private final ExecutionHandlerContext context;
+ private final RedisCommandType command;
+ private final List<RedisKey> keys;
+ private final byte[][] commandOptions;
+ private final long timeout;
+ private final long timeAdded;
+ private Runnable cleanupTask;
+
+ /**
+ * Constructor to create an instance of a BlockingCommandListener in response to a blocking
+ * command. When receiving a relevant event, blocking commands simply resubmit the command
+ * into the Netty pipeline.
+ *
+ * @param context the associated ExecutionHandlerContext
+ * @param command the blocking command
+ * @param timeout the timeout for the command to block in milliseconds
+ * @param keys the list of keys the command is interested in
+ * @param commandOptions any additional options (other than the keys and timeout) required to
+ * resubmit the command.
+ */
+ public BlockingCommandListener(ExecutionHandlerContext context, RedisCommandType command,
+ long timeout, List<RedisKey> keys, byte[]... commandOptions) {
+ this.context = context;
+ this.command = command;
+ this.timeout = timeout;
+ this.keys = Collections.unmodifiableList(keys);
+ this.commandOptions = commandOptions;
+ timeAdded = System.currentTimeMillis();
+ }
+
+ @Override
+ public List<RedisKey> keys() {
+ return keys;
+ }
+
+ @Override
+ public EventResponse process(RedisCommandType commandType, RedisKey key) {
+ if (!keys.contains(key)) {
+ return EventResponse.CONTINUE;
+ }
+
+ resubmitCommand();
+ return EventResponse.REMOVE_AND_STOP;
+ }
+
+ @Override
+ public void resubmitCommand() {
+ List<byte[]> byteArgs = new ArrayList<>(keys.size() + 1);
+ byteArgs.add(command.name().getBytes());
+
+ keys.forEach(x -> byteArgs.add(x.toBytes()));
+ byteArgs.addAll(Arrays.asList(commandOptions));
+
+ // Recalculate the timeout since we've already been waiting
+ long adjustedTimeout = timeout;
+ if (adjustedTimeout > 0) {
+ adjustedTimeout = (adjustedTimeout - (System.currentTimeMillis() - timeAdded)) / 1000;
+ adjustedTimeout = Math.max(1, adjustedTimeout);
Review comment:
It seems wrong here to say if we are going to wait we need to wait at least a second. Since the timeout is expressed as a double it could be a fraction of a second. It seems like this code should support an adjustedTimeout that is < 1.
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventListener.java
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.List;
+
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+/**
+ * Interface intended to be implemented in order to receive Redis events. Specifically this would be
+ * keyspace events or blocking commands. EventListeners are registered with the
+ * {@link EventDistributor}.
+ */
+public interface EventListener {
+
+ /**
+ * Receive and process an event. This method should execute very quickly. The return value
+ * determines additional process steps for the given event.
+ *
+ * @param commandType the command triggering the event
+ * @param key the key triggering the event
+ * @return response determining subsequent processing steps
+ */
+ EventResponse process(RedisCommandType commandType, RedisKey key);
+
+ /**
+ * Return the list of keys this listener is interested in.
+ */
+ List<RedisKey> keys();
+
+ /**
+ * Method to resubmit a command if appropriate. This is only relevant for listeners that process
+ * events for blocking commands. Listeners that handle keyspace event notification will not use
+ * this.
+ */
+ default void resubmitCommand() {};
+
+ /**
+ * Retrieve the timeout for this listener. The default is no timeout.
+ */
+ default long getTimeout() {
+ return 0;
+ }
+
+ /**
+ * Set a runnable that is responsible for any cleanup when this listener is removed.
+ */
+ default void setCleanupTask(Runnable r) {}
Review comment:
All of these "defaults" used here seem like a possible indicator that we should have two interfaces. default can be very helpful when trying to maintain backwards compatibility but since this is a brand new interface I'm not sure we should use it
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventResponse.java
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+/**
+ * Response returned by {@link EventListener#process(RedisCommandType, RedisKey)}
+ */
+public enum EventResponse {
Review comment:
This enum also seems like an indicator that we should have two listener interfaces. It seems odd to have an enum who comments say if you are a keyspace event listener then use CONTINUE; if a blocking event listener then use REMOVE_AND_STOP.
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/BlockingCommandListener.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListener implements EventListener {
+
+ private final ExecutionHandlerContext context;
+ private final RedisCommandType command;
+ private final List<RedisKey> keys;
+ private final byte[][] commandOptions;
+ private final long timeout;
+ private final long timeAdded;
+ private Runnable cleanupTask;
+
+ /**
+ * Constructor to create an instance of a BlockingCommandListener in response to a blocking
+ * command. When receiving a relevant event, blocking commands simply resubmit the command
+ * into the Netty pipeline.
+ *
+ * @param context the associated ExecutionHandlerContext
+ * @param command the blocking command
+ * @param timeout the timeout for the command to block in milliseconds
+ * @param keys the list of keys the command is interested in
+ * @param commandOptions any additional options (other than the keys and timeout) required to
+ * resubmit the command.
+ */
+ public BlockingCommandListener(ExecutionHandlerContext context, RedisCommandType command,
+ long timeout, List<RedisKey> keys, byte[]... commandOptions) {
+ this.context = context;
+ this.command = command;
+ this.timeout = timeout;
+ this.keys = Collections.unmodifiableList(keys);
+ this.commandOptions = commandOptions;
+ timeAdded = System.currentTimeMillis();
+ }
+
+ @Override
+ public List<RedisKey> keys() {
+ return keys;
+ }
+
+ @Override
+ public EventResponse process(RedisCommandType commandType, RedisKey key) {
+ if (!keys.contains(key)) {
+ return EventResponse.CONTINUE;
+ }
+
+ resubmitCommand();
+ return EventResponse.REMOVE_AND_STOP;
+ }
+
+ @Override
+ public void resubmitCommand() {
+ List<byte[]> byteArgs = new ArrayList<>(keys.size() + 1);
+ byteArgs.add(command.name().getBytes());
+
+ keys.forEach(x -> byteArgs.add(x.toBytes()));
+ byteArgs.addAll(Arrays.asList(commandOptions));
+
+ // Recalculate the timeout since we've already been waiting
+ long adjustedTimeout = timeout;
+ if (adjustedTimeout > 0) {
+ adjustedTimeout = (adjustedTimeout - (System.currentTimeMillis() - timeAdded)) / 1000;
+ adjustedTimeout = Math.max(1, adjustedTimeout);
+ }
+
+ // The commands we are currently supporting all have the timeout at the end of the argument
+ // list. Some newer Redis 7 commands (BLMPOP and BZMPOP) have the timeout as the first argument
+ // after the command. We'll need to adjust this once those commands are supported.
+ byteArgs.add(Coder.longToBytes(adjustedTimeout));
+
+ context.resubmitCommand(new Command(command, byteArgs));
Review comment:
Couldn't we just replace in place the Command.commandElems item that contains the timeout? I think we can know the index of the timeout and then just stick a different byte[] in the ArrayList at this index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] dschneider-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
dschneider-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r824943346
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/BlockingCommandListener.java
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.commands.executor.RedisResponse;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListener implements EventListener {
+
+ private final ExecutionHandlerContext context;
+ private final RedisCommandType command;
+ private final List<RedisKey> keys;
+ private final List<byte[]> commandArgs;
+ private final long timeoutNanos;
+ private final long timeSubmitted;
+ private Runnable cleanupTask;
+
+ /**
+ * Constructor to create an instance of a BlockingCommandListener in response to a blocking
+ * command. When receiving a relevant event, blocking commands simply resubmit the command
+ * into the Netty pipeline.
+ *
+ * @param context the associated ExecutionHandlerContext
+ * @param command the blocking command associated with this listener
+ * @param keys the list of keys the command is interested in
+ * @param timeoutSeconds the timeout for the command to block in seconds
+ * @param commandArgs all arguments to the command which are used for resubmission
+ */
+ public BlockingCommandListener(ExecutionHandlerContext context, RedisCommandType command,
+ List<RedisKey> keys, double timeoutSeconds, List<byte[]> commandArgs) {
+ this.context = context;
+ this.command = command;
Review comment:
I was confused by this being named "command". I thought it was the actual Command instance. "commandType" would be better.
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/BlockingCommandListener.java
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.commands.executor.RedisResponse;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListener implements EventListener {
+
+ private final ExecutionHandlerContext context;
+ private final RedisCommandType command;
+ private final List<RedisKey> keys;
+ private final List<byte[]> commandArgs;
+ private final long timeoutNanos;
+ private final long timeSubmitted;
+ private Runnable cleanupTask;
+
+ /**
+ * Constructor to create an instance of a BlockingCommandListener in response to a blocking
+ * command. When receiving a relevant event, blocking commands simply resubmit the command
+ * into the Netty pipeline.
+ *
+ * @param context the associated ExecutionHandlerContext
+ * @param command the blocking command associated with this listener
+ * @param keys the list of keys the command is interested in
+ * @param timeoutSeconds the timeout for the command to block in seconds
+ * @param commandArgs all arguments to the command which are used for resubmission
+ */
+ public BlockingCommandListener(ExecutionHandlerContext context, RedisCommandType command,
+ List<RedisKey> keys, double timeoutSeconds, List<byte[]> commandArgs) {
+ this.context = context;
+ this.command = command;
+ this.timeoutNanos = (long) (timeoutSeconds * 1e9);
+ this.keys = Collections.unmodifiableList(keys);
+ this.commandArgs = commandArgs;
+ timeSubmitted = System.nanoTime();
+ }
+
+ @Override
+ public List<RedisKey> keys() {
+ return keys;
+ }
+
+ @Override
+ public EventResponse process(RedisCommandType commandType, RedisKey key) {
+ if (!keys.contains(key)) {
+ return EventResponse.CONTINUE;
+ }
+
+ resubmitCommand();
+ return EventResponse.REMOVE_AND_STOP;
+ }
+
+ @Override
+ public void resubmitCommand() {
+ // Recalculate the timeout since we've already been waiting
+ double adjustedTimeoutSeconds = 0;
+ if (timeoutNanos > 0) {
+ long adjustedTimeoutNanos = timeoutNanos - (System.nanoTime() - timeSubmitted);
+ adjustedTimeoutNanos = Math.max(1, adjustedTimeoutNanos);
+ adjustedTimeoutSeconds = ((double) adjustedTimeoutNanos) / 1e9;
+ }
+
+ // The commands we are currently supporting all have the timeout at the end of the argument
+ // list. Some newer Redis 7 commands (BLMPOP and BZMPOP) have the timeout as the first argument
+ // after the command. We'll need to adjust this once those commands are supported.
+ commandArgs.set(commandArgs.size() - 1, Coder.doubleToBytes(adjustedTimeoutSeconds));
+
+ context.resubmitCommand(new Command(command, commandArgs));
Review comment:
why don't we just pass down to blocking commands a reference to the Command instance? That way we don't need to keep recreating its commandArgs. All we need to do is ask that Command instance to update one of its args (at a specific index) with a new byte array.
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventDistributor.java
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.cache.partition.PartitionListenerAdapter;
+import org.apache.geode.logging.internal.executors.LoggingThreadFactory;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributor extends PartitionListenerAdapter {
+
+ private final Map<RedisKey, Queue<EventListener>> listeners = new ConcurrentHashMap<>();
+
+ private final ScheduledThreadPoolExecutor timerExecutor =
+ new ScheduledThreadPoolExecutor(1,
+ new LoggingThreadFactory("GeodeForRedisEventTimer-", true));
+
+ private int keysRegistered = 0;
Review comment:
since this is only for testing I think it would be better to just have the method that returns this value to tests to instead iterator over the data structures asking each listener how many keys it has. I don't think the tests care how performant asking for this value is. And that way we do not complicate the EventDistributor implementation with having to update keysRegistered in a thread safe way. I was reviewing the synchronization on this class and a couple of times I considered a change that then would be a problem because this field would not get updated in a thread safe way
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventDistributor.java
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.cache.partition.PartitionListenerAdapter;
+import org.apache.geode.logging.internal.executors.LoggingThreadFactory;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributor extends PartitionListenerAdapter {
+
+ private final Map<RedisKey, Queue<EventListener>> listeners = new ConcurrentHashMap<>();
+
+ private final ScheduledThreadPoolExecutor timerExecutor =
+ new ScheduledThreadPoolExecutor(1,
+ new LoggingThreadFactory("GeodeForRedisEventTimer-", true));
+
+ private int keysRegistered = 0;
+
+ public EventDistributor() {
+ timerExecutor.setRemoveOnCancelPolicy(true);
+ }
+
+ public synchronized void registerListener(EventListener listener) {
+ for (RedisKey key : listener.keys()) {
+ listeners.computeIfAbsent(key, k -> new LinkedBlockingQueue<>()).add(listener);
+ }
+
+ if (listener.getTimeout() != 0) {
+ scheduleTimeout(listener, listener.getTimeout());
+ }
+
+ keysRegistered += listener.keys().size();
+ }
+
+ public void fireEvent(RedisCommandType command, RedisKey key) {
+ Queue<EventListener> listenerList = listeners.get(key);
+ if (listenerList == null) {
+ return;
+ }
+
+ for (EventListener listener : listenerList) {
+ if (listener.process(command, key) == EventResponse.REMOVE_AND_STOP) {
+ removeListener(listener);
+ break;
+ }
+ }
+ }
+
+ /**
+ * The total number of keys registered by all listeners (includes duplicates).
+ */
+ @VisibleForTesting
+ public int size() {
Review comment:
this should probably be named "getRegisteredKeys" since it is only for testing. "size" makes it more likely to be used by product code.
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventDistributor.java
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.cache.partition.PartitionListenerAdapter;
+import org.apache.geode.logging.internal.executors.LoggingThreadFactory;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributor extends PartitionListenerAdapter {
+
+ private final Map<RedisKey, Queue<EventListener>> listeners = new ConcurrentHashMap<>();
+
+ private final ScheduledThreadPoolExecutor timerExecutor =
+ new ScheduledThreadPoolExecutor(1,
+ new LoggingThreadFactory("GeodeForRedisEventTimer-", true));
+
+ private int keysRegistered = 0;
+
+ public EventDistributor() {
+ timerExecutor.setRemoveOnCancelPolicy(true);
+ }
+
+ public synchronized void registerListener(EventListener listener) {
Review comment:
I don't think this method needs to be synchronized. I think the only concurrency issue you have with these data structures is a race with removeListener when it completely removes the queue because it thought it was empty. If this method adds to the queue right after that method checks isEmpty then this thread will add to a queue that is not longer in the map. So you can protect that by narrowing you sync to just be the putIfAbsent+add call.
The other concurrency problem you have is making the listener itself thread safe. Basically you have possible races with one thread calling process on it and another calling scheduleTimeout or cleanup. I think you should allow the listener to be responsible for making sure that if it has already been processed then it will not be scheduled. I think you do this by adding a scheduleTimeout method on the listener that takes a ScheduledExecutorService and takes care of calling schedule and remembering the ScheduleFuture and it could only do this if the listener has not already been processed or closed. It is likely that the listener will use synchronization to make it thread safe but now it is on the listener instead of the EventDistributor singleton.
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventDistributor.java
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.cache.partition.PartitionListenerAdapter;
+import org.apache.geode.logging.internal.executors.LoggingThreadFactory;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributor extends PartitionListenerAdapter {
+
+ private final Map<RedisKey, Queue<EventListener>> listeners = new ConcurrentHashMap<>();
+
+ private final ScheduledThreadPoolExecutor timerExecutor =
+ new ScheduledThreadPoolExecutor(1,
+ new LoggingThreadFactory("GeodeForRedisEventTimer-", true));
+
+ private int keysRegistered = 0;
+
+ public EventDistributor() {
+ timerExecutor.setRemoveOnCancelPolicy(true);
+ }
+
+ public synchronized void registerListener(EventListener listener) {
+ for (RedisKey key : listener.keys()) {
+ listeners.computeIfAbsent(key, k -> new LinkedBlockingQueue<>()).add(listener);
Review comment:
should you use JavaWorkarounds.computeIfAbsent to make this call?
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventDistributor.java
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.cache.partition.PartitionListenerAdapter;
+import org.apache.geode.logging.internal.executors.LoggingThreadFactory;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributor extends PartitionListenerAdapter {
+
+ private final Map<RedisKey, Queue<EventListener>> listeners = new ConcurrentHashMap<>();
+
+ private final ScheduledThreadPoolExecutor timerExecutor =
+ new ScheduledThreadPoolExecutor(1,
+ new LoggingThreadFactory("GeodeForRedisEventTimer-", true));
+
+ private int keysRegistered = 0;
+
+ public EventDistributor() {
+ timerExecutor.setRemoveOnCancelPolicy(true);
+ }
+
+ public synchronized void registerListener(EventListener listener) {
+ for (RedisKey key : listener.keys()) {
+ listeners.computeIfAbsent(key, k -> new LinkedBlockingQueue<>()).add(listener);
+ }
+
+ if (listener.getTimeout() != 0) {
+ scheduleTimeout(listener, listener.getTimeout());
+ }
+
+ keysRegistered += listener.keys().size();
+ }
+
+ public void fireEvent(RedisCommandType command, RedisKey key) {
+ Queue<EventListener> listenerList = listeners.get(key);
+ if (listenerList == null) {
+ return;
+ }
+
+ for (EventListener listener : listenerList) {
+ if (listener.process(command, key) == EventResponse.REMOVE_AND_STOP) {
+ removeListener(listener);
+ break;
+ }
+ }
+ }
+
+ /**
+ * The total number of keys registered by all listeners (includes duplicates).
+ */
+ @VisibleForTesting
+ public int size() {
+ return keysRegistered;
+ }
+
+ @Override
+ public void afterBucketRemoved(int bucketId, Iterable<?> keys) {
+ Set<EventListener> resubmittingList = new HashSet<>();
+ for (Map.Entry<RedisKey, Queue<EventListener>> entry : listeners.entrySet()) {
+ if (entry.getKey().getBucketId() == bucketId) {
+ resubmittingList.addAll(entry.getValue());
+ }
+ }
+
+ resubmittingList.forEach(x -> {
+ x.resubmitCommand();
+ removeListener(x);
+ });
+ }
+
+ private synchronized void removeListener(EventListener listener) {
+ boolean listenerRemoved = false;
+ for (RedisKey key : listener.keys()) {
+ Queue<EventListener> listenerList = listeners.get(key);
+ if (listenerList == null) {
+ continue;
+ }
+ listenerRemoved |= listenerList.remove(listener);
+ if (listenerList.isEmpty()) {
Review comment:
I think the only concurrency issue you have is these two lines (isEmpty and remove) and the add call in registerListener. If you got rid of these two lines then you would have no concurrency issues but you would leave empty queues in the map that may never be used again. But if you do this listeners.remove(key) call then you have an issue. The way I addressed this in the pubsub data structures was to create the queue non-empty in registerListener and if it every finds an empty queue it refuses to add to it and instead spins around and reads from the map again until it find a non-empty one or creates its own non-empty one. To make this thread safe you would create you own queue class whose add and remove inc and dec an atomic. The constructor of that class would always start with one element so the atomic would start at 1. If it ever goes to 0 that sticks.
This gets complicated so for now maybe you should keep some synchronization. You can make the sync issue more clear by narrowing your sync to be around just the putIfAbsent+add call and around this isEmpty+remove calls. Also you only need to do the sync(isEmpty+remove) calls if listenerList.remove returned true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r827965981
##########
File path: geode-for-redis/src/integrationTest/java/org/apache/geode/redis/internal/commands/executor/list/AbstractBLPopIntegrationTest.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import static org.apache.geode.redis.RedisCommandArgumentsTestHelper.assertAtLeastNArgs;
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.BIND_ADDRESS;
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.REDIS_CLIENT_TIMEOUT;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.HostAndPort;
+import redis.clients.jedis.JedisCluster;
+import redis.clients.jedis.Protocol;
+
+import org.apache.geode.redis.RedisIntegrationTest;
+import org.apache.geode.redis.internal.RedisConstants;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.junit.rules.ExecutorServiceRule;
+
+public abstract class AbstractBLPopIntegrationTest implements RedisIntegrationTest {
+ private static final String KEY = "key";
+
+ protected JedisCluster jedis;
+
+ public abstract void awaitEventDistributorSize(int size) throws Exception;
+
+ @ClassRule
+ public static ExecutorServiceRule executor = new ExecutorServiceRule();
+
+ @Before
+ public void setUp() {
+ jedis = new JedisCluster(new HostAndPort(BIND_ADDRESS, getPort()), REDIS_CLIENT_TIMEOUT);
+ }
+
+ @After
+ public void tearDown() {
+ flushAll();
+ jedis.close();
+ }
+
+ @Test
+ public void testInvalidArguments_throwErrors() {
+ assertAtLeastNArgs(jedis, Protocol.Command.BLPOP, 2);
+ }
+
+ @Test
+ public void testInvalidTimeout_throwsError() {
+ assertThatThrownBy(() -> jedis.sendCommand("key1", Protocol.Command.BLPOP, "key1",
+ "0.A"))
+ .hasMessage(RedisConstants.ERROR_TIMEOUT_INVALID);
+ }
+
+ @Test
+ public void testKeysInDifferentSlots_throwsError() {
+ assertThatThrownBy(() -> jedis.sendCommand("key1", Protocol.Command.BLPOP, "key1",
+ "key2", "0"))
+ .hasMessage(RedisConstants.ERROR_WRONG_SLOT);
+ }
+
+ @Test
+ public void testNegativeTimeout_throwsError() {
+ assertThatThrownBy(() -> jedis.blpop(-1, "key1"))
+ .hasMessage(RedisConstants.ERROR_NEGATIVE_TIMEOUT);
+ }
+
+ @Test
+ public void testBLPopForNonListKey() {
+ jedis.set("not-a-list", "value");
+
+ assertThatThrownBy(() -> jedis.blpop(0, "not-a-list"))
+ .hasMessage(RedisConstants.ERROR_WRONG_TYPE);
+ }
+
+ @Test
+ public void testBLPopWhenValueExists() {
+ jedis.lpush(KEY, "value1", "value2");
+
+ List<String> result = jedis.blpop(0, KEY);
+
+ assertThat(result).containsExactly(KEY, "value2");
+ assertThat(jedis.lpop(KEY)).isEqualTo("value1");
+ }
+
+ @Test
+ public void testBLPopWithDoesNotError_whenTimeoutHasExponent() {
+ jedis.lpush(KEY, "value1", "value2");
+
+ Object result = jedis.sendCommand(KEY, Protocol.Command.BLPOP, KEY, "1E+3");
+
+ assertThat(result).isNotNull();
+ }
+
+ @Test
+ public void testBLPopWhenValueDoesNotExist() throws Exception {
+ Future<List<String>> future = executor.submit(() -> jedis.blpop(0, KEY));
+
+ awaitEventDistributorSize(1);
+ jedis.lpush(KEY, "value1", "value2");
+
+ assertThat(future.get()).containsExactly(KEY, "value2");
+ assertThat(jedis.lpop(KEY)).isEqualTo("value1");
+ }
+
+ @Test
+ public void testBLPopWhenTimeoutIsExceeded() {
+ int timeout = 10;
+ Future<List<String>> future = executor.submit(() -> jedis.blpop(timeout, KEY));
+ GeodeAwaitility.await().atMost(timeout * 2, TimeUnit.SECONDS)
+ .untilAsserted(() -> assertThat(future.get()).isNull());
+ }
+
+ @Test
+ public void testConcurrentBLPop() throws Exception {
+ int totalElements = 10_000;
+ List<Object> accumulated = Collections.synchronizedList(new ArrayList<>(totalElements + 2));
+ AtomicBoolean running = new AtomicBoolean(true);
+
+ Future<Void> future1 = executor.submit(() -> doBLPop(running, accumulated));
+ Future<Void> future2 = executor.submit(() -> doBLPop(running, accumulated));
+
+ List<String> result = new ArrayList<>();
+ for (int i = 0; i < totalElements; i++) {
+ jedis.lpush(KEY, "value" + i);
+ result.add("value" + i);
+ }
+
+ GeodeAwaitility.await().atMost(Duration.ofSeconds(5))
+ .untilAsserted(() -> assertThat(accumulated.size()).isEqualTo(totalElements));
Review comment:
Fixed.
##########
File path: geode-for-redis/src/integrationTest/java/org/apache/geode/redis/internal/commands/executor/list/AbstractBLPopIntegrationTest.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import static org.apache.geode.redis.RedisCommandArgumentsTestHelper.assertAtLeastNArgs;
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.BIND_ADDRESS;
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.REDIS_CLIENT_TIMEOUT;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.HostAndPort;
+import redis.clients.jedis.JedisCluster;
+import redis.clients.jedis.Protocol;
+
+import org.apache.geode.redis.RedisIntegrationTest;
+import org.apache.geode.redis.internal.RedisConstants;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.junit.rules.ExecutorServiceRule;
+
+public abstract class AbstractBLPopIntegrationTest implements RedisIntegrationTest {
Review comment:
Added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r825630131
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/BlockingCommandListener.java
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.commands.executor.RedisResponse;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListener implements EventListener {
+
+ private final ExecutionHandlerContext context;
+ private final RedisCommandType command;
+ private final List<RedisKey> keys;
+ private final List<byte[]> commandArgs;
+ private final long timeoutNanos;
+ private final long timeSubmitted;
+ private Runnable cleanupTask;
+
+ /**
+ * Constructor to create an instance of a BlockingCommandListener in response to a blocking
+ * command. When receiving a relevant event, blocking commands simply resubmit the command
+ * into the Netty pipeline.
+ *
+ * @param context the associated ExecutionHandlerContext
+ * @param command the blocking command associated with this listener
+ * @param keys the list of keys the command is interested in
+ * @param timeoutSeconds the timeout for the command to block in seconds
+ * @param commandArgs all arguments to the command which are used for resubmission
+ */
+ public BlockingCommandListener(ExecutionHandlerContext context, RedisCommandType command,
+ List<RedisKey> keys, double timeoutSeconds, List<byte[]> commandArgs) {
+ this.context = context;
+ this.command = command;
+ this.timeoutNanos = (long) (timeoutSeconds * 1e9);
+ this.keys = Collections.unmodifiableList(keys);
+ this.commandArgs = commandArgs;
+ timeSubmitted = System.nanoTime();
+ }
+
+ @Override
+ public List<RedisKey> keys() {
+ return keys;
+ }
+
+ @Override
+ public EventResponse process(RedisCommandType commandType, RedisKey key) {
+ if (!keys.contains(key)) {
+ return EventResponse.CONTINUE;
+ }
+
+ resubmitCommand();
+ return EventResponse.REMOVE_AND_STOP;
+ }
+
+ @Override
+ public void resubmitCommand() {
+ // Recalculate the timeout since we've already been waiting
+ double adjustedTimeoutSeconds = 0;
+ if (timeoutNanos > 0) {
+ long adjustedTimeoutNanos = timeoutNanos - (System.nanoTime() - timeSubmitted);
Review comment:
I think so, since `10 - (9 - 6) != (10 - 9) - 6`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal merged pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal merged pull request #7408:
URL: https://github.com/apache/geode/pull/7408
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r824371418
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/commands/executor/list/BLPopExecutor.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.geode.redis.internal.RedisConstants;
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.executor.CommandExecutor;
+import org.apache.geode.redis.internal.commands.executor.RedisResponse;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.data.RedisList;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BLPopExecutor implements CommandExecutor {
+
+ @Override
+ public RedisResponse executeCommand(Command command, ExecutionHandlerContext context) {
+ List<byte[]> arguments = command.getCommandArguments();
+ int keyCount = arguments.size() - 1;
+ double timeoutMillis;
+ try {
+ timeoutMillis = Coder.bytesToDouble(arguments.get(keyCount)) * 1000;
+ } catch (NumberFormatException e) {
+ return RedisResponse.error(RedisConstants.ERROR_TIMEOUT_INVALID);
+ }
+
+ List<RedisKey> keys = new ArrayList<>(keyCount);
+ for (int i = 0; i < keyCount; i++) {
+ keys.add(new RedisKey(arguments.get(i)));
+ }
+
+ List<byte[]> popped = context.lockedExecute(keys.get(0), keys,
+ () -> RedisList.blpop(context, keys, (int) timeoutMillis));
Review comment:
Hopefully I've fixed all the timeout issues now. I'm keeping timeout as a double throughout, except internally to `BlockingCommandListener` where it is converted to nanoseconds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r824375457
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventDistributor.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.cache.partition.PartitionListenerAdapter;
+import org.apache.geode.logging.internal.executors.LoggingThreadFactory;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributor extends PartitionListenerAdapter {
+
+ private final Map<RedisKey, Queue<EventListener>> listeners = new ConcurrentHashMap<>();
+
+ private final ScheduledThreadPoolExecutor timerExecutor =
+ new ScheduledThreadPoolExecutor(1,
+ new LoggingThreadFactory("GeodeForRedisEventTimer-", true));
+
+ private int keysRegistered = 0;
+
+ public EventDistributor() {
+ timerExecutor.setRemoveOnCancelPolicy(true);
+ }
+
+ public synchronized void registerListener(EventListener listener) {
+ for (RedisKey key : listener.keys()) {
+ listeners.computeIfAbsent(key, k -> new LinkedBlockingQueue<>()).add(listener);
+ }
+
+ if (listener.getTimeout() != 0) {
+ scheduleTimeout(listener, listener.getTimeout());
+ }
+
+ keysRegistered += listener.keys().size();
+ }
+
+ public void fireEvent(RedisCommandType command, RedisKey key) {
+ Queue<EventListener> listenerList = listeners.get(key);
+ if (listenerList == null) {
+ return;
+ }
+
+ for (EventListener listener : listenerList) {
+ if (listener.process(command, key) == EventResponse.REMOVE_AND_STOP) {
+ removeListener(listener);
+ break;
+ }
+ }
+ }
+
+ /**
+ * The total number of keys registered by all listeners (includes duplicates).
+ */
+ @VisibleForTesting
+ public int size() {
+ return keysRegistered;
+ }
+
+ @Override
+ public void afterBucketRemoved(int bucketId, Iterable<?> keys) {
+ Set<EventListener> resubmittingList = new HashSet<>();
+ for (Map.Entry<RedisKey, Queue<EventListener>> entry : listeners.entrySet()) {
+ if (entry.getKey().getBucketId() == bucketId) {
+ resubmittingList.addAll(entry.getValue());
+ }
+ }
+
+ resubmittingList.forEach(x -> {
+ x.resubmitCommand();
+ removeListener(x);
+ });
+ }
+
+ private synchronized void removeListener(EventListener listener) {
+ boolean listenerRemoved = false;
+ for (RedisKey key : listener.keys()) {
+ Queue<EventListener> listenerList = listeners.get(key);
+ if (listenerList == null) {
+ continue;
+ }
+ listenerRemoved |= listenerList.remove(listener);
+ if (listenerList.isEmpty()) {
+ listeners.remove(key);
+ }
+ }
+
+ if (listenerRemoved) {
+ keysRegistered -= listener.keys().size();
+ listener.cleanup();
+ }
+ }
+
+ private void scheduleTimeout(EventListener listener, long timeout) {
+ ScheduledFuture<?> scheduledTask =
+ timerExecutor.schedule(() -> removeListener(listener), timeout, TimeUnit.MILLISECONDS);
+ listener.setCleanupTask(() -> scheduledTask.cancel(false));
+ }
Review comment:
Nice. Also fixed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r824371790
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/BlockingCommandListener.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListener implements EventListener {
+
+ private final ExecutionHandlerContext context;
+ private final RedisCommandType command;
+ private final List<RedisKey> keys;
+ private final byte[][] commandOptions;
+ private final long timeout;
+ private final long timeAdded;
+ private Runnable cleanupTask;
+
+ /**
+ * Constructor to create an instance of a BlockingCommandListener in response to a blocking
+ * command. When receiving a relevant event, blocking commands simply resubmit the command
+ * into the Netty pipeline.
+ *
+ * @param context the associated ExecutionHandlerContext
+ * @param command the blocking command
+ * @param timeout the timeout for the command to block in milliseconds
+ * @param keys the list of keys the command is interested in
+ * @param commandOptions any additional options (other than the keys and timeout) required to
+ * resubmit the command.
+ */
+ public BlockingCommandListener(ExecutionHandlerContext context, RedisCommandType command,
+ long timeout, List<RedisKey> keys, byte[]... commandOptions) {
+ this.context = context;
+ this.command = command;
+ this.timeout = timeout;
+ this.keys = Collections.unmodifiableList(keys);
+ this.commandOptions = commandOptions;
+ timeAdded = System.currentTimeMillis();
+ }
+
+ @Override
+ public List<RedisKey> keys() {
+ return keys;
+ }
+
+ @Override
+ public EventResponse process(RedisCommandType commandType, RedisKey key) {
+ if (!keys.contains(key)) {
+ return EventResponse.CONTINUE;
+ }
+
+ resubmitCommand();
+ return EventResponse.REMOVE_AND_STOP;
+ }
+
+ @Override
+ public void resubmitCommand() {
+ List<byte[]> byteArgs = new ArrayList<>(keys.size() + 1);
+ byteArgs.add(command.name().getBytes());
+
+ keys.forEach(x -> byteArgs.add(x.toBytes()));
+ byteArgs.addAll(Arrays.asList(commandOptions));
+
+ // Recalculate the timeout since we've already been waiting
+ long adjustedTimeout = timeout;
+ if (adjustedTimeout > 0) {
+ adjustedTimeout = (adjustedTimeout - (System.currentTimeMillis() - timeAdded)) / 1000;
+ adjustedTimeout = Math.max(1, adjustedTimeout);
+ }
+
+ // The commands we are currently supporting all have the timeout at the end of the argument
+ // list. Some newer Redis 7 commands (BLMPOP and BZMPOP) have the timeout as the first argument
+ // after the command. We'll need to adjust this once those commands are supported.
+ byteArgs.add(Coder.longToBytes(adjustedTimeout));
+
+ context.resubmitCommand(new Command(command, byteArgs));
Review comment:
The problem is that the `Command` instance is never passed down, only the `ExecutionHandlerContext` which does not have a reference to the `Command`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r827964327
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventListener.java
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.List;
+import java.util.concurrent.ScheduledExecutorService;
+
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+/**
+ * Interface intended to be implemented in order to receive Redis events. Specifically this would be
+ * keyspace events or blocking commands. EventListeners are registered with the
+ * {@link EventDistributor}.
+ */
+public interface EventListener {
+
+ /**
+ * Receive and process an event. This method should execute very quickly. The return value
+ * determines additional process steps for the given event.
+ *
+ * @param commandType the command triggering the event
+ * @param key the key triggering the event
+ * @return response determining subsequent processing steps
+ */
+ EventResponse process(RedisCommandType commandType, RedisKey key);
+
+ /**
+ * Return the list of keys this listener is interested in.
+ */
+ List<RedisKey> keys();
+
+ /**
+ * Method to resubmit a command if appropriate. This is only relevant for listeners that process
+ * events for blocking commands. Listeners that handle keyspace event notification will not use
+ * this.
+ */
+ void resubmitCommand();
+
+ /**
+ * Retrieve the timeout in nanoseconds for this listener
+ */
+ // long getTimeout();
+
+ void scheduleTimeout(ScheduledExecutorService executor, EventDistributor distributor);
Review comment:
Fixed
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/commands/executor/list/BLPopExecutor.java
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import static org.apache.geode.redis.internal.RedisConstants.ERROR_NEGATIVE_TIMEOUT;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.geode.redis.internal.RedisConstants;
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.executor.CommandExecutor;
+import org.apache.geode.redis.internal.commands.executor.RedisResponse;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.data.RedisList;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BLPopExecutor implements CommandExecutor {
+
+ @Override
+ public RedisResponse executeCommand(Command command, ExecutionHandlerContext context) {
+ List<byte[]> arguments = command.getCommandArguments();
+ double timeoutSeconds;
+ try {
+ timeoutSeconds = Coder.bytesToDouble(arguments.get(arguments.size() - 1));
+ } catch (NumberFormatException e) {
+ return RedisResponse.error(RedisConstants.ERROR_TIMEOUT_INVALID);
+ }
+
+ if (timeoutSeconds < 0) {
+ return RedisResponse.error(ERROR_NEGATIVE_TIMEOUT);
+ }
+
+ int keyCount = arguments.size() - 1;
+ List<RedisKey> keys = new ArrayList<>(keyCount);
+ for (int i = 0; i < keyCount; i++) {
+ keys.add(new RedisKey(arguments.get(i)));
+ }
+
+ List<byte[]> popped = context.lockedExecute(keys.get(0), keys,
+ () -> RedisList.blpop(context, command, keys, timeoutSeconds));
Review comment:
Excellent catch!
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/BlockingCommandListener.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.commands.executor.RedisResponse;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListener implements EventListener {
+
+ private final ExecutionHandlerContext context;
+ private final Command command;
+ private final List<RedisKey> keys;
+ private final double timeoutSeconds;
+ private final long timeSubmitted;
+ private AtomicBoolean active = new AtomicBoolean(true);
Review comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] dschneider-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
dschneider-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r828304833
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/BlockingCommandListener.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.commands.executor.RedisResponse;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListener implements EventListener {
+
+ private final ExecutionHandlerContext context;
+ private final Command command;
+ private final List<RedisKey> keys;
+ private final double timeoutSeconds;
+ private final long timeSubmitted;
+ private final AtomicBoolean active = new AtomicBoolean(true);
+
+ /**
+ * Constructor to create an instance of a BlockingCommandListener in response to a blocking
+ * command. When receiving a relevant event, blocking commands simply resubmit the command
+ * into the Netty pipeline.
+ *
+ * @param context the associated ExecutionHandlerContext
+ * @param command the blocking command associated with this listener
+ * @param keys the list of keys the command is interested in
+ * @param timeoutSeconds the timeout for the command to block in seconds
+ */
+ public BlockingCommandListener(ExecutionHandlerContext context, Command command,
+ List<RedisKey> keys, double timeoutSeconds) {
+ this.context = context;
+ this.command = command;
+ this.timeoutSeconds = timeoutSeconds;
+ this.keys = Collections.unmodifiableList(keys);
+ timeSubmitted = System.nanoTime();
+ }
+
+ @Override
+ public List<RedisKey> keys() {
+ return keys;
+ }
+
+ @Override
+ public EventResponse process(RedisCommandType commandType, RedisKey key) {
+ if (!keys.contains(key)) {
+ return EventResponse.CONTINUE;
+ }
+
+ if (active.compareAndSet(true, false)) {
+ resubmitCommand();
+ }
+ return EventResponse.REMOVE_AND_STOP;
+ }
+
+ @Override
+ public void resubmitCommand() {
+ // Recalculate the timeout since we've already been waiting
+ double adjustedTimeoutSeconds = 0;
+ if (timeoutSeconds > 0.0D) {
+ long timeoutNanos = (long) (timeoutSeconds * 1e9);
+ long adjustedTimeoutNanos = timeoutNanos - (System.nanoTime() - timeSubmitted);
+ adjustedTimeoutNanos = Math.max(1, adjustedTimeoutNanos);
+ adjustedTimeoutSeconds = ((double) adjustedTimeoutNanos) / 1e9;
+ }
+
+ // The commands we are currently supporting all have the timeout at the end of the argument
+ // list. Some newer Redis 7 commands (BLMPOP and BZMPOP) have the timeout as the first argument
+ // after the command. We'll need to adjust this once those commands are supported.
+ List<byte[]> commandArguments = command.getCommandArguments();
+ commandArguments.set(commandArguments.size() - 1, Coder.doubleToBytes(adjustedTimeoutSeconds));
+
+ context.resubmitCommand(command);
+ }
+
+ @Override
+ public void scheduleTimeout(ScheduledExecutorService executor, EventDistributor distributor) {
+ if (timeoutSeconds == 0) {
Review comment:
is it worth also returning early if active is false?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r825630367
##########
File path: geode-for-redis/src/test/java/org/apache/geode/redis/internal/eventing/BlockingCommandListenerTest.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+import org.junit.Test;
+import org.mockito.ArgumentCaptor;
+
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListenerTest {
+
+ @Test
+ public void testTimeoutIsAdjusted() {
+ ExecutionHandlerContext context = mock(ExecutionHandlerContext.class);
+ List<byte[]> commandArgs = Arrays.asList("KEY".getBytes(), "0".getBytes());
Review comment:
Can't be done since `0` is a `char`.
##########
File path: geode-for-redis/src/test/java/org/apache/geode/redis/internal/eventing/BlockingCommandListenerTest.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+import org.junit.Test;
+import org.mockito.ArgumentCaptor;
+
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListenerTest {
+
+ @Test
+ public void testTimeoutIsAdjusted() {
+ ExecutionHandlerContext context = mock(ExecutionHandlerContext.class);
+ List<byte[]> commandArgs = Arrays.asList("KEY".getBytes(), "0".getBytes());
+ BlockingCommandListener listener =
+ new BlockingCommandListener(context, RedisCommandType.BLPOP, Collections.emptyList(),
+ 1.0D,
+ commandArgs);
+
+ listener.resubmitCommand();
+
+ ArgumentCaptor<Command> argumentCaptor = ArgumentCaptor.forClass(Command.class);
+ verify(context, times(1)).resubmitCommand(argumentCaptor.capture());
+
+ double timeout = Coder.bytesToDouble(argumentCaptor.getValue().getCommandArguments().get(0));
+ assertThat(timeout).isLessThan(1.0D);
+ }
+
+ @Test
+ public void testAdjustedTimeoutDoesNotBecomeNegative() {
+ ExecutionHandlerContext context = mock(ExecutionHandlerContext.class);
+ List<byte[]> commandArgs = Arrays.asList("KEY".getBytes(), "0".getBytes());
Review comment:
Ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r824375900
##########
File path: geode-for-redis/src/integrationTest/java/org/apache/geode/redis/internal/commands/executor/list/AbstractBLPopIntegrationTest.java
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.BIND_ADDRESS;
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.REDIS_CLIENT_TIMEOUT;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.HostAndPort;
+import redis.clients.jedis.JedisCluster;
+import redis.clients.jedis.Protocol;
+
+import org.apache.geode.redis.RedisIntegrationTest;
+import org.apache.geode.redis.internal.RedisConstants;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.junit.rules.ExecutorServiceRule;
+
+public abstract class AbstractBLPopIntegrationTest implements RedisIntegrationTest {
+ private static final String KEY = "key";
+
+ protected JedisCluster jedis;
+
+ public abstract void awaitEventDistributorSize(int size) throws Exception;
+
+ @ClassRule
+ public static ExecutorServiceRule executor = new ExecutorServiceRule();
+
+ @Before
+ public void setUp() {
+ jedis = new JedisCluster(new HostAndPort(BIND_ADDRESS, getPort()), REDIS_CLIENT_TIMEOUT);
+ }
+
+ @After
+ public void tearDown() {
+ flushAll();
+ jedis.close();
+ }
+
+ @Test
+ public void testInvalidArguments_throwErrors() {
+ assertThatThrownBy(() -> jedis.sendCommand("key", Protocol.Command.BLPOP))
+ .hasMessageContaining("ERR wrong number of arguments for 'blpop' command");
+ assertThatThrownBy(() -> jedis.sendCommand("key1", Protocol.Command.BLPOP, "key"))
+ .hasMessageContaining("ERR wrong number of arguments for 'blpop' command");
+ }
Review comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r824373663
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/commands/executor/list/BLPopExecutor.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.geode.redis.internal.RedisConstants;
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.executor.CommandExecutor;
+import org.apache.geode.redis.internal.commands.executor.RedisResponse;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.data.RedisList;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BLPopExecutor implements CommandExecutor {
+
+ @Override
+ public RedisResponse executeCommand(Command command, ExecutionHandlerContext context) {
+ List<byte[]> arguments = command.getCommandArguments();
+ int keyCount = arguments.size() - 1;
Review comment:
Yes it is. Further down it is used as the count of keys. I've moved this declaration down and simply use `size() -1` to reference the last element instead of this variable. I think that's clearer even if less succinct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r825628007
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventDistributor.java
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.cache.partition.PartitionListenerAdapter;
+import org.apache.geode.logging.internal.executors.LoggingThreadFactory;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributor extends PartitionListenerAdapter {
+
+ private final Map<RedisKey, Queue<EventListener>> listeners = new ConcurrentHashMap<>();
+
+ private final ScheduledThreadPoolExecutor timerExecutor =
+ new ScheduledThreadPoolExecutor(1,
+ new LoggingThreadFactory("GeodeForRedisEventTimer-", true));
+
+ private int keysRegistered = 0;
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r827965816
##########
File path: geode-for-redis/src/integrationTest/java/org/apache/geode/redis/internal/eventing/EventDistributorTest.java
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.stream.Collectors;
+
+import org.junit.Test;
+
+import org.apache.geode.redis.ConcurrentLoopingThreads;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributorTest {
+
+ public static class TestEventListener implements EventListener {
+ private final List<RedisKey> keys;
+ private int fired = 0;
+
+ public TestEventListener(RedisKey... keys) {
+ this.keys = Arrays.stream(keys).collect(Collectors.toList());
+ }
+
+ public int getFired() {
+ return fired;
+ }
+
+ @Override
+ public EventResponse process(RedisCommandType commandType, RedisKey key) {
+ fired += 1;
+ return EventResponse.REMOVE_AND_STOP;
+ }
+
+ @Override
+ public List<RedisKey> keys() {
+ return keys;
+ }
+
+ @Override
+ public void resubmitCommand() {}
+
+ @Override
+ public void scheduleTimeout(ScheduledExecutorService executor, EventDistributor distributor) {}
+ }
+
+ @Test
+ public void firingEventRemovesListener() {
+ RedisKey keyA = new RedisKey("a".getBytes());
+ RedisKey keyB = new RedisKey("b".getBytes());
+ EventDistributor distributor = new EventDistributor();
+ TestEventListener listener = new TestEventListener(keyA, keyB);
+ distributor.registerListener(listener);
+
+ distributor.fireEvent(null, keyA);
+ assertThat(listener.getFired()).isEqualTo(1);
+ assertThat(distributor.getRegisteredKeys()).isEqualTo(0);
+ }
+
+ @Test
+ public void firingEventRemovesFirstListener_whenMultipleExist() {
+ RedisKey keyA = new RedisKey("a".getBytes());
+ RedisKey keyB = new RedisKey("b".getBytes());
+ EventDistributor distributor = new EventDistributor();
+ TestEventListener listener1 = new TestEventListener(keyA, keyB);
+ TestEventListener listener2 = new TestEventListener(keyA, keyB);
+ distributor.registerListener(listener1);
+ distributor.registerListener(listener2);
+
+ assertThat(distributor.getRegisteredKeys()).isEqualTo(4);
+
+ distributor.fireEvent(null, keyA);
+ assertThat(listener1.getFired()).isEqualTo(1);
+ assertThat(listener2.getFired()).isEqualTo(0);
+ assertThat(distributor.getRegisteredKeys()).isEqualTo(2);
+
+ distributor.fireEvent(null, keyA);
+ assertThat(listener1.getFired()).isEqualTo(1);
+ assertThat(listener2.getFired()).isEqualTo(1);
+ assertThat(distributor.getRegisteredKeys()).isEqualTo(0);
+ }
+
+ @Test
+ public void listenerIsRemovedAfterBucketMoves() {
+ RedisKey keyA = new RedisKey("a".getBytes());
+ RedisKey keyB = new RedisKey("b".getBytes());
+ EventDistributor distributor = new EventDistributor();
+ TestEventListener listener1 = new TestEventListener(keyA);
+ distributor.registerListener(listener1);
+ distributor.registerListener(new TestEventListener(keyB));
+
+ assertThat(distributor.getRegisteredKeys()).isEqualTo(2);
+
+ distributor.afterBucketRemoved(keyA.getBucketId(), null);
+
+ assertThat(distributor.getRegisteredKeys()).isEqualTo(1);
+ }
+
+ @Test
+ public void concurrencyOfManyRegistrationsAndBucketMovementForTheSameKeys() {
+ EventDistributor distributor = new EventDistributor();
+ RedisKey keyA = new RedisKey("keyA".getBytes());
+ RedisKey keyB = new RedisKey("keyB".getBytes());
+ RedisKey keyC = new RedisKey("keyC".getBytes());
+
+ // Should not produce any exceptions
+ new ConcurrentLoopingThreads(10_000,
+ i -> distributor.registerListener(new TestEventListener(keyA, keyB, keyC)),
+ i -> distributor.registerListener(new TestEventListener(keyB, keyC, keyA)),
+ i -> distributor.registerListener(new TestEventListener(keyC, keyA, keyB)),
+ i -> distributor.afterBucketRemoved(keyA.getBucketId(), null),
+ i -> distributor.afterBucketRemoved(keyB.getBucketId(), null),
+ i -> distributor.afterBucketRemoved(keyC.getBucketId(), null),
+ i -> distributor.fireEvent(null, keyA),
+ i -> distributor.fireEvent(null, keyB),
+ i -> distributor.fireEvent(null, keyC))
+ .runInLockstep();
+
+ distributor.fireEvent(null, keyA);
+ distributor.fireEvent(null, keyB);
+ distributor.fireEvent(null, keyC);
+ assertThat(distributor.getRegisteredKeys()).isEqualTo(0);
+ }
+
+ @Test
+ public void concurrencyOfManyRegistrationAndRemovalOfSameListener() {
+ EventDistributor distributor = new EventDistributor();
+ RedisKey keyA = new RedisKey("keyA".getBytes());
+ RedisKey keyB = new RedisKey("keyB".getBytes());
+ RedisKey keyC = new RedisKey("keyC".getBytes());
+ AtomicReference<EventListener> listenerRef1 =
+ new AtomicReference<>(new TestEventListener(keyA, keyB, keyC));
+ AtomicReference<EventListener> listenerRef2 =
+ new AtomicReference<>(new TestEventListener(keyB, keyC, keyA));
+ AtomicReference<EventListener> listenerRef3 =
+ new AtomicReference<>(new TestEventListener(keyC, keyA, keyB));
+
+ // Should not produce any exceptions
+ new ConcurrentLoopingThreads(10_000,
+ i -> distributor.registerListener(listenerRef1.get()),
+ i -> distributor.registerListener(listenerRef2.get()),
+ i -> distributor.registerListener(listenerRef3.get()),
+ i -> distributor.removeListener(listenerRef1.get()),
+ i -> distributor.removeListener(listenerRef2.get()),
+ i -> distributor.removeListener(listenerRef3.get()),
+ i -> distributor.fireEvent(null, keyA),
+ i -> distributor.fireEvent(null, keyB),
+ i -> distributor.fireEvent(null, keyC))
+ .runWithAction(() -> {
+ listenerRef1.set(new TestEventListener(keyA, keyB, keyC));
+ listenerRef2.set(new TestEventListener(keyB, keyC, keyA));
+ listenerRef3.set(new TestEventListener(keyC, keyA, keyB));
+ });
+
+ distributor.fireEvent(null, keyA);
+ distributor.fireEvent(null, keyB);
+ distributor.fireEvent(null, keyC);
+ assertThat(distributor.getRegisteredKeys()).isEqualTo(0);
+ }
+
+ @Test
+ public void ensureNotRegisteringListenerOnQueueJustRemoved() {
Review comment:
That's understandable. I'm not quite sure how to name it but I added a longer comment to try to explain the intent of the test. Perhaps you have a better suggestion for a name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r827964035
##########
File path: geode-for-redis/src/integrationTest/java/org/apache/geode/redis/internal/commands/executor/list/AbstractBLPopIntegrationTest.java
##########
@@ -92,6 +97,15 @@ public void testBLPopWhenValueExists() {
assertThat(jedis.lpop(KEY)).isEqualTo("value1");
}
+ @Test
+ public void testBLPopWithDoesNotError_whenTimeoutHasExponent() {
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] lgtm-com[bot] commented on pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on pull request #7408:
URL: https://github.com/apache/geode/pull/7408#issuecomment-1073145787
This pull request **introduces 1 alert** when merging 2445404f22913de94d75c7f2c947ef48195d7bf5 into d44df3751b45ea8bba5f702350a4428566618ff4 - [view on LGTM.com](https://lgtm.com/projects/g/apache/geode/rev/pr-66f8dedccdad817b9ff9c80533a14f221880dd84)
**new alerts:**
* 1 for Spurious Javadoc @param tags
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r820878240
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventDistributor.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Timer;
+import java.util.TimerTask;
+
+import org.apache.logging.log4j.Logger;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.cache.partition.PartitionListenerAdapter;
+import org.apache.geode.logging.internal.log4j.api.LogService;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributor extends PartitionListenerAdapter {
+
+ private static final Logger logger = LogService.getLogger();
+ private final Map<RedisKey, List<EventListener>> listeners = new HashMap<>();
+ private final Map<EventListener, TimerTask> timerTasks = new HashMap<>();
+ private final Timer timer = new Timer("GeodeForRedisEventTimer", true);
+ private int keysRegistered = 0;
+
+ public synchronized void registerListener(EventListener listener) {
+ for (RedisKey key : listener.keys()) {
+ listeners.computeIfAbsent(key, k -> new ArrayList<>()).add(listener);
+ }
+
+ if (listener.getTimeout() != 0) {
+ scheduleTimeout(listener, listener.getTimeout());
+ }
+
+ keysRegistered += listener.keys().size();
+ }
+
+ public synchronized void fireEvent(RedisCommandType command, RedisKey key) {
+ if (listeners.isEmpty()) {
+ return;
+ }
+
+ List<EventListener> listenerList = listeners.get(key);
+ if (listenerList == null) {
+ return;
+ }
+
+ for (EventListener listener : listenerList) {
+ if (listener.process(command, key) == EventResponse.REMOVE_AND_STOP) {
+ removeListener(listener);
+ break;
+ }
+ }
+ }
+
+ /**
+ * The total number of keys registered by all listeners (includes duplicates).
+ */
+ @VisibleForTesting
+ public int size() {
+ return keysRegistered;
+ }
+
+ @Override
+ public synchronized void afterBucketRemoved(int bucketId, Iterable<?> keys) {
+ Set<EventListener> resubmittingList = new HashSet<>();
+ for (Map.Entry<RedisKey, List<EventListener>> entry : listeners.entrySet()) {
+ if (entry.getKey().getBucketId() == bucketId) {
+ resubmittingList.addAll(entry.getValue());
+ }
+ }
+
+ resubmittingList.forEach(x -> {
+ x.resubmitCommand();
+ removeListener(x);
+ });
+ }
+
+ private synchronized void removeListener(EventListener listener) {
+ boolean listenerRemoved = false;
+ for (RedisKey key : listener.keys()) {
+ List<EventListener> listenerList = listeners.get(key);
+ listenerRemoved |= listenerList.remove(listener);
+ if (listenerList.isEmpty()) {
+ listeners.remove(key);
+ }
+ }
+
+ if (listenerRemoved) {
+ keysRegistered -= listener.keys().size();
+ if (listener.getTimeout() > 0) {
+ timerTasks.remove(listener).cancel();
Review comment:
OK - good suggestion. I've switched to using the ScheduledThreadPoolExecutor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r825628527
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/BlockingCommandListener.java
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.commands.executor.RedisResponse;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListener implements EventListener {
+
+ private final ExecutionHandlerContext context;
+ private final RedisCommandType command;
+ private final List<RedisKey> keys;
+ private final List<byte[]> commandArgs;
+ private final long timeoutNanos;
+ private final long timeSubmitted;
+ private Runnable cleanupTask;
+
+ /**
+ * Constructor to create an instance of a BlockingCommandListener in response to a blocking
+ * command. When receiving a relevant event, blocking commands simply resubmit the command
+ * into the Netty pipeline.
+ *
+ * @param context the associated ExecutionHandlerContext
+ * @param command the blocking command associated with this listener
+ * @param keys the list of keys the command is interested in
+ * @param timeoutSeconds the timeout for the command to block in seconds
+ * @param commandArgs all arguments to the command which are used for resubmission
+ */
+ public BlockingCommandListener(ExecutionHandlerContext context, RedisCommandType command,
+ List<RedisKey> keys, double timeoutSeconds, List<byte[]> commandArgs) {
+ this.context = context;
+ this.command = command;
Review comment:
Removed in favor of passing the actual `Command` down.
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/BlockingCommandListener.java
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.commands.executor.RedisResponse;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListener implements EventListener {
+
+ private final ExecutionHandlerContext context;
+ private final RedisCommandType command;
+ private final List<RedisKey> keys;
+ private final List<byte[]> commandArgs;
+ private final long timeoutNanos;
+ private final long timeSubmitted;
+ private Runnable cleanupTask;
+
+ /**
+ * Constructor to create an instance of a BlockingCommandListener in response to a blocking
+ * command. When receiving a relevant event, blocking commands simply resubmit the command
+ * into the Netty pipeline.
+ *
+ * @param context the associated ExecutionHandlerContext
+ * @param command the blocking command associated with this listener
+ * @param keys the list of keys the command is interested in
+ * @param timeoutSeconds the timeout for the command to block in seconds
+ * @param commandArgs all arguments to the command which are used for resubmission
+ */
+ public BlockingCommandListener(ExecutionHandlerContext context, RedisCommandType command,
+ List<RedisKey> keys, double timeoutSeconds, List<byte[]> commandArgs) {
+ this.context = context;
+ this.command = command;
+ this.timeoutNanos = (long) (timeoutSeconds * 1e9);
+ this.keys = Collections.unmodifiableList(keys);
+ this.commandArgs = commandArgs;
+ timeSubmitted = System.nanoTime();
+ }
+
+ @Override
+ public List<RedisKey> keys() {
+ return keys;
+ }
+
+ @Override
+ public EventResponse process(RedisCommandType commandType, RedisKey key) {
+ if (!keys.contains(key)) {
+ return EventResponse.CONTINUE;
+ }
+
+ resubmitCommand();
+ return EventResponse.REMOVE_AND_STOP;
+ }
+
+ @Override
+ public void resubmitCommand() {
+ // Recalculate the timeout since we've already been waiting
+ double adjustedTimeoutSeconds = 0;
+ if (timeoutNanos > 0) {
+ long adjustedTimeoutNanos = timeoutNanos - (System.nanoTime() - timeSubmitted);
+ adjustedTimeoutNanos = Math.max(1, adjustedTimeoutNanos);
+ adjustedTimeoutSeconds = ((double) adjustedTimeoutNanos) / 1e9;
+ }
+
+ // The commands we are currently supporting all have the timeout at the end of the argument
+ // list. Some newer Redis 7 commands (BLMPOP and BZMPOP) have the timeout as the first argument
+ // after the command. We'll need to adjust this once those commands are supported.
+ commandArgs.set(commandArgs.size() - 1, Coder.doubleToBytes(adjustedTimeoutSeconds));
+
+ context.resubmitCommand(new Command(command, commandArgs));
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r825627887
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventDistributor.java
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.cache.partition.PartitionListenerAdapter;
+import org.apache.geode.logging.internal.executors.LoggingThreadFactory;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributor extends PartitionListenerAdapter {
+
+ private final Map<RedisKey, Queue<EventListener>> listeners = new ConcurrentHashMap<>();
+
+ private final ScheduledThreadPoolExecutor timerExecutor =
+ new ScheduledThreadPoolExecutor(1,
+ new LoggingThreadFactory("GeodeForRedisEventTimer-", true));
+
+ private int keysRegistered = 0;
+
+ public EventDistributor() {
+ timerExecutor.setRemoveOnCancelPolicy(true);
+ }
+
+ public synchronized void registerListener(EventListener listener) {
Review comment:
From the tests I added, it does seem like it. At least even around the iteration of the keys since otherwise a listener could only be partially registered before a remove attempt.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r824375851
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventResponse.java
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+/**
+ * Response returned by {@link EventListener#process(RedisCommandType, RedisKey)}
+ */
+public enum EventResponse {
+ /**
+ * This response would be used by listeners handling keyspace events
+ */
+ CONTINUE,
+ /**
+ * This response would be used by listeners handling blocking commands
+ */
+ REMOVE_AND_STOP;
Review comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r824375802
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventListener.java
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.List;
+
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+/**
+ * Interface intended to be implemented in order to receive Redis events. Specifically this would be
+ * keyspace events or blocking commands. EventListeners are registered with the
+ * {@link EventDistributor}.
+ */
+public interface EventListener {
+
+ /**
+ * Receive and process an event. This method should execute very quickly. The return value
+ * determines additional process steps for the given event.
+ *
+ * @param commandType the command triggering the event
+ * @param key the key triggering the event
+ * @return response determining subsequent processing steps
+ */
+ EventResponse process(RedisCommandType commandType, RedisKey key);
+
+ /**
+ * Return the list of keys this listener is interested in.
+ */
+ List<RedisKey> keys();
+
+ /**
+ * Method to resubmit a command if appropriate. This is only relevant for listeners that process
+ * events for blocking commands. Listeners that handle keyspace event notification will not use
+ * this.
+ */
+ default void resubmitCommand() {};
+
+ /**
+ * Retrieve the timeout for this listener. The default is no timeout.
Review comment:
Theoretically it's implementation dependent so I think it's OK not to mention it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] nonbinaryprogrammer commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
nonbinaryprogrammer commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r825047945
##########
File path: geode-for-redis/src/test/java/org/apache/geode/redis/internal/eventing/BlockingCommandListenerTest.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+import org.junit.Test;
+import org.mockito.ArgumentCaptor;
+
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListenerTest {
+
+ @Test
+ public void testTimeoutIsAdjusted() {
+ ExecutionHandlerContext context = mock(ExecutionHandlerContext.class);
+ List<byte[]> commandArgs = Arrays.asList("KEY".getBytes(), "0".getBytes());
Review comment:
you can avoid a call to `getBytes()` by just doing `'0'`
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/BlockingCommandListener.java
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.commands.executor.RedisResponse;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListener implements EventListener {
+
+ private final ExecutionHandlerContext context;
+ private final RedisCommandType command;
+ private final List<RedisKey> keys;
+ private final List<byte[]> commandArgs;
+ private final long timeoutNanos;
+ private final long timeSubmitted;
+ private Runnable cleanupTask;
+
+ /**
+ * Constructor to create an instance of a BlockingCommandListener in response to a blocking
+ * command. When receiving a relevant event, blocking commands simply resubmit the command
+ * into the Netty pipeline.
+ *
+ * @param context the associated ExecutionHandlerContext
+ * @param command the blocking command associated with this listener
+ * @param keys the list of keys the command is interested in
+ * @param timeoutSeconds the timeout for the command to block in seconds
+ * @param commandArgs all arguments to the command which are used for resubmission
+ */
+ public BlockingCommandListener(ExecutionHandlerContext context, RedisCommandType command,
+ List<RedisKey> keys, double timeoutSeconds, List<byte[]> commandArgs) {
+ this.context = context;
+ this.command = command;
+ this.timeoutNanos = (long) (timeoutSeconds * 1e9);
+ this.keys = Collections.unmodifiableList(keys);
+ this.commandArgs = commandArgs;
+ timeSubmitted = System.nanoTime();
+ }
+
+ @Override
+ public List<RedisKey> keys() {
+ return keys;
+ }
+
+ @Override
+ public EventResponse process(RedisCommandType commandType, RedisKey key) {
+ if (!keys.contains(key)) {
+ return EventResponse.CONTINUE;
+ }
+
+ resubmitCommand();
+ return EventResponse.REMOVE_AND_STOP;
+ }
+
+ @Override
+ public void resubmitCommand() {
+ // Recalculate the timeout since we've already been waiting
+ double adjustedTimeoutSeconds = 0;
+ if (timeoutNanos > 0) {
+ long adjustedTimeoutNanos = timeoutNanos - (System.nanoTime() - timeSubmitted);
Review comment:
I don't think there is any reason to have parens here
##########
File path: geode-for-redis/src/integrationTest/java/org/apache/geode/redis/internal/commands/executor/list/AbstractBLPopIntegrationTest.java
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.BIND_ADDRESS;
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.REDIS_CLIENT_TIMEOUT;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.HostAndPort;
+import redis.clients.jedis.JedisCluster;
+import redis.clients.jedis.Protocol;
+
+import org.apache.geode.redis.RedisIntegrationTest;
+import org.apache.geode.redis.internal.RedisConstants;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.junit.rules.ExecutorServiceRule;
+
+public abstract class AbstractBLPopIntegrationTest implements RedisIntegrationTest {
+ private static final String KEY = "key";
+
+ protected JedisCluster jedis;
+
+ public abstract void awaitEventDistributorSize(int size) throws Exception;
+
+ @ClassRule
+ public static ExecutorServiceRule executor = new ExecutorServiceRule();
+
+ @Before
+ public void setUp() {
+ jedis = new JedisCluster(new HostAndPort(BIND_ADDRESS, getPort()), REDIS_CLIENT_TIMEOUT);
+ }
+
+ @After
+ public void tearDown() {
+ flushAll();
+ jedis.close();
+ }
+
+ @Test
+ public void testInvalidArguments_throwErrors() {
+ assertThatThrownBy(() -> jedis.sendCommand("key", Protocol.Command.BLPOP))
+ .hasMessageContaining("ERR wrong number of arguments for 'blpop' command");
+ assertThatThrownBy(() -> jedis.sendCommand("key1", Protocol.Command.BLPOP, "key"))
+ .hasMessageContaining("ERR wrong number of arguments for 'blpop' command");
+ }
+
+ @Test
+ public void testInvalidTimeout_throwsError() {
+ assertThatThrownBy(() -> jedis.sendCommand("key1", Protocol.Command.BLPOP, "key1",
+ "0.A"))
+ .hasMessageContaining(RedisConstants.ERROR_TIMEOUT_INVALID);
+ }
+
+ @Test
+ public void testKeysInDifferentSlots_throwsError() {
+ assertThatThrownBy(() -> jedis.sendCommand("key1", Protocol.Command.BLPOP, "key1",
+ "key2", "0"))
+ .hasMessageContaining(RedisConstants.ERROR_WRONG_SLOT);
+ }
+
Review comment:
I'd like to see a test of what happens if we try to BLPOP a key that isn't a list
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/data/RedisList.java
##########
@@ -154,6 +160,35 @@ public long lpush(List<byte[]> elementsToAdd, Region<RedisKey, RedisData> region
return popped;
}
+ public static List<byte[]> blpop(ExecutionHandlerContext context, List<RedisKey> keys,
+ double timeoutSeconds) {
+ RegionProvider regionProvider = context.getRegionProvider();
+ for (RedisKey key : keys) {
+ RedisList list = regionProvider.getTypedRedisData(REDIS_LIST, key, false);
+ if (!list.isNull()) {
+ byte[] poppedValue = list.lpop(context.getRegion(), key);
+
+ // return the key and value
+ List<byte[]> result = new ArrayList<>(2);
+ result.add(key.toBytes());
+ result.add(poppedValue);
+ return result;
+ }
+ }
+
+ List<byte[]> commandArgs = new ArrayList<>(keys.size() + 1);
+ commandArgs.add(RedisCommandType.BLPOP.name().getBytes());
+ keys.forEach(x -> commandArgs.add(x.toBytes()));
+ // This is a placeholder for the timeout. If the command is resubmitted it will be updated with
+ // an adjusted timeout value.
+ commandArgs.add("0".getBytes());
Review comment:
why not just `commandArgs.add('0');`?
##########
File path: geode-for-redis/src/test/java/org/apache/geode/redis/internal/eventing/BlockingCommandListenerTest.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+import org.junit.Test;
+import org.mockito.ArgumentCaptor;
+
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BlockingCommandListenerTest {
+
+ @Test
+ public void testTimeoutIsAdjusted() {
+ ExecutionHandlerContext context = mock(ExecutionHandlerContext.class);
+ List<byte[]> commandArgs = Arrays.asList("KEY".getBytes(), "0".getBytes());
+ BlockingCommandListener listener =
+ new BlockingCommandListener(context, RedisCommandType.BLPOP, Collections.emptyList(),
+ 1.0D,
+ commandArgs);
+
+ listener.resubmitCommand();
+
+ ArgumentCaptor<Command> argumentCaptor = ArgumentCaptor.forClass(Command.class);
+ verify(context, times(1)).resubmitCommand(argumentCaptor.capture());
+
+ double timeout = Coder.bytesToDouble(argumentCaptor.getValue().getCommandArguments().get(0));
+ assertThat(timeout).isLessThan(1.0D);
+ }
+
+ @Test
+ public void testAdjustedTimeoutDoesNotBecomeNegative() {
+ ExecutionHandlerContext context = mock(ExecutionHandlerContext.class);
+ List<byte[]> commandArgs = Arrays.asList("KEY".getBytes(), "0".getBytes());
Review comment:
same here
##########
File path: geode-for-redis/src/integrationTest/java/org/apache/geode/redis/internal/eventing/EventDistributorTest.java
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.stream.Collectors;
+
+import org.junit.Test;
+
+import org.apache.geode.redis.ConcurrentLoopingThreads;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+
+public class EventDistributorTest {
+
+ public static class TestEventListener implements EventListener {
+ private final List<RedisKey> keys;
+ private int fired = 0;
+ private final long timeout;
+
+ public TestEventListener(RedisKey... keys) {
+ this(0, keys);
+ }
+
+ public TestEventListener(long timeout, RedisKey... keys) {
+ this.keys = Arrays.stream(keys).collect(Collectors.toList());
+ this.timeout = timeout;
+ }
+
+ public int getFired() {
+ return fired;
+ }
+
+ @Override
+ public EventResponse process(RedisCommandType commandType, RedisKey key) {
+ fired += 1;
+ return EventResponse.REMOVE_AND_STOP;
+ }
+
+ @Override
+ public List<RedisKey> keys() {
+ return keys;
+ }
+
+ @Override
+ public void resubmitCommand() {}
+
+ @Override
+ public long getTimeout() {
+ return timeout;
+ }
+
+ @Override
+ public void timeout() {}
+
+ @Override
+ public void setCleanupTask(Runnable r) {}
+
+ @Override
+ public void cleanup() {}
+ }
+
+ @Test
+ public void firingEventRemovesListener() {
+ RedisKey keyA = new RedisKey("a".getBytes());
+ RedisKey keyB = new RedisKey("b".getBytes());
+ EventDistributor distributor = new EventDistributor();
+ TestEventListener listener = new TestEventListener(keyA, keyB);
+ distributor.registerListener(listener);
+
+ distributor.fireEvent(null, keyA);
+ assertThat(listener.getFired()).isEqualTo(1);
+ assertThat(distributor.size()).isEqualTo(0);
+ }
+
+ @Test
+ public void firingEventRemovesFirstListener_whenMultipleExist() {
+ RedisKey keyA = new RedisKey("a".getBytes());
+ RedisKey keyB = new RedisKey("b".getBytes());
+ EventDistributor distributor = new EventDistributor();
+ TestEventListener listener1 = new TestEventListener(keyA, keyB);
+ TestEventListener listener2 = new TestEventListener(keyA, keyB);
+ distributor.registerListener(listener1);
+ distributor.registerListener(listener2);
+
+ assertThat(distributor.size()).isEqualTo(4);
+
+ distributor.fireEvent(null, keyA);
+ assertThat(listener1.getFired()).isEqualTo(1);
+ assertThat(listener2.getFired()).isEqualTo(0);
+ assertThat(distributor.size()).isEqualTo(2);
+
+ distributor.fireEvent(null, keyA);
+ assertThat(listener1.getFired()).isEqualTo(1);
+ assertThat(listener2.getFired()).isEqualTo(1);
+ assertThat(distributor.size()).isEqualTo(0);
+ }
+
+ @Test
+ public void listenerIsRemovedAfterTimeout() {
+ RedisKey keyA = new RedisKey("a".getBytes());
+ EventDistributor distributor = new EventDistributor();
+ TestEventListener listener1 = new TestEventListener(1, keyA);
Review comment:
what are the units for this timeout? it doesn't say anywhere in the comments or code. If the timeout is in seconds then this seems like it has the potential to be a very flaky test
##########
File path: geode-for-redis/src/integrationTest/java/org/apache/geode/redis/internal/eventing/EventDistributorTest.java
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.stream.Collectors;
+
+import org.junit.Test;
+
+import org.apache.geode.redis.ConcurrentLoopingThreads;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+
+public class EventDistributorTest {
+
+ public static class TestEventListener implements EventListener {
+ private final List<RedisKey> keys;
+ private int fired = 0;
+ private final long timeout;
+
+ public TestEventListener(RedisKey... keys) {
+ this(0, keys);
+ }
+
+ public TestEventListener(long timeout, RedisKey... keys) {
+ this.keys = Arrays.stream(keys).collect(Collectors.toList());
+ this.timeout = timeout;
+ }
+
+ public int getFired() {
+ return fired;
+ }
+
+ @Override
+ public EventResponse process(RedisCommandType commandType, RedisKey key) {
+ fired += 1;
+ return EventResponse.REMOVE_AND_STOP;
+ }
+
+ @Override
+ public List<RedisKey> keys() {
+ return keys;
+ }
+
+ @Override
+ public void resubmitCommand() {}
+
+ @Override
+ public long getTimeout() {
+ return timeout;
+ }
+
+ @Override
+ public void timeout() {}
+
+ @Override
+ public void setCleanupTask(Runnable r) {}
+
+ @Override
+ public void cleanup() {}
+ }
+
+ @Test
+ public void firingEventRemovesListener() {
+ RedisKey keyA = new RedisKey("a".getBytes());
+ RedisKey keyB = new RedisKey("b".getBytes());
+ EventDistributor distributor = new EventDistributor();
+ TestEventListener listener = new TestEventListener(keyA, keyB);
+ distributor.registerListener(listener);
+
+ distributor.fireEvent(null, keyA);
+ assertThat(listener.getFired()).isEqualTo(1);
+ assertThat(distributor.size()).isEqualTo(0);
+ }
+
+ @Test
+ public void firingEventRemovesFirstListener_whenMultipleExist() {
+ RedisKey keyA = new RedisKey("a".getBytes());
+ RedisKey keyB = new RedisKey("b".getBytes());
+ EventDistributor distributor = new EventDistributor();
+ TestEventListener listener1 = new TestEventListener(keyA, keyB);
+ TestEventListener listener2 = new TestEventListener(keyA, keyB);
+ distributor.registerListener(listener1);
+ distributor.registerListener(listener2);
+
+ assertThat(distributor.size()).isEqualTo(4);
+
+ distributor.fireEvent(null, keyA);
+ assertThat(listener1.getFired()).isEqualTo(1);
+ assertThat(listener2.getFired()).isEqualTo(0);
+ assertThat(distributor.size()).isEqualTo(2);
+
+ distributor.fireEvent(null, keyA);
+ assertThat(listener1.getFired()).isEqualTo(1);
+ assertThat(listener2.getFired()).isEqualTo(1);
+ assertThat(distributor.size()).isEqualTo(0);
+ }
+
+ @Test
+ public void listenerIsRemovedAfterTimeout() {
+ RedisKey keyA = new RedisKey("a".getBytes());
+ EventDistributor distributor = new EventDistributor();
+ TestEventListener listener1 = new TestEventListener(1, keyA);
Review comment:
looking farther down the code it looks like that is in seconds. seems like the timeout should be less than the `atMost` in the awaitility
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] DonalEvans commented on a change in pull request #7408: GEODE-10048: Add framework for Redis events and BLPOP command
Posted by GitBox <gi...@apache.org>.
DonalEvans commented on a change in pull request #7408:
URL: https://github.com/apache/geode/pull/7408#discussion_r823005434
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/commands/executor/list/BLPopExecutor.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.geode.redis.internal.RedisConstants;
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.executor.CommandExecutor;
+import org.apache.geode.redis.internal.commands.executor.RedisResponse;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.data.RedisList;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BLPopExecutor implements CommandExecutor {
+
+ @Override
+ public RedisResponse executeCommand(Command command, ExecutionHandlerContext context) {
+ List<byte[]> arguments = command.getCommandArguments();
+ int keyCount = arguments.size() - 1;
Review comment:
This variable name is confusing given how it's used when determining the timeout. Should it be something like `timeoutIndex` instead?
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/commands/executor/list/BLPopExecutor.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.geode.redis.internal.RedisConstants;
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.executor.CommandExecutor;
+import org.apache.geode.redis.internal.commands.executor.RedisResponse;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.data.RedisList;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BLPopExecutor implements CommandExecutor {
+
+ @Override
+ public RedisResponse executeCommand(Command command, ExecutionHandlerContext context) {
+ List<byte[]> arguments = command.getCommandArguments();
+ int keyCount = arguments.size() - 1;
+ double timeoutMillis;
+ try {
+ timeoutMillis = Coder.bytesToDouble(arguments.get(keyCount)) * 1000;
+ } catch (NumberFormatException e) {
+ return RedisResponse.error(RedisConstants.ERROR_TIMEOUT_INVALID);
+ }
Review comment:
We also need a check that the timeout isn't negative. If it is, Redis returns `ERR timeout is negative`.
##########
File path: geode-for-redis/src/integrationTest/java/org/apache/geode/redis/internal/commands/executor/list/AbstractBLPopIntegrationTest.java
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.BIND_ADDRESS;
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.REDIS_CLIENT_TIMEOUT;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.HostAndPort;
+import redis.clients.jedis.JedisCluster;
+import redis.clients.jedis.Protocol;
+
+import org.apache.geode.redis.RedisIntegrationTest;
+import org.apache.geode.redis.internal.RedisConstants;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.junit.rules.ExecutorServiceRule;
+
+public abstract class AbstractBLPopIntegrationTest implements RedisIntegrationTest {
+ private static final String KEY = "key";
+
+ protected JedisCluster jedis;
+
+ public abstract void awaitEventDistributorSize(int size) throws Exception;
+
+ @ClassRule
+ public static ExecutorServiceRule executor = new ExecutorServiceRule();
+
+ @Before
+ public void setUp() {
+ jedis = new JedisCluster(new HostAndPort(BIND_ADDRESS, getPort()), REDIS_CLIENT_TIMEOUT);
+ }
+
+ @After
+ public void tearDown() {
+ flushAll();
+ jedis.close();
+ }
+
+ @Test
+ public void testInvalidArguments_throwErrors() {
+ assertThatThrownBy(() -> jedis.sendCommand("key", Protocol.Command.BLPOP))
+ .hasMessageContaining("ERR wrong number of arguments for 'blpop' command");
+ assertThatThrownBy(() -> jedis.sendCommand("key1", Protocol.Command.BLPOP, "key"))
+ .hasMessageContaining("ERR wrong number of arguments for 'blpop' command");
+ }
+
+ @Test
+ public void testInvalidTimeout_throwsError() {
+ assertThatThrownBy(() -> jedis.sendCommand("key1", Protocol.Command.BLPOP, "key1",
+ "0.A"))
+ .hasMessageContaining(RedisConstants.ERROR_TIMEOUT_INVALID);
Review comment:
Could the `hasMessageContaining()` calls in this class be replaced with `hasMessage()` to be consistent with the changes from GEODE-10049?
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/commands/executor/list/BLPopExecutor.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.geode.redis.internal.RedisConstants;
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.executor.CommandExecutor;
+import org.apache.geode.redis.internal.commands.executor.RedisResponse;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.data.RedisList;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BLPopExecutor implements CommandExecutor {
+
+ @Override
+ public RedisResponse executeCommand(Command command, ExecutionHandlerContext context) {
+ List<byte[]> arguments = command.getCommandArguments();
+ int keyCount = arguments.size() - 1;
+ double timeoutMillis;
+ try {
+ timeoutMillis = Coder.bytesToDouble(arguments.get(keyCount)) * 1000;
+ } catch (NumberFormatException e) {
+ return RedisResponse.error(RedisConstants.ERROR_TIMEOUT_INVALID);
+ }
+
+ List<RedisKey> keys = new ArrayList<>(keyCount);
+ for (int i = 0; i < keyCount; i++) {
+ keys.add(new RedisKey(arguments.get(i)));
+ }
+
+ List<byte[]> popped = context.lockedExecute(keys.get(0), keys,
+ () -> RedisList.blpop(context, keys, (int) timeoutMillis));
Review comment:
Inside `BlockingCommandListener.java`, the timeout is stored as a `long`. If we decide to use a double rather than an integer for the timeout argument, it might be best to have `timeoutMillis` be a `long` and assign it using:
```
timeoutMillis = (long) (bytesToDouble(arguments.get(keyCount)) * 1000);
```
or to change the timeout to be an `int` throughout.
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventResponse.java
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+/**
+ * Response returned by {@link EventListener#process(RedisCommandType, RedisKey)}
+ */
+public enum EventResponse {
+ /**
+ * This response would be used by listeners handling keyspace events
+ */
+ CONTINUE,
+ /**
+ * This response would be used by listeners handling blocking commands
+ */
+ REMOVE_AND_STOP;
Review comment:
Unnecessary semicolon here.
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/commands/executor/list/BLPopExecutor.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.geode.redis.internal.RedisConstants;
+import org.apache.geode.redis.internal.commands.Command;
+import org.apache.geode.redis.internal.commands.executor.CommandExecutor;
+import org.apache.geode.redis.internal.commands.executor.RedisResponse;
+import org.apache.geode.redis.internal.data.RedisKey;
+import org.apache.geode.redis.internal.data.RedisList;
+import org.apache.geode.redis.internal.netty.Coder;
+import org.apache.geode.redis.internal.netty.ExecutionHandlerContext;
+
+public class BLPopExecutor implements CommandExecutor {
+
+ @Override
+ public RedisResponse executeCommand(Command command, ExecutionHandlerContext context) {
+ List<byte[]> arguments = command.getCommandArguments();
+ int keyCount = arguments.size() - 1;
+ double timeoutMillis;
+ try {
+ timeoutMillis = Coder.bytesToDouble(arguments.get(keyCount)) * 1000;
Review comment:
Specifying the timeout as a double is a feature new in Redis 6. In other commands added for Lists (specifically LPOP and RPOP) we've matched the behaviour of Redis 5. For consistency, should the implementation of BLPOP be taking only integer values for timeout?
##########
File path: geode-for-redis/src/integrationTest/java/org/apache/geode/redis/internal/commands/executor/list/AbstractBLPopIntegrationTest.java
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.BIND_ADDRESS;
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.REDIS_CLIENT_TIMEOUT;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.HostAndPort;
+import redis.clients.jedis.JedisCluster;
+import redis.clients.jedis.Protocol;
+
+import org.apache.geode.redis.RedisIntegrationTest;
+import org.apache.geode.redis.internal.RedisConstants;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.junit.rules.ExecutorServiceRule;
+
+public abstract class AbstractBLPopIntegrationTest implements RedisIntegrationTest {
+ private static final String KEY = "key";
+
+ protected JedisCluster jedis;
+
+ public abstract void awaitEventDistributorSize(int size) throws Exception;
+
+ @ClassRule
+ public static ExecutorServiceRule executor = new ExecutorServiceRule();
+
+ @Before
+ public void setUp() {
+ jedis = new JedisCluster(new HostAndPort(BIND_ADDRESS, getPort()), REDIS_CLIENT_TIMEOUT);
+ }
+
+ @After
+ public void tearDown() {
+ flushAll();
+ jedis.close();
+ }
+
+ @Test
+ public void testInvalidArguments_throwErrors() {
+ assertThatThrownBy(() -> jedis.sendCommand("key", Protocol.Command.BLPOP))
+ .hasMessageContaining("ERR wrong number of arguments for 'blpop' command");
+ assertThatThrownBy(() -> jedis.sendCommand("key1", Protocol.Command.BLPOP, "key"))
+ .hasMessageContaining("ERR wrong number of arguments for 'blpop' command");
+ }
Review comment:
This can be simplified to:
```
@Test
public void testInvalidNumberOfArguments_Errors() {
assertAtLeastNArgs(jedis, Protocol.Command.BLPOP, 2);
}
```
##########
File path: geode-for-redis/src/integrationTest/java/org/apache/geode/redis/internal/commands/executor/list/AbstractBLPopIntegrationTest.java
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode.redis.internal.commands.executor.list;
+
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.BIND_ADDRESS;
+import static org.apache.geode.test.dunit.rules.RedisClusterStartupRule.REDIS_CLIENT_TIMEOUT;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.HostAndPort;
+import redis.clients.jedis.JedisCluster;
+import redis.clients.jedis.Protocol;
+
+import org.apache.geode.redis.RedisIntegrationTest;
+import org.apache.geode.redis.internal.RedisConstants;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.junit.rules.ExecutorServiceRule;
+
+public abstract class AbstractBLPopIntegrationTest implements RedisIntegrationTest {
+ private static final String KEY = "key";
+
+ protected JedisCluster jedis;
+
+ public abstract void awaitEventDistributorSize(int size) throws Exception;
+
+ @ClassRule
+ public static ExecutorServiceRule executor = new ExecutorServiceRule();
+
+ @Before
+ public void setUp() {
+ jedis = new JedisCluster(new HostAndPort(BIND_ADDRESS, getPort()), REDIS_CLIENT_TIMEOUT);
+ }
+
+ @After
+ public void tearDown() {
+ flushAll();
+ jedis.close();
+ }
+
+ @Test
+ public void testInvalidArguments_throwErrors() {
+ assertThatThrownBy(() -> jedis.sendCommand("key", Protocol.Command.BLPOP))
+ .hasMessageContaining("ERR wrong number of arguments for 'blpop' command");
+ assertThatThrownBy(() -> jedis.sendCommand("key1", Protocol.Command.BLPOP, "key"))
+ .hasMessageContaining("ERR wrong number of arguments for 'blpop' command");
+ }
+
+ @Test
+ public void testInvalidTimeout_throwsError() {
Review comment:
We need to also add a test for when the timeout is negative.
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventListener.java
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.List;
+
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+/**
+ * Interface intended to be implemented in order to receive Redis events. Specifically this would be
+ * keyspace events or blocking commands. EventListeners are registered with the
+ * {@link EventDistributor}.
+ */
+public interface EventListener {
+
+ /**
+ * Receive and process an event. This method should execute very quickly. The return value
+ * determines additional process steps for the given event.
+ *
+ * @param commandType the command triggering the event
+ * @param key the key triggering the event
+ * @return response determining subsequent processing steps
+ */
+ EventResponse process(RedisCommandType commandType, RedisKey key);
+
+ /**
+ * Return the list of keys this listener is interested in.
+ */
+ List<RedisKey> keys();
+
+ /**
+ * Method to resubmit a command if appropriate. This is only relevant for listeners that process
+ * events for blocking commands. Listeners that handle keyspace event notification will not use
+ * this.
+ */
+ default void resubmitCommand() {};
+
+ /**
+ * Retrieve the timeout for this listener. The default is no timeout.
Review comment:
Should this comment specify the units for the timeout?
##########
File path: geode-for-redis/src/main/java/org/apache/geode/redis/internal/eventing/EventDistributor.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.geode.redis.internal.eventing;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.geode.annotations.VisibleForTesting;
+import org.apache.geode.cache.partition.PartitionListenerAdapter;
+import org.apache.geode.logging.internal.executors.LoggingThreadFactory;
+import org.apache.geode.redis.internal.commands.RedisCommandType;
+import org.apache.geode.redis.internal.data.RedisKey;
+
+public class EventDistributor extends PartitionListenerAdapter {
+
+ private final Map<RedisKey, Queue<EventListener>> listeners = new ConcurrentHashMap<>();
+
+ private final ScheduledThreadPoolExecutor timerExecutor =
+ new ScheduledThreadPoolExecutor(1,
+ new LoggingThreadFactory("GeodeForRedisEventTimer-", true));
+
+ private int keysRegistered = 0;
+
+ public EventDistributor() {
+ timerExecutor.setRemoveOnCancelPolicy(true);
+ }
+
+ public synchronized void registerListener(EventListener listener) {
+ for (RedisKey key : listener.keys()) {
+ listeners.computeIfAbsent(key, k -> new LinkedBlockingQueue<>()).add(listener);
+ }
+
+ if (listener.getTimeout() != 0) {
+ scheduleTimeout(listener, listener.getTimeout());
+ }
+
+ keysRegistered += listener.keys().size();
+ }
+
+ public void fireEvent(RedisCommandType command, RedisKey key) {
+ Queue<EventListener> listenerList = listeners.get(key);
+ if (listenerList == null) {
+ return;
+ }
+
+ for (EventListener listener : listenerList) {
+ if (listener.process(command, key) == EventResponse.REMOVE_AND_STOP) {
+ removeListener(listener);
+ break;
+ }
+ }
+ }
+
+ /**
+ * The total number of keys registered by all listeners (includes duplicates).
+ */
+ @VisibleForTesting
+ public int size() {
+ return keysRegistered;
+ }
+
+ @Override
+ public void afterBucketRemoved(int bucketId, Iterable<?> keys) {
+ Set<EventListener> resubmittingList = new HashSet<>();
+ for (Map.Entry<RedisKey, Queue<EventListener>> entry : listeners.entrySet()) {
+ if (entry.getKey().getBucketId() == bucketId) {
+ resubmittingList.addAll(entry.getValue());
+ }
+ }
+
+ resubmittingList.forEach(x -> {
+ x.resubmitCommand();
+ removeListener(x);
+ });
+ }
+
+ private synchronized void removeListener(EventListener listener) {
+ boolean listenerRemoved = false;
+ for (RedisKey key : listener.keys()) {
+ Queue<EventListener> listenerList = listeners.get(key);
+ if (listenerList == null) {
+ continue;
+ }
+ listenerRemoved |= listenerList.remove(listener);
+ if (listenerList.isEmpty()) {
+ listeners.remove(key);
+ }
+ }
+
+ if (listenerRemoved) {
+ keysRegistered -= listener.keys().size();
+ listener.cleanup();
+ }
+ }
+
+ private void scheduleTimeout(EventListener listener, long timeout) {
+ ScheduledFuture<?> scheduledTask =
+ timerExecutor.schedule(() -> removeListener(listener), timeout, TimeUnit.MILLISECONDS);
+ listener.setCleanupTask(() -> scheduledTask.cancel(false));
+ }
Review comment:
If BLPOP is executed with a non-zero timeout, then once that timeout elapses, the client should receive a `null` response. The below test passes with open source Redis but fails with the current implementation in geode-for-redis:
```
@Test
public void testBLPopWhenTimeoutIsExceeded() {
int timeout = 10;
Future<List<String>> future = executor.submit(() -> jedis.blpop(timeout, KEY));
await().atMost(timeout * 2, TimeUnit.SECONDS).untilAsserted(() -> assertThat(future.get()).isNull());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org