You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@pekko.apache.org by "Roiocam (via GitHub)" <gi...@apache.org> on 2024/04/19 09:34:16 UTC

[PR] perf: avoid large offset query via limit windowing [pekko-persistence-jdbc]

Roiocam opened a new pull request, #180:
URL: https://github.com/apache/pekko-persistence-jdbc/pull/180

   ## Motivation
   
   solves: #140
   
   Looks like we don't need to change a lot.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [PR] perf: avoid large offset query via limit windowing [pekko-persistence-jdbc]

Posted by "laglangyue (via GitHub)" <gi...@apache.org>.
laglangyue commented on PR #180:
URL: https://github.com/apache/pekko-persistence-jdbc/pull/180#issuecomment-2116486956

   lgtm


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [PR] perf: avoid large offset query via limit windowing [pekko-persistence-jdbc]

Posted by "nvollmar (via GitHub)" <gi...@apache.org>.
nvollmar commented on code in PR #180:
URL: https://github.com/apache/pekko-persistence-jdbc/pull/180#discussion_r1613480943


##########
core/src/main/scala/org/apache/pekko/persistence/jdbc/journal/dao/BaseJournalDaoWithReadMessages.scala:
##########
@@ -38,13 +38,28 @@ trait BaseJournalDaoWithReadMessages extends JournalDaoWithReadMessages {
       toSequenceNr: Long,
       batchSize: Int,
       refreshInterval: Option[(FiniteDuration, Scheduler)]): Source[Try[(PersistentRepr, Long)], NotUsed] = {
+    internalBatchStream(persistenceId, fromSequenceNr, toSequenceNr, batchSize, refreshInterval).mapConcat(identity)
+  }
 
+  /**
+   * separate this method for unit tests.
+   */
+  @InternalApi
+  private[dao] def internalBatchStream(persistenceId: String,
+      fromSequenceNr: Long,
+      toSequenceNr: Long,
+      batchSize: Int,
+      refreshInterval: Option[(FiniteDuration, Scheduler)]) = {

Review Comment:
   it is generally good practice to do input validation to catch failures early and make constraints explicit



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [PR] perf: avoid large offset query via limit windowing [pekko-persistence-jdbc]

Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on PR #180:
URL: https://github.com/apache/pekko-persistence-jdbc/pull/180#issuecomment-2131419462

   @Roiocam can you merge this if you are happy it is ready? This is the last PR that we need before we can do an RC for this module.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [PR] perf: avoid large offset query via limit windowing [pekko-persistence-jdbc]

Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on code in PR #180:
URL: https://github.com/apache/pekko-persistence-jdbc/pull/180#discussion_r1581740633


##########
core/src/main/scala/org/apache/pekko/persistence/jdbc/journal/dao/BaseJournalDaoWithReadMessages.scala:
##########
@@ -38,13 +38,28 @@ trait BaseJournalDaoWithReadMessages extends JournalDaoWithReadMessages {
       toSequenceNr: Long,
       batchSize: Int,
       refreshInterval: Option[(FiniteDuration, Scheduler)]): Source[Try[(PersistentRepr, Long)], NotUsed] = {
+    internalBatchStream(persistenceId, fromSequenceNr, toSequenceNr, batchSize, refreshInterval).mapConcat(identity)
+  }
 
+  /**
+   * separate this method for unit tests.
+   */
+  @InternalApi
+  private[dao] def internalBatchStream(persistenceId: String,
+      fromSequenceNr: Long,
+      toSequenceNr: Long,
+      batchSize: Int,
+      refreshInterval: Option[(FiniteDuration, Scheduler)]) = {

Review Comment:
   should add some check for `fromSequenceNr` and `batchSize`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [PR] perf: avoid large offset query via limit windowing [pekko-persistence-jdbc]

Posted by "Roiocam (via GitHub)" <gi...@apache.org>.
Roiocam commented on code in PR #180:
URL: https://github.com/apache/pekko-persistence-jdbc/pull/180#discussion_r1611371770


##########
core/src/main/scala/org/apache/pekko/persistence/jdbc/journal/dao/BaseJournalDaoWithReadMessages.scala:
##########
@@ -38,13 +38,28 @@ trait BaseJournalDaoWithReadMessages extends JournalDaoWithReadMessages {
       toSequenceNr: Long,
       batchSize: Int,
       refreshInterval: Option[(FiniteDuration, Scheduler)]): Source[Try[(PersistentRepr, Long)], NotUsed] = {
+    internalBatchStream(persistenceId, fromSequenceNr, toSequenceNr, batchSize, refreshInterval).mapConcat(identity)
+  }
 
+  /**
+   * separate this method for unit tests.
+   */
+  @InternalApi
+  private[dao] def internalBatchStream(persistenceId: String,
+      fromSequenceNr: Long,
+      toSequenceNr: Long,
+      batchSize: Int,
+      refreshInterval: Option[(FiniteDuration, Scheduler)]) = {

Review Comment:
   > should add some check for fromSequenceNr and batchSize
   
   what does that mean? batch size didn't allow `zero` value, and the `fromSequenceNr` was generate from acotr it self.
   
   > and add checking for where fromSequenceNr > toSequenceNr too
   
   `fromSequenceNr > toSequenceNr` is enforced down below.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [PR] perf: avoid large offset query via limit windowing [pekko-persistence-jdbc]

Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on code in PR #180:
URL: https://github.com/apache/pekko-persistence-jdbc/pull/180#discussion_r1580672500


##########
core/src/test/scala/org/apache/pekko/persistence/jdbc/query/JournalDaoStreamMessagesMemoryTest.scala:
##########
@@ -14,138 +14,126 @@
 
 package org.apache.pekko.persistence.jdbc.query
 
-import java.lang.management.ManagementFactory
-import java.lang.management.MemoryMXBean
-import java.util.UUID
-
-import org.apache.pekko
-import pekko.actor.ActorSystem
-import pekko.persistence.{ AtomicWrite, PersistentRepr }
-import pekko.persistence.jdbc.journal.dao.legacy.{ ByteArrayJournalDao, JournalTables }
-import pekko.serialization.SerializationExtension
-import pekko.stream.scaladsl.{ Sink, Source }
 import com.typesafe.config.{ ConfigValue, ConfigValueFactory }
+import org.apache.pekko
 import org.scalatest.concurrent.PatienceConfiguration.Timeout
 import org.slf4j.LoggerFactory
+import pekko.persistence.jdbc.query.JournalDaoStreamMessagesMemoryTest.fetchSize
+import pekko.persistence.{ AtomicWrite, PersistentRepr }
+import pekko.stream.scaladsl.{ Sink, Source }
+import pekko.stream.testkit.scaladsl.TestSink
+import pekko.stream.{ Materializer, SystemMaterializer }
 
+import java.lang.management.{ ManagementFactory, MemoryMXBean }
+import java.util.UUID
 import scala.collection.immutable
 import scala.concurrent.ExecutionContext
 import scala.concurrent.duration._
 import scala.util.{ Failure, Success }
-import pekko.stream.testkit.scaladsl.TestSink
-import org.scalatest.matchers.should.Matchers
 
 object JournalDaoStreamMessagesMemoryTest {
 
-  val configOverrides: Map[String, ConfigValue] = Map("jdbc-journal.fetch-size" -> ConfigValueFactory.fromAnyRef("100"))
+  val fetchSize: Int = 100
+  val MB: Int = 1024 * 1024
 
-  val MB = 1024 * 1024
+  val configOverrides: Map[String, ConfigValue] = Map(
+    "jdbc-journal.fetch-size" -> ConfigValueFactory.fromAnyRef("100"))
 }
 
 abstract class JournalDaoStreamMessagesMemoryTest(configFile: String)
-    extends QueryTestSpec(configFile, JournalDaoStreamMessagesMemoryTest.configOverrides)
-    with JournalTables
-    with Matchers {
+    extends QueryTestSpec(configFile, JournalDaoStreamMessagesMemoryTest.configOverrides) {
+
   import JournalDaoStreamMessagesMemoryTest.MB
 
   private val log = LoggerFactory.getLogger(this.getClass)
 
-  val journalSequenceActorConfig = readJournalConfig.journalSequenceRetrievalConfiguration
-  val journalTableCfg = journalConfig.journalTableConfiguration
+  val memoryMBean: MemoryMXBean = ManagementFactory.getMemoryMXBean
 
-  implicit val askTimeout: FiniteDuration = 50.millis
+  it should "stream events" in withActorSystem { implicit system =>
+    implicit val ec: ExecutionContext = system.dispatcher
+    implicit val mat: Materializer = SystemMaterializer(system).materializer
 
-  def generateId: Int = 0
+    withDao { dao =>
+      val persistenceId = UUID.randomUUID().toString
 
-  val memoryMBean: MemoryMXBean = ManagementFactory.getMemoryMXBean
+      val writerUuid = UUID.randomUUID().toString
+
+      val payloadSize = 5000 // 5000 bytes
+      val eventsPerBatch = 1000
+
+      val maxMem = 64 * MB
 
-  behavior.of("Replaying Persistence Actor")
-
-  it should "stream events" in {
-    if (newDao)
-      pending
-    withActorSystem { implicit system: ActorSystem =>
-      withDatabase { db =>
-        implicit val ec: ExecutionContext = system.dispatcher
-
-        val persistenceId = UUID.randomUUID().toString
-        val dao = new ByteArrayJournalDao(db, profile, journalConfig, SerializationExtension(system))
-
-        val payloadSize = 5000 // 5000 bytes
-        val eventsPerBatch = 1000
-
-        val maxMem = 64 * MB
-
-        val numberOfInsertBatches = {
-          // calculate the number of batches using a factor to make sure we go a little bit over the limit
-          (maxMem / (payloadSize * eventsPerBatch) * 1.2).round.toInt
-        }
-        val totalMessages = numberOfInsertBatches * eventsPerBatch
-        val totalMessagePayload = totalMessages * payloadSize
-        log.info(
-          s"batches: $numberOfInsertBatches (with $eventsPerBatch events), total messages: $totalMessages, total msgs size: $totalMessagePayload")
-
-        // payload can be the same when inserting to avoid unnecessary memory usage
-        val payload = Array.fill(payloadSize)('a'.toByte)
-
-        val lastInsert =
-          Source
-            .fromIterator(() => (1 to numberOfInsertBatches).toIterator)
-            .mapAsync(1) { i =>
-              val end = i * eventsPerBatch
-              val start = end - (eventsPerBatch - 1)
-              log.info(s"batch $i - events from $start to $end")
-              val atomicWrites =
-                (start to end).map { j =>
-                  AtomicWrite(immutable.Seq(PersistentRepr(payload, j, persistenceId)))
-                }.toSeq
-
-              dao.asyncWriteMessages(atomicWrites).map(_ => i)
-            }
-            .runWith(Sink.last)
-
-        // wait until we write all messages
-        // being very generous, 1 second per message
-        lastInsert.futureValue(Timeout(totalMessages.seconds))
-
-        log.info("Events written, starting replay")
-
-        // sleep and gc to have some kind of stable measurement of current heap usage
-        Thread.sleep(1000)
-        System.gc()
-        Thread.sleep(1000)
-        val usedBefore = memoryMBean.getHeapMemoryUsage.getUsed
-
-        val messagesSrc =
-          dao.messagesWithBatch(persistenceId, 0, totalMessages, batchSize = 100, None)
-        val probe =
-          messagesSrc
-            .map {
-              case Success((repr, _)) =>
-                if (repr.sequenceNr % 100 == 0)
-                  log.info(s"fetched: ${repr.persistenceId} - ${repr.sequenceNr}/$totalMessages")
-              case Failure(exception) =>
-                log.error("Failure when reading messages.", exception)
-            }
-            .runWith(TestSink.probe)
-
-        probe.request(10)
-        probe.within(20.seconds) {
-          probe.expectNextN(10)
-        }
-
-        // sleep and gc to have some kind of stable measurement of current heap usage
-        Thread.sleep(2000)
-        System.gc()
-        Thread.sleep(1000)
-        val usedAfter = memoryMBean.getHeapMemoryUsage.getUsed
-
-        log.info(s"Used heap before ${usedBefore / MB} MB, after ${usedAfter / MB} MB")
-        // actual usage is much less than 10 MB
-        (usedAfter - usedBefore) should be <= (10L * MB)
-
-        probe.cancel()
+      val numberOfInsertBatches = {
+        // calculate the number of batches using a factor to make sure we go a little bit over the limit
+        (maxMem / (payloadSize * eventsPerBatch) * 1.2).round.toInt
       }
+      val totalMessages = numberOfInsertBatches * eventsPerBatch
+      val totalMessagePayload = totalMessages * payloadSize
+      log.info(
+        s"batches: $numberOfInsertBatches (with $eventsPerBatch events), total messages: $totalMessages, total msgs size: $totalMessagePayload")
+
+      // payload can be the same when inserting to avoid unnecessary memory usage
+      val payload = Array.fill(payloadSize)('a'.toByte)
+
+      val lastInsert =
+        Source
+          .fromIterator(() => (1 to numberOfInsertBatches).iterator)
+          .mapAsync(1) { i =>
+            val end = i * eventsPerBatch
+            val start = end - (eventsPerBatch - 1)
+            log.info(s"batch $i - events from $start to $end")
+            val atomicWrites =
+              (start to end).map { j =>
+                AtomicWrite(immutable.Seq(PersistentRepr(payload, j, persistenceId, writerUuid = writerUuid)))
+              }
+            dao.asyncWriteMessages(atomicWrites).map(_ => i)
+          }
+          .runWith(Sink.last)
+
+      // wait until we write all messages
+      // being very generous, 1 second per message
+      lastInsert.futureValue(Timeout(totalMessages.seconds))
+
+      log.info("Events written, starting replay")
+
+      // sleep and gc to have some kind of stable measurement of current heap usage
+      Thread.sleep(1000)
+      System.gc()
+      Thread.sleep(1000)
+      val usedBefore = memoryMBean.getHeapMemoryUsage.getUsed
+
+      val messagesSrc =
+        dao.messagesWithBatch(persistenceId, 0, totalMessages, batchSize = fetchSize, None)
+      val probe =
+        messagesSrc
+          .map {
+            case Success((repr, _)) =>
+              if (repr.sequenceNr % 100 == 0)
+                log.info(s"fetched: ${repr.persistenceId} - ${repr.sequenceNr}/$totalMessages")
+            case Failure(exception) =>
+              log.error("Failure when reading messages.", exception)
+          }
+          .runWith(TestSink.probe)
+
+      probe.request(10)
+      probe.within(20.seconds) {
+        probe.expectNextN(10)
+      }
+
+      // sleep and gc to have some kind of stable measurement of current heap usage

Review Comment:
   ok - I see there is an assertion - this does look like it could be problematic - any tests that have magic sleep times are liable to fail occasionally depending on machine performance



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [PR] perf: avoid large offset query via limit windowing [pekko-persistence-jdbc]

Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on code in PR #180:
URL: https://github.com/apache/pekko-persistence-jdbc/pull/180#discussion_r1581740573


##########
core/src/main/scala/org/apache/pekko/persistence/jdbc/journal/dao/BaseJournalDaoWithReadMessages.scala:
##########
@@ -38,13 +38,28 @@ trait BaseJournalDaoWithReadMessages extends JournalDaoWithReadMessages {
       toSequenceNr: Long,
       batchSize: Int,
       refreshInterval: Option[(FiniteDuration, Scheduler)]): Source[Try[(PersistentRepr, Long)], NotUsed] = {
+    internalBatchStream(persistenceId, fromSequenceNr, toSequenceNr, batchSize, refreshInterval).mapConcat(identity)
+  }
 
+  /**
+   * separate this method for unit tests.
+   */
+  @InternalApi
+  private[dao] def internalBatchStream(persistenceId: String,

Review Comment:
   better to move the persistentceId to the next line.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [PR] perf: avoid large offset query via limit windowing [pekko-persistence-jdbc]

Posted by "Roiocam (via GitHub)" <gi...@apache.org>.
Roiocam commented on code in PR #180:
URL: https://github.com/apache/pekko-persistence-jdbc/pull/180#discussion_r1604241037


##########
core/src/test/scala/org/apache/pekko/persistence/jdbc/query/JournalDaoStreamMessagesMemoryTest.scala:
##########
@@ -15,13 +15,14 @@
 package org.apache.pekko.persistence.jdbc.query
 
 import com.typesafe.config.{ ConfigValue, ConfigValueFactory }
-import org.apache.pekko.persistence.jdbc.query.JournalDaoStreamMessagesMemoryTest.fetchSize
-import org.apache.pekko.persistence.{ AtomicWrite, PersistentRepr }
-import org.apache.pekko.stream.scaladsl.{ Sink, Source }
-import org.apache.pekko.stream.testkit.scaladsl.TestSink
-import org.apache.pekko.stream.{ Materializer, SystemMaterializer }
+import org.apache.pekko

Review Comment:
   fixed



##########
core/src/test/scala/org/apache/pekko/persistence/jdbc/journal/dao/LimitWindowingStreamTest.scala:
##########
@@ -20,13 +20,14 @@
 package org.apache.pekko.persistence.jdbc.journal.dao
 
 import com.typesafe.config.{ ConfigValue, ConfigValueFactory }
-import org.apache.pekko.persistence.jdbc.journal.dao.LimitWindowingStreamTest.fetchSize
-import org.apache.pekko.persistence.jdbc.query.{ H2Cleaner, QueryTestSpec }
-import org.apache.pekko.persistence.{ AtomicWrite, PersistentRepr }
-import org.apache.pekko.stream.scaladsl.{ Keep, Sink, Source }
-import org.apache.pekko.stream.{ Materializer, SystemMaterializer }
+import org.apache.pekko

Review Comment:
   fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [PR] perf: avoid large offset query via limit windowing [pekko-persistence-jdbc]

Posted by "Roiocam (via GitHub)" <gi...@apache.org>.
Roiocam merged PR #180:
URL: https://github.com/apache/pekko-persistence-jdbc/pull/180


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [PR] perf: avoid large offset query via limit windowing [pekko-persistence-jdbc]

Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on code in PR #180:
URL: https://github.com/apache/pekko-persistence-jdbc/pull/180#discussion_r1580666227


##########
core/src/test/scala/org/apache/pekko/persistence/jdbc/journal/dao/LimitWindowingStreamTest.scala:
##########
@@ -20,13 +20,14 @@
 package org.apache.pekko.persistence.jdbc.journal.dao
 
 import com.typesafe.config.{ ConfigValue, ConfigValueFactory }
-import org.apache.pekko.persistence.jdbc.journal.dao.LimitWindowingStreamTest.fetchSize
-import org.apache.pekko.persistence.jdbc.query.{ H2Cleaner, QueryTestSpec }
-import org.apache.pekko.persistence.{ AtomicWrite, PersistentRepr }
-import org.apache.pekko.stream.scaladsl.{ Keep, Sink, Source }
-import org.apache.pekko.stream.{ Materializer, SystemMaterializer }
+import org.apache.pekko

Review Comment:
   vould you keep this with other pekko imports?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [PR] perf: avoid large offset query via limit windowing [pekko-persistence-jdbc]

Posted by "Roiocam (via GitHub)" <gi...@apache.org>.
Roiocam commented on PR #180:
URL: https://github.com/apache/pekko-persistence-jdbc/pull/180#issuecomment-2115770406

   > @Roiocam I want to release v1.1.0-M1 of this module within the next week or 2. Is this PR something that we can get merged in that timeframe or could we add it after v1.1.0-M1 is released?
   
   Busy paid time works, I will fix these over the weekend.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [PR] perf: avoid large offset query via limit windowing [pekko-persistence-jdbc]

Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on code in PR #180:
URL: https://github.com/apache/pekko-persistence-jdbc/pull/180#discussion_r1580670428


##########
core/src/test/scala/org/apache/pekko/persistence/jdbc/query/JournalDaoStreamMessagesMemoryTest.scala:
##########
@@ -14,138 +14,126 @@
 
 package org.apache.pekko.persistence.jdbc.query
 
-import java.lang.management.ManagementFactory
-import java.lang.management.MemoryMXBean
-import java.util.UUID
-
-import org.apache.pekko
-import pekko.actor.ActorSystem
-import pekko.persistence.{ AtomicWrite, PersistentRepr }
-import pekko.persistence.jdbc.journal.dao.legacy.{ ByteArrayJournalDao, JournalTables }
-import pekko.serialization.SerializationExtension
-import pekko.stream.scaladsl.{ Sink, Source }
 import com.typesafe.config.{ ConfigValue, ConfigValueFactory }
+import org.apache.pekko
 import org.scalatest.concurrent.PatienceConfiguration.Timeout
 import org.slf4j.LoggerFactory
+import pekko.persistence.jdbc.query.JournalDaoStreamMessagesMemoryTest.fetchSize
+import pekko.persistence.{ AtomicWrite, PersistentRepr }
+import pekko.stream.scaladsl.{ Sink, Source }
+import pekko.stream.testkit.scaladsl.TestSink
+import pekko.stream.{ Materializer, SystemMaterializer }
 
+import java.lang.management.{ ManagementFactory, MemoryMXBean }
+import java.util.UUID
 import scala.collection.immutable
 import scala.concurrent.ExecutionContext
 import scala.concurrent.duration._
 import scala.util.{ Failure, Success }
-import pekko.stream.testkit.scaladsl.TestSink
-import org.scalatest.matchers.should.Matchers
 
 object JournalDaoStreamMessagesMemoryTest {
 
-  val configOverrides: Map[String, ConfigValue] = Map("jdbc-journal.fetch-size" -> ConfigValueFactory.fromAnyRef("100"))
+  val fetchSize: Int = 100
+  val MB: Int = 1024 * 1024
 
-  val MB = 1024 * 1024
+  val configOverrides: Map[String, ConfigValue] = Map(
+    "jdbc-journal.fetch-size" -> ConfigValueFactory.fromAnyRef("100"))
 }
 
 abstract class JournalDaoStreamMessagesMemoryTest(configFile: String)
-    extends QueryTestSpec(configFile, JournalDaoStreamMessagesMemoryTest.configOverrides)
-    with JournalTables
-    with Matchers {
+    extends QueryTestSpec(configFile, JournalDaoStreamMessagesMemoryTest.configOverrides) {
+
   import JournalDaoStreamMessagesMemoryTest.MB
 
   private val log = LoggerFactory.getLogger(this.getClass)
 
-  val journalSequenceActorConfig = readJournalConfig.journalSequenceRetrievalConfiguration
-  val journalTableCfg = journalConfig.journalTableConfiguration
+  val memoryMBean: MemoryMXBean = ManagementFactory.getMemoryMXBean
 
-  implicit val askTimeout: FiniteDuration = 50.millis
+  it should "stream events" in withActorSystem { implicit system =>
+    implicit val ec: ExecutionContext = system.dispatcher
+    implicit val mat: Materializer = SystemMaterializer(system).materializer
 
-  def generateId: Int = 0
+    withDao { dao =>
+      val persistenceId = UUID.randomUUID().toString
 
-  val memoryMBean: MemoryMXBean = ManagementFactory.getMemoryMXBean
+      val writerUuid = UUID.randomUUID().toString
+
+      val payloadSize = 5000 // 5000 bytes
+      val eventsPerBatch = 1000
+
+      val maxMem = 64 * MB
 
-  behavior.of("Replaying Persistence Actor")
-
-  it should "stream events" in {
-    if (newDao)
-      pending
-    withActorSystem { implicit system: ActorSystem =>
-      withDatabase { db =>
-        implicit val ec: ExecutionContext = system.dispatcher
-
-        val persistenceId = UUID.randomUUID().toString
-        val dao = new ByteArrayJournalDao(db, profile, journalConfig, SerializationExtension(system))
-
-        val payloadSize = 5000 // 5000 bytes
-        val eventsPerBatch = 1000
-
-        val maxMem = 64 * MB
-
-        val numberOfInsertBatches = {
-          // calculate the number of batches using a factor to make sure we go a little bit over the limit
-          (maxMem / (payloadSize * eventsPerBatch) * 1.2).round.toInt
-        }
-        val totalMessages = numberOfInsertBatches * eventsPerBatch
-        val totalMessagePayload = totalMessages * payloadSize
-        log.info(
-          s"batches: $numberOfInsertBatches (with $eventsPerBatch events), total messages: $totalMessages, total msgs size: $totalMessagePayload")
-
-        // payload can be the same when inserting to avoid unnecessary memory usage
-        val payload = Array.fill(payloadSize)('a'.toByte)
-
-        val lastInsert =
-          Source
-            .fromIterator(() => (1 to numberOfInsertBatches).toIterator)
-            .mapAsync(1) { i =>
-              val end = i * eventsPerBatch
-              val start = end - (eventsPerBatch - 1)
-              log.info(s"batch $i - events from $start to $end")
-              val atomicWrites =
-                (start to end).map { j =>
-                  AtomicWrite(immutable.Seq(PersistentRepr(payload, j, persistenceId)))
-                }.toSeq
-
-              dao.asyncWriteMessages(atomicWrites).map(_ => i)
-            }
-            .runWith(Sink.last)
-
-        // wait until we write all messages
-        // being very generous, 1 second per message
-        lastInsert.futureValue(Timeout(totalMessages.seconds))
-
-        log.info("Events written, starting replay")
-
-        // sleep and gc to have some kind of stable measurement of current heap usage
-        Thread.sleep(1000)
-        System.gc()
-        Thread.sleep(1000)
-        val usedBefore = memoryMBean.getHeapMemoryUsage.getUsed
-
-        val messagesSrc =
-          dao.messagesWithBatch(persistenceId, 0, totalMessages, batchSize = 100, None)
-        val probe =
-          messagesSrc
-            .map {
-              case Success((repr, _)) =>
-                if (repr.sequenceNr % 100 == 0)
-                  log.info(s"fetched: ${repr.persistenceId} - ${repr.sequenceNr}/$totalMessages")
-              case Failure(exception) =>
-                log.error("Failure when reading messages.", exception)
-            }
-            .runWith(TestSink.probe)
-
-        probe.request(10)
-        probe.within(20.seconds) {
-          probe.expectNextN(10)
-        }
-
-        // sleep and gc to have some kind of stable measurement of current heap usage
-        Thread.sleep(2000)
-        System.gc()
-        Thread.sleep(1000)
-        val usedAfter = memoryMBean.getHeapMemoryUsage.getUsed
-
-        log.info(s"Used heap before ${usedBefore / MB} MB, after ${usedAfter / MB} MB")
-        // actual usage is much less than 10 MB
-        (usedAfter - usedBefore) should be <= (10L * MB)
-
-        probe.cancel()
+      val numberOfInsertBatches = {
+        // calculate the number of batches using a factor to make sure we go a little bit over the limit
+        (maxMem / (payloadSize * eventsPerBatch) * 1.2).round.toInt
       }
+      val totalMessages = numberOfInsertBatches * eventsPerBatch
+      val totalMessagePayload = totalMessages * payloadSize
+      log.info(
+        s"batches: $numberOfInsertBatches (with $eventsPerBatch events), total messages: $totalMessages, total msgs size: $totalMessagePayload")
+
+      // payload can be the same when inserting to avoid unnecessary memory usage
+      val payload = Array.fill(payloadSize)('a'.toByte)
+
+      val lastInsert =
+        Source
+          .fromIterator(() => (1 to numberOfInsertBatches).iterator)
+          .mapAsync(1) { i =>
+            val end = i * eventsPerBatch
+            val start = end - (eventsPerBatch - 1)
+            log.info(s"batch $i - events from $start to $end")
+            val atomicWrites =
+              (start to end).map { j =>
+                AtomicWrite(immutable.Seq(PersistentRepr(payload, j, persistenceId, writerUuid = writerUuid)))
+              }
+            dao.asyncWriteMessages(atomicWrites).map(_ => i)
+          }
+          .runWith(Sink.last)
+
+      // wait until we write all messages
+      // being very generous, 1 second per message
+      lastInsert.futureValue(Timeout(totalMessages.seconds))
+
+      log.info("Events written, starting replay")
+
+      // sleep and gc to have some kind of stable measurement of current heap usage
+      Thread.sleep(1000)
+      System.gc()
+      Thread.sleep(1000)
+      val usedBefore = memoryMBean.getHeapMemoryUsage.getUsed
+
+      val messagesSrc =
+        dao.messagesWithBatch(persistenceId, 0, totalMessages, batchSize = fetchSize, None)
+      val probe =
+        messagesSrc
+          .map {
+            case Success((repr, _)) =>
+              if (repr.sequenceNr % 100 == 0)
+                log.info(s"fetched: ${repr.persistenceId} - ${repr.sequenceNr}/$totalMessages")
+            case Failure(exception) =>
+              log.error("Failure when reading messages.", exception)
+          }
+          .runWith(TestSink.probe)
+
+      probe.request(10)
+      probe.within(20.seconds) {
+        probe.expectNextN(10)
+      }
+
+      // sleep and gc to have some kind of stable measurement of current heap usage

Review Comment:
   could this log info be surrounded by a condition that is normally disabled? this logging takes a few seconds and noone is going to look at it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [PR] perf: avoid large offset query via limit windowing [pekko-persistence-jdbc]

Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on code in PR #180:
URL: https://github.com/apache/pekko-persistence-jdbc/pull/180#discussion_r1581740725


##########
core/src/main/scala/org/apache/pekko/persistence/jdbc/journal/dao/BaseJournalDaoWithReadMessages.scala:
##########
@@ -38,13 +38,28 @@ trait BaseJournalDaoWithReadMessages extends JournalDaoWithReadMessages {
       toSequenceNr: Long,
       batchSize: Int,
       refreshInterval: Option[(FiniteDuration, Scheduler)]): Source[Try[(PersistentRepr, Long)], NotUsed] = {
+    internalBatchStream(persistenceId, fromSequenceNr, toSequenceNr, batchSize, refreshInterval).mapConcat(identity)
+  }
 
+  /**
+   * separate this method for unit tests.
+   */
+  @InternalApi
+  private[dao] def internalBatchStream(persistenceId: String,
+      fromSequenceNr: Long,
+      toSequenceNr: Long,
+      batchSize: Int,
+      refreshInterval: Option[(FiniteDuration, Scheduler)]) = {
     Source
       .unfoldAsync[(Long, FlowControl), Seq[Try[(PersistentRepr, Long)]]]((Math.max(1, fromSequenceNr), Continue)) {
         case (from, control) =>
+          def limitWindow(from: Long): Long = {

Review Comment:
   limitedWindows?



##########
core/src/main/scala/org/apache/pekko/persistence/jdbc/journal/dao/BaseJournalDaoWithReadMessages.scala:
##########
@@ -38,13 +38,28 @@ trait BaseJournalDaoWithReadMessages extends JournalDaoWithReadMessages {
       toSequenceNr: Long,
       batchSize: Int,
       refreshInterval: Option[(FiniteDuration, Scheduler)]): Source[Try[(PersistentRepr, Long)], NotUsed] = {
+    internalBatchStream(persistenceId, fromSequenceNr, toSequenceNr, batchSize, refreshInterval).mapConcat(identity)
+  }
 
+  /**
+   * separate this method for unit tests.
+   */
+  @InternalApi
+  private[dao] def internalBatchStream(persistenceId: String,
+      fromSequenceNr: Long,
+      toSequenceNr: Long,
+      batchSize: Int,
+      refreshInterval: Option[(FiniteDuration, Scheduler)]) = {
     Source
       .unfoldAsync[(Long, FlowControl), Seq[Try[(PersistentRepr, Long)]]]((Math.max(1, fromSequenceNr), Continue)) {
         case (from, control) =>
+          def limitWindow(from: Long): Long = {

Review Comment:
   limitedWindows?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [PR] perf: avoid large offset query via limit windowing [pekko-persistence-jdbc]

Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on code in PR #180:
URL: https://github.com/apache/pekko-persistence-jdbc/pull/180#discussion_r1581741042


##########
core/src/main/scala/org/apache/pekko/persistence/jdbc/journal/dao/BaseJournalDaoWithReadMessages.scala:
##########
@@ -38,13 +38,28 @@ trait BaseJournalDaoWithReadMessages extends JournalDaoWithReadMessages {
       toSequenceNr: Long,
       batchSize: Int,
       refreshInterval: Option[(FiniteDuration, Scheduler)]): Source[Try[(PersistentRepr, Long)], NotUsed] = {
+    internalBatchStream(persistenceId, fromSequenceNr, toSequenceNr, batchSize, refreshInterval).mapConcat(identity)
+  }
 
+  /**
+   * separate this method for unit tests.
+   */
+  @InternalApi
+  private[dao] def internalBatchStream(persistenceId: String,
+      fromSequenceNr: Long,
+      toSequenceNr: Long,
+      batchSize: Int,
+      refreshInterval: Option[(FiniteDuration, Scheduler)]) = {

Review Comment:
   and add checking for where `fromSequenceNr > toSequenceNr` too



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [PR] perf: avoid large offset query via limit windowing [pekko-persistence-jdbc]

Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on PR #180:
URL: https://github.com/apache/pekko-persistence-jdbc/pull/180#issuecomment-2115353594

   @Roiocam I want to release v1.1.0-M1 of this module within the next week or 2. Is this PR something that we can get merged in that timeframe or could we add it after v1.1.0-M1 is released?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [PR] perf: avoid large offset query via limit windowing [pekko-persistence-jdbc]

Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on code in PR #180:
URL: https://github.com/apache/pekko-persistence-jdbc/pull/180#discussion_r1580666227


##########
core/src/test/scala/org/apache/pekko/persistence/jdbc/journal/dao/LimitWindowingStreamTest.scala:
##########
@@ -20,13 +20,14 @@
 package org.apache.pekko.persistence.jdbc.journal.dao
 
 import com.typesafe.config.{ ConfigValue, ConfigValueFactory }
-import org.apache.pekko.persistence.jdbc.journal.dao.LimitWindowingStreamTest.fetchSize
-import org.apache.pekko.persistence.jdbc.query.{ H2Cleaner, QueryTestSpec }
-import org.apache.pekko.persistence.{ AtomicWrite, PersistentRepr }
-import org.apache.pekko.stream.scaladsl.{ Keep, Sink, Source }
-import org.apache.pekko.stream.{ Materializer, SystemMaterializer }
+import org.apache.pekko

Review Comment:
   could you keep this with other pekko imports?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [PR] perf: avoid large offset query via limit windowing [pekko-persistence-jdbc]

Posted by "Roiocam (via GitHub)" <gi...@apache.org>.
Roiocam commented on code in PR #180:
URL: https://github.com/apache/pekko-persistence-jdbc/pull/180#discussion_r1580405653


##########
core/src/test/scala/org/apache/pekko/persistence/jdbc/query/LegacyJournalDaoStreamMessagesMemoryTest.scala:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * license agreements; and to You under the Apache License, version 2.0:
+ *
+ *   https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * This file is part of the Apache Pekko project, which was derived from Akka.
+ */
+
+/*
+ * Copyright (C) 2014 - 2019 Dennis Vriend <https://github.com/dnvriend>
+ * Copyright (C) 2019 - 2021 Lightbend Inc. <https://www.lightbend.com>
+ */
+
+package org.apache.pekko.persistence.jdbc.query
+
+import com.typesafe.config.{ ConfigValue, ConfigValueFactory }
+import org.apache.pekko.persistence.jdbc.journal.dao.legacy.ByteArrayJournalDao
+import org.apache.pekko.persistence.{ AtomicWrite, PersistentRepr }
+import org.apache.pekko.serialization.SerializationExtension
+import org.apache.pekko.stream.scaladsl.{ Sink, Source }
+import org.apache.pekko.stream.testkit.scaladsl.TestSink
+import org.scalatest.concurrent.PatienceConfiguration.Timeout
+import org.slf4j.LoggerFactory
+
+import java.lang.management.{ ManagementFactory, MemoryMXBean }
+import java.util.UUID
+import scala.collection.immutable
+import scala.concurrent.ExecutionContext
+import scala.concurrent.duration._
+import scala.util.{ Failure, Success }
+
+object LegacyJournalDaoStreamMessagesMemoryTest {
+
+  val configOverrides: Map[String, ConfigValue] = Map(
+    "jdbc-journal.fetch-size" -> ConfigValueFactory.fromAnyRef("100"),
+    "jdbc-journal.dao" -> ConfigValueFactory.fromAnyRef(
+      "org.apache.pekko.persistence.jdbc.journal.dao.legacy.ByteArrayJournalDao"))

Review Comment:
   use legacy dao to make it work.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [PR] perf: avoid large offset query via limit windowing [pekko-persistence-jdbc]

Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on code in PR #180:
URL: https://github.com/apache/pekko-persistence-jdbc/pull/180#discussion_r1580666795


##########
core/src/test/scala/org/apache/pekko/persistence/jdbc/query/JournalDaoStreamMessagesMemoryTest.scala:
##########
@@ -15,13 +15,14 @@
 package org.apache.pekko.persistence.jdbc.query
 
 import com.typesafe.config.{ ConfigValue, ConfigValueFactory }
-import org.apache.pekko.persistence.jdbc.query.JournalDaoStreamMessagesMemoryTest.fetchSize
-import org.apache.pekko.persistence.{ AtomicWrite, PersistentRepr }
-import org.apache.pekko.stream.scaladsl.{ Sink, Source }
-import org.apache.pekko.stream.testkit.scaladsl.TestSink
-import org.apache.pekko.stream.{ Materializer, SystemMaterializer }
+import org.apache.pekko

Review Comment:
   same issue with import order



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org