You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "fonsdant (via GitHub)" <gi...@apache.org> on 2024/12/24 19:53:36 UTC
[PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
fonsdant opened a new pull request, #18314:
URL: https://github.com/apache/kafka/pull/18314
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on code in PR #18314:
URL: https://github.com/apache/kafka/pull/18314#discussion_r1933119733
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
+ </li>
+ <li><code>ProcessorSupplier</code>: A processor supplier that can create one or more <code>Processor</code>
+ instances; and</li>
+ <li><code>FixedKeyProcessorSupplier</code>: A processor supplier that can create one or more
+ <code>FixedKeyProcessor</code> instances.
+ </li>
+ </ul>
+ <h4>Examples</h4>
+ <p>Follow the examples below to learn how to apply <code>process</code> and <code>processValues</code> to your
+ <code>KStream</code>.
+ </p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Operation</th>
+ <th>State Type</th>
+ </tr>
</thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
</tbody>
</table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <h5 id="cumulative-discounts-for-a-loyalty-program"">Cumulative Discounts for a Loyalty Program</h5>
+ <ul>
+ <li><strong>Idea:</strong> A stream of purchase events contains user IDs and transaction amounts. Use a state
+ store
+ to accumulate the total spending of each user. When their total crosses a threshold, apply a discount on
+ their next transaction and update their accumulated total.</li>
+ <li><strong>Real-World Context:</strong> In a retail loyalty program, tracking cumulative customer spending
+ enables
+ dynamic rewards, such as issuing a discount when a customer's total purchases exceed a predefined limit.
+ </li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+
+public class CumulativeDiscountsForALoyaltyProgramExample {
+ private static final double DISCOUNT_THRESHOLD = 100.0;
+ private static final String CUSTOMER_SPENDING_STORE = "customer-spending-store";
+ private static final String DISCOUNT_NOTIFICATION_MESSAGE =
+ "Discount applied! You have received a reward for your purchases.";
+ private static final String DISCOUNT_NOTIFICATIONS_TOPIC = "discount-notifications-topic";
+ private static final String PURCHASE_EVENTS_TOPIC = "purchase-events-topic";
+
+ public static void applyDiscountWithProcess(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ Serdes.String(),
+ Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Processor with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.process(CumulativeDiscountProcessor::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ private static class CumulativeDiscountProcessor implements Processor<String, Double, String, String> {
+ private KeyValueStore<String, Double> spendingStore;
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public void process(final Record<String, Double> record) {
+ if (record.value() == null) {
+ return; // Skip null purchase amounts
+ }
- @Override
- void process(Record<PageId, Long> record) {
- // Here you would format and send the alert email.
- //
- // In this specific example, you would be able to include
- // information about the page's ID and its view count
- }
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(record.key());
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += record.value();
+ spendingStore.put(record.key(), currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(record.key(), currentSpending - DISCOUNT_THRESHOLD);
+ // Send a discount notification
+ context.forward(record.withValue(DISCOUNT_NOTIFICATION_MESSAGE));
+ }
+ }
+ }
+}</code></pre>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li><strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g.,
+ INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to a dedicated topic too.</li>
+ <li><strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity
+ ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks,
+ and
+ INFO logs are stored for basic reporting purposes.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- void close() {
- // Any code for clean up would go here, for example tearing down the email client and anything
- // else you created in the init() method
- // This processor instance will not be used again after this call.
- }
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new)
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
+
+ private static class LogSeverityProcessor extends ContextualProcessor<String, String, String, String> {
+ @Override
+ public void process(final Record<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ final int colonIndex = record.value().indexOf(':');
+ final String severity = colonIndex > 0 ? record.value().substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Route logs based on severity
+ switch (severity) {
+ case "ERROR":
+ context().forward(record.withKey(ERROR_LOGS_TOPIC));
+ break;
+ case "WARN":
+ context().forward(record.withKey(WARN_LOGS_TOPIC));
+ break;
+ case "INFO":
+ // INFO logs are ignored
+ break;
+ default:
+ // Forward to an "unknown" topic for logs with unrecognized severities
+ context().forward(record.withKey(UNKNOWN_LOGS_TOPIC));
+ }
+ }
+ }
}</code></pre>
- <div class="admonition tip">
- <p><b>Tip</b></p>
- <p class="last">Even though we do not demonstrate it in this example, a stream processor can access any available state stores by
- calling <code class="docutils literal"><span class="pre">ProcessorContext#getStateStore()</span></code>.
- State stores are only available if they have been connected to the processor, or if they are global stores. While global stores do not need to be connected explicitly, they only allow for read-only access.
- There are two ways to connect state stores to a processor:
- <ul class="simple">
- <li>By passing the name of a store that has already been added via <code class="docutils literal"><span class="pre">Topology#addStateStore()</span></code> to the corresponding <code class="docutils literal"><span class="pre">KStream#process()</span></code> method call.</li>
- <li>Implementing <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the <code class="docutils literal"><span class="pre">ProcessorSupplier</span></code>
- passed to <code class="docutils literal"><span class="pre">KStream#process()</span></code>. In this case there is no need to call <code class="docutils literal"><span class="pre">StreamsBuilder#addStateStore()</span></code>
- beforehand, the store will be automatically added for you. You can also implement <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the
- <code class="docutils literal"><span class="pre">Value*</span></code> or <code class="docutils literal"><span class="pre">*WithKey</span></code> supplier variants, or <code class="docutils literal"><span class="pre">TransformerSupplier</span></code> or any of its variants.
- </li>
- </ul>
- </div>
- <p>Then we can leverage the <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code> processor in the DSL via <code class="docutils literal"><span class="pre">KStream#process</span></code>.</p>
- <pre class="line-numbers"><code class="language-java">KStream<String, GenericRecord> pageViews = ...;
-
-// Send an email notification when the view count of a page reaches one thousand.
-pageViews.groupByKey()
- .count()
- .filter((PageId pageId, Long viewCount) -> viewCount == 1000)
- // PopularPageEmailAlert is your custom processor that implements the
- // `Processor` interface, see further down below.
- .process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
- </div>
+ <h5 id="traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</h5>
+ <ul>
+ <li><strong>Idea:</strong> A radar monitors cars passing along a road stretch. A system counts the cars for each
+ day, maintaining a cumulative total for the current day in a state store. At the end of the day, the count
+ is
+ emitted and the state is cleared for the next day.</li>
+ <li><strong>Real-World Context:</strong> A car counting system can be useful for determining measures for
+ widening
+ or controlling traffic depending on the number of cars passing through the monitored stretch.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.common.serialization;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessorContext;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.state.Stores;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.format.DateTimeFormatter;
+
+public class TrafficRadarMonitoringCarCountExample {
+ private static final String DAILY_COUNT_STORE = "price-state-store";
+ private static final String DAILY_COUNT_TOPIC = "price-state-topic";
+ private static final String RADAR_COUNT_TOPIC = "car-radar-topic";
+
+ public static void countWithProcessValues(final StreamsBuilder builder) {
+ // Define a state store for tracking daily car counts
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(DAILY_COUNT_STORE),
+ Serdes.String(),
+ Serdes.Long()
+ )
+ );
+ final KStream<Void, String> radarStream = builder.stream(RADAR_COUNT_TOPIC);
+ // Apply the FixedKeyProcessor with the state store
+ radarStream.processValues(DailyCarCountProcessor::new, DAILY_COUNT_STORE)
+ .to(DAILY_COUNT_TOPIC);
+ }
+
+ private static class DailyCarCountProcessor implements FixedKeyProcessor<Void, String, String> {
+ private FixedKeyProcessorContext<Void, String> context;
+ private KeyValueStore<String, Long> stateStore;
+ private static final DateTimeFormatter DATE_FORMATTER =
+ DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(ZoneId.systemDefault());
+
+ @Override
+ public void init(final FixedKeyProcessorContext<Void, String> context) {
+ this.context = context;
+ stateStore = context.getStateStore(DAILY_COUNT_STORE);
+ }
+
+ @Override
+ public void process(final FixedKeyRecord<Void, String> record) {
+ if (record.value() == null) {
+ return; // Skip null events
+ }
+
+ // Derive the current day from the event timestamp
+ final long timestamp = System.currentTimeMillis(); // Use system time for simplicity
+ final String currentDay = DATE_FORMATTER.format(Instant.ofEpochMilli(timestamp));
+ // Retrieve the current count for the day
+ Long dailyCount = stateStore.get(currentDay);
+ if (dailyCount == null) {
+ dailyCount = 0L;
+ }
+ // Increment the count
+ dailyCount++;
+ stateStore.put(currentDay, dailyCount);
+
+ // Emit the current day's count
+ context.forward(record.withValue(String.format("Day: %s, Car Count: %s", currentDay, dailyCount)));
+ }
+ }
+}</code></pre>
+ <h5 id="replacing-slang-in-text-messages">Replacing Slang in Text Messages</h5>
+ <ul>
+ <li><strong>Idea:</strong> A messaging stream contains user-generated content, and you want to replace slang
+ words
+ with their formal equivalents (e.g., "u" becomes "you", "brb" becomes "be
+ right back"). The operation only modifies the message value and keeps the key intact.</li>
+ <li><strong>Real-World Context:</strong> In customer support chat systems, normalizing text by replacing slang
+ with
+ formal equivalents ensures that automated sentiment analysis tools work accurately and provide reliable
+ insights.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualFixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
+
+public class ReplacingSlangTextInMessagesExample {
+ private static final Map<String, String> SLANG_DICTIONARY = Map.of(
+ "u", "you",
+ "brb", "be right back",
+ "omg", "oh my god",
+ "btw", "by the way"
+ );
+ private static final String INPUT_MESSAGES_TOPIC = "input-messages-topic";
+ private static final String OUTPUT_MESSAGES_TOPIC = "output-messages-topic";
+
+ public static void replaceWithProcessValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.processValues(SlangReplacementProcessor::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ private static class SlangReplacementProcessor extends ContextualFixedKeyProcessor<String, String, String> {
+ @Override
+ public void process(final FixedKeyRecord<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+
+ // Replace slang words in the message
+ final String[] words = record.value().split("\\s+");
+ for (final String word : words) {
+ String replacedWord = SLANG_DICTIONARY.getOrDefault(word, word);
+ context().forward(record.withValue(replacedWord));
+ }
+ }
+ }
+}</code></pre>
+ <h4>Keynotes</h4>
+ <ul>
+ <li><strong>Type Safety and Flexibility:</strong> The process and processValues APIs utilize
+ <code>ProcessorContext</code> and <code>Record</code> or <code>FixedKeyRecord</code> objects for better type
+ safety and flexibility of custom processing logic.
+ </li>
+ <li><strong>Clear State and Logic Management:</strong> Implementations for <code>Processor</code> or
+ <code>FixedKeyProcessor</code> should manage state and logic clearly. Use <code>context().forward()</code>
+ for emitting records downstream.
+ </li>
+ <li><strong>Unified API:</strong> Consolidates multiple methods into a single, versatile API.</li>
+ <li><strong>Future-Proof:</strong> Ensures compatibility with the latest Kafka Streams releases.</li>
+ </ul>
+ </div>
+ <div class="section" id="transformers-removal-and-migration-to-processors">
+ <a class="headerlink" href="#transformers-removal-and-migration-to-processors" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id37">Transformers removal and migration to processors</a>
+ </h3>
+ </a>
+ <p>As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>,<br><code>transformValues</code>, <code>flatTransformValues</code>, and
+ <code>process</code> have been removed. These methods have been replaced with the<br>more versatile Processor API.
+ This guide provides detailed steps for migrating existing code to use the new Processor<br>API and explains the
+ benefits of the changes.</p>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ <li><code>KStream#process</code></li>
+ </ul>
+ <p>The Processor API now serves as a unified replacement for all these methods. It simplifies the API surface
+ while<br>maintaining support for both stateless and stateful operations.</p>
+ <h4>Migration Examples</h4>
+ <p>To migrate from the deprecated <code>transform</code>, <code>transformValues</code>, <code>flatTransform</code>, and
+ <code>flatTransformValues</code> methods to the<br>Processor API (PAPI) in Kafka Streams, let's resume the
+ previouss examples. The new <code>process</code> and <code>processValues</code><br>methods enable a more flexible
+ and reusable approach by requiring implementations of the <code>Processor</code>
+ or<br><code>FixedKeyProcessor</code> interfaces.</p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Migrating from</th>
+ <th>Migrating to</th>
+ <th>State Type</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program-removal">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>transform</code></td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
Review Comment:
Updated.
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
+ </li>
+ <li><code>ProcessorSupplier</code>: A processor supplier that can create one or more <code>Processor</code>
+ instances; and</li>
+ <li><code>FixedKeyProcessorSupplier</code>: A processor supplier that can create one or more
+ <code>FixedKeyProcessor</code> instances.
+ </li>
+ </ul>
+ <h4>Examples</h4>
+ <p>Follow the examples below to learn how to apply <code>process</code> and <code>processValues</code> to your
+ <code>KStream</code>.
+ </p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Operation</th>
+ <th>State Type</th>
+ </tr>
</thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
</tbody>
</table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <h5 id="cumulative-discounts-for-a-loyalty-program"">Cumulative Discounts for a Loyalty Program</h5>
+ <ul>
+ <li><strong>Idea:</strong> A stream of purchase events contains user IDs and transaction amounts. Use a state
+ store
+ to accumulate the total spending of each user. When their total crosses a threshold, apply a discount on
+ their next transaction and update their accumulated total.</li>
+ <li><strong>Real-World Context:</strong> In a retail loyalty program, tracking cumulative customer spending
+ enables
+ dynamic rewards, such as issuing a discount when a customer's total purchases exceed a predefined limit.
+ </li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+
+public class CumulativeDiscountsForALoyaltyProgramExample {
+ private static final double DISCOUNT_THRESHOLD = 100.0;
+ private static final String CUSTOMER_SPENDING_STORE = "customer-spending-store";
+ private static final String DISCOUNT_NOTIFICATION_MESSAGE =
+ "Discount applied! You have received a reward for your purchases.";
+ private static final String DISCOUNT_NOTIFICATIONS_TOPIC = "discount-notifications-topic";
+ private static final String PURCHASE_EVENTS_TOPIC = "purchase-events-topic";
+
+ public static void applyDiscountWithProcess(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ Serdes.String(),
+ Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Processor with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.process(CumulativeDiscountProcessor::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ private static class CumulativeDiscountProcessor implements Processor<String, Double, String, String> {
+ private KeyValueStore<String, Double> spendingStore;
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public void process(final Record<String, Double> record) {
+ if (record.value() == null) {
+ return; // Skip null purchase amounts
+ }
- @Override
- void process(Record<PageId, Long> record) {
- // Here you would format and send the alert email.
- //
- // In this specific example, you would be able to include
- // information about the page's ID and its view count
- }
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(record.key());
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += record.value();
+ spendingStore.put(record.key(), currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(record.key(), currentSpending - DISCOUNT_THRESHOLD);
+ // Send a discount notification
+ context.forward(record.withValue(DISCOUNT_NOTIFICATION_MESSAGE));
+ }
+ }
+ }
+}</code></pre>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li><strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g.,
+ INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to a dedicated topic too.</li>
+ <li><strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity
+ ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks,
+ and
+ INFO logs are stored for basic reporting purposes.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- void close() {
- // Any code for clean up would go here, for example tearing down the email client and anything
- // else you created in the init() method
- // This processor instance will not be used again after this call.
- }
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new)
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
+
+ private static class LogSeverityProcessor extends ContextualProcessor<String, String, String, String> {
+ @Override
+ public void process(final Record<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ final int colonIndex = record.value().indexOf(':');
+ final String severity = colonIndex > 0 ? record.value().substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Route logs based on severity
+ switch (severity) {
+ case "ERROR":
+ context().forward(record.withKey(ERROR_LOGS_TOPIC));
+ break;
+ case "WARN":
+ context().forward(record.withKey(WARN_LOGS_TOPIC));
+ break;
+ case "INFO":
+ // INFO logs are ignored
+ break;
+ default:
+ // Forward to an "unknown" topic for logs with unrecognized severities
+ context().forward(record.withKey(UNKNOWN_LOGS_TOPIC));
+ }
+ }
+ }
}</code></pre>
- <div class="admonition tip">
- <p><b>Tip</b></p>
- <p class="last">Even though we do not demonstrate it in this example, a stream processor can access any available state stores by
- calling <code class="docutils literal"><span class="pre">ProcessorContext#getStateStore()</span></code>.
- State stores are only available if they have been connected to the processor, or if they are global stores. While global stores do not need to be connected explicitly, they only allow for read-only access.
- There are two ways to connect state stores to a processor:
- <ul class="simple">
- <li>By passing the name of a store that has already been added via <code class="docutils literal"><span class="pre">Topology#addStateStore()</span></code> to the corresponding <code class="docutils literal"><span class="pre">KStream#process()</span></code> method call.</li>
- <li>Implementing <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the <code class="docutils literal"><span class="pre">ProcessorSupplier</span></code>
- passed to <code class="docutils literal"><span class="pre">KStream#process()</span></code>. In this case there is no need to call <code class="docutils literal"><span class="pre">StreamsBuilder#addStateStore()</span></code>
- beforehand, the store will be automatically added for you. You can also implement <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the
- <code class="docutils literal"><span class="pre">Value*</span></code> or <code class="docutils literal"><span class="pre">*WithKey</span></code> supplier variants, or <code class="docutils literal"><span class="pre">TransformerSupplier</span></code> or any of its variants.
- </li>
- </ul>
- </div>
- <p>Then we can leverage the <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code> processor in the DSL via <code class="docutils literal"><span class="pre">KStream#process</span></code>.</p>
- <pre class="line-numbers"><code class="language-java">KStream<String, GenericRecord> pageViews = ...;
-
-// Send an email notification when the view count of a page reaches one thousand.
-pageViews.groupByKey()
- .count()
- .filter((PageId pageId, Long viewCount) -> viewCount == 1000)
- // PopularPageEmailAlert is your custom processor that implements the
- // `Processor` interface, see further down below.
- .process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
- </div>
+ <h5 id="traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</h5>
+ <ul>
+ <li><strong>Idea:</strong> A radar monitors cars passing along a road stretch. A system counts the cars for each
+ day, maintaining a cumulative total for the current day in a state store. At the end of the day, the count
+ is
+ emitted and the state is cleared for the next day.</li>
+ <li><strong>Real-World Context:</strong> A car counting system can be useful for determining measures for
+ widening
+ or controlling traffic depending on the number of cars passing through the monitored stretch.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.common.serialization;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessorContext;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.state.Stores;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.format.DateTimeFormatter;
+
+public class TrafficRadarMonitoringCarCountExample {
+ private static final String DAILY_COUNT_STORE = "price-state-store";
+ private static final String DAILY_COUNT_TOPIC = "price-state-topic";
+ private static final String RADAR_COUNT_TOPIC = "car-radar-topic";
+
+ public static void countWithProcessValues(final StreamsBuilder builder) {
+ // Define a state store for tracking daily car counts
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(DAILY_COUNT_STORE),
+ Serdes.String(),
+ Serdes.Long()
+ )
+ );
+ final KStream<Void, String> radarStream = builder.stream(RADAR_COUNT_TOPIC);
+ // Apply the FixedKeyProcessor with the state store
+ radarStream.processValues(DailyCarCountProcessor::new, DAILY_COUNT_STORE)
+ .to(DAILY_COUNT_TOPIC);
+ }
+
+ private static class DailyCarCountProcessor implements FixedKeyProcessor<Void, String, String> {
+ private FixedKeyProcessorContext<Void, String> context;
+ private KeyValueStore<String, Long> stateStore;
+ private static final DateTimeFormatter DATE_FORMATTER =
+ DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(ZoneId.systemDefault());
+
+ @Override
+ public void init(final FixedKeyProcessorContext<Void, String> context) {
+ this.context = context;
+ stateStore = context.getStateStore(DAILY_COUNT_STORE);
+ }
+
+ @Override
+ public void process(final FixedKeyRecord<Void, String> record) {
+ if (record.value() == null) {
+ return; // Skip null events
+ }
+
+ // Derive the current day from the event timestamp
+ final long timestamp = System.currentTimeMillis(); // Use system time for simplicity
+ final String currentDay = DATE_FORMATTER.format(Instant.ofEpochMilli(timestamp));
+ // Retrieve the current count for the day
+ Long dailyCount = stateStore.get(currentDay);
+ if (dailyCount == null) {
+ dailyCount = 0L;
+ }
+ // Increment the count
+ dailyCount++;
+ stateStore.put(currentDay, dailyCount);
+
+ // Emit the current day's count
+ context.forward(record.withValue(String.format("Day: %s, Car Count: %s", currentDay, dailyCount)));
+ }
+ }
+}</code></pre>
+ <h5 id="replacing-slang-in-text-messages">Replacing Slang in Text Messages</h5>
+ <ul>
+ <li><strong>Idea:</strong> A messaging stream contains user-generated content, and you want to replace slang
+ words
+ with their formal equivalents (e.g., "u" becomes "you", "brb" becomes "be
+ right back"). The operation only modifies the message value and keeps the key intact.</li>
+ <li><strong>Real-World Context:</strong> In customer support chat systems, normalizing text by replacing slang
+ with
+ formal equivalents ensures that automated sentiment analysis tools work accurately and provide reliable
+ insights.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualFixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
+
+public class ReplacingSlangTextInMessagesExample {
+ private static final Map<String, String> SLANG_DICTIONARY = Map.of(
+ "u", "you",
+ "brb", "be right back",
+ "omg", "oh my god",
+ "btw", "by the way"
+ );
+ private static final String INPUT_MESSAGES_TOPIC = "input-messages-topic";
+ private static final String OUTPUT_MESSAGES_TOPIC = "output-messages-topic";
+
+ public static void replaceWithProcessValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.processValues(SlangReplacementProcessor::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ private static class SlangReplacementProcessor extends ContextualFixedKeyProcessor<String, String, String> {
+ @Override
+ public void process(final FixedKeyRecord<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+
+ // Replace slang words in the message
+ final String[] words = record.value().split("\\s+");
+ for (final String word : words) {
+ String replacedWord = SLANG_DICTIONARY.getOrDefault(word, word);
+ context().forward(record.withValue(replacedWord));
+ }
+ }
+ }
+}</code></pre>
+ <h4>Keynotes</h4>
+ <ul>
+ <li><strong>Type Safety and Flexibility:</strong> The process and processValues APIs utilize
+ <code>ProcessorContext</code> and <code>Record</code> or <code>FixedKeyRecord</code> objects for better type
+ safety and flexibility of custom processing logic.
+ </li>
+ <li><strong>Clear State and Logic Management:</strong> Implementations for <code>Processor</code> or
+ <code>FixedKeyProcessor</code> should manage state and logic clearly. Use <code>context().forward()</code>
+ for emitting records downstream.
+ </li>
+ <li><strong>Unified API:</strong> Consolidates multiple methods into a single, versatile API.</li>
+ <li><strong>Future-Proof:</strong> Ensures compatibility with the latest Kafka Streams releases.</li>
+ </ul>
+ </div>
+ <div class="section" id="transformers-removal-and-migration-to-processors">
+ <a class="headerlink" href="#transformers-removal-and-migration-to-processors" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id37">Transformers removal and migration to processors</a>
+ </h3>
+ </a>
+ <p>As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>,<br><code>transformValues</code>, <code>flatTransformValues</code>, and
+ <code>process</code> have been removed. These methods have been replaced with the<br>more versatile Processor API.
+ This guide provides detailed steps for migrating existing code to use the new Processor<br>API and explains the
+ benefits of the changes.</p>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ <li><code>KStream#process</code></li>
+ </ul>
+ <p>The Processor API now serves as a unified replacement for all these methods. It simplifies the API surface
+ while<br>maintaining support for both stateless and stateful operations.</p>
+ <h4>Migration Examples</h4>
+ <p>To migrate from the deprecated <code>transform</code>, <code>transformValues</code>, <code>flatTransform</code>, and
+ <code>flatTransformValues</code> methods to the<br>Processor API (PAPI) in Kafka Streams, let's resume the
+ previouss examples. The new <code>process</code> and <code>processValues</code><br>methods enable a more flexible
+ and reusable approach by requiring implementations of the <code>Processor</code>
+ or<br><code>FixedKeyProcessor</code> interfaces.</p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Migrating from</th>
+ <th>Migrating to</th>
+ <th>State Type</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program-removal">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>transform</code></td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity-removal">Categorizing Logs by Severity</a></td>
+ <td><code>flatTransform</code></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count-removal">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>transformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on code in PR #18314:
URL: https://github.com/apache/kafka/pull/18314#discussion_r1933125039
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
Review Comment:
Oh, so I think we have an opportunity to update the javadocs. I have took these description from there.
https://github.com/apache/kafka/blob/f18457f2b8f7adc5fb8ea2c46dcf38973aaf4667/streams/src/main/java/org/apache/kafka/streams/processor/api/ContextualProcessor.java#L21
https://github.com/apache/kafka/blob/f18457f2b8f7adc5fb8ea2c46dcf38973aaf4667/streams/src/main/java/org/apache/kafka/streams/processor/api/ContextualFixedKeyProcessor.java#L22
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on code in PR #18314:
URL: https://github.com/apache/kafka/pull/18314#discussion_r1933121266
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
Review Comment:
Oh, sorry, I have this in mind, but this time has ended up going unnoticed, after Markdown to HTML conversion. I ended up "batch updating".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #18314:
URL: https://github.com/apache/kafka/pull/18314#discussion_r1896960916
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3243,6 +3244,177 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
// `Processor` interface, see further down below.
.process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
</div>
+ <div class="section" id="migrating-from-transform-to-process">
+ <h2>
+ <a class="headerlink" href="#migrating-from-transform-to-process" title="Permalink to this headline">
+ Migrating from transform to process
+ </a>
+ </h2>
+ <div>
+ <p>
+ As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>, <code>transformValues</code>, and <code>flatTransformValues</code>, have
+ been removed. These methods have been replaced with the more versatile <code>process</code> API. This
+ guide provides detailed steps for migrating existing code to use the new <code>process</code> API and
+ explains the benefits of the changes.
+ </p>
+ <h3>Overview of Changes</h3>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ </ul>
+ <p>The <code>process</code> API now serves as a unified replacement for all these methods. It simplifies the
+ API surface while maintaining support for both stateless and stateful operations.</p>
+
+ <h3>Migration Steps</h3>
+
+ <h4>1. Migrate from <code>transform</code> to <code>process</code></h4>
+
+ <pre class="line-numbers"><code class="language-java">// Before
+KStream<String, String> transformedStream = stream.transform(
+ () -> new CustomTransformer(),
+ "state-store"
+);
+
+// After
+KStream<String, String> processedStream = stream.process(
+ () -> new CustomProcessor(),
Review Comment:
It seems switching from `KStream#transform()` to `KStream#process()` is the easy part. I guess the more interesting question is, how to rewrite `CustomTransformer` to `CustomProcessor()` ?
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3243,6 +3244,177 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
// `Processor` interface, see further down below.
.process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
</div>
+ <div class="section" id="migrating-from-transform-to-process">
+ <h2>
+ <a class="headerlink" href="#migrating-from-transform-to-process" title="Permalink to this headline">
+ Migrating from transform to process
+ </a>
+ </h2>
+ <div>
+ <p>
+ As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>, <code>transformValues</code>, and <code>flatTransformValues</code>, have
+ been removed. These methods have been replaced with the more versatile <code>process</code> API. This
+ guide provides detailed steps for migrating existing code to use the new <code>process</code> API and
+ explains the benefits of the changes.
+ </p>
+ <h3>Overview of Changes</h3>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ </ul>
+ <p>The <code>process</code> API now serves as a unified replacement for all these methods. It simplifies the
+ API surface while maintaining support for both stateless and stateful operations.</p>
+
+ <h3>Migration Steps</h3>
+
+ <h4>1. Migrate from <code>transform</code> to <code>process</code></h4>
+
+ <pre class="line-numbers"><code class="language-java">// Before
+KStream<String, String> transformedStream = stream.transform(
+ () -> new CustomTransformer(),
+ "state-store"
+);
+
+// After
+KStream<String, String> processedStream = stream.process(
+ () -> new CustomProcessor(),
+ "state-store"
+);</code></pre>
+
+ <h4>2. Migrate from <code>flatTransform</code> to <code>process</code></h4>
+ <pre class="line-numbers"><code class="language-java">// Before
+KStream<String, String> flatTransformedStream = stream.flatTransform(
+ () -> new CustomFlatTransformer(),
+ "state-store"
+);
+
+// After
+KStream<String, String> processedStream = stream.process(
+ () -> new Processor<String, String>() {
+ @Override
+ public void init(ProcessorContext context) {
+ // Initialization logic here
+ }
+
+ @Override
+ public void process(String key, String value) {
+ Iterable<KeyValue<String, String>> results = customFlatTransformation(key, value);
+ for (KeyValue<String, String> result : results) {
+ context.forward(result.key, result.value);
Review Comment:
Needs an update
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3243,6 +3244,177 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
// `Processor` interface, see further down below.
.process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
</div>
+ <div class="section" id="migrating-from-transform-to-process">
+ <h2>
+ <a class="headerlink" href="#migrating-from-transform-to-process" title="Permalink to this headline">
+ Migrating from transform to process
+ </a>
+ </h2>
+ <div>
+ <p>
+ As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>, <code>transformValues</code>, and <code>flatTransformValues</code>, have
+ been removed. These methods have been replaced with the more versatile <code>process</code> API. This
+ guide provides detailed steps for migrating existing code to use the new <code>process</code> API and
+ explains the benefits of the changes.
+ </p>
+ <h3>Overview of Changes</h3>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ </ul>
+ <p>The <code>process</code> API now serves as a unified replacement for all these methods. It simplifies the
+ API surface while maintaining support for both stateless and stateful operations.</p>
+
+ <h3>Migration Steps</h3>
+
+ <h4>1. Migrate from <code>transform</code> to <code>process</code></h4>
+
+ <pre class="line-numbers"><code class="language-java">// Before
+KStream<String, String> transformedStream = stream.transform(
+ () -> new CustomTransformer(),
+ "state-store"
+);
+
+// After
+KStream<String, String> processedStream = stream.process(
+ () -> new CustomProcessor(),
+ "state-store"
+);</code></pre>
+
+ <h4>2. Migrate from <code>flatTransform</code> to <code>process</code></h4>
+ <pre class="line-numbers"><code class="language-java">// Before
+KStream<String, String> flatTransformedStream = stream.flatTransform(
+ () -> new CustomFlatTransformer(),
+ "state-store"
+);
+
+// After
+KStream<String, String> processedStream = stream.process(
+ () -> new Processor<String, String>() {
+ @Override
+ public void init(ProcessorContext context) {
+ // Initialization logic here
+ }
+
+ @Override
+ public void process(String key, String value) {
Review Comment:
```suggestion
public void process(Record<String, String> record) {
```
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3243,6 +3244,177 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
// `Processor` interface, see further down below.
.process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
</div>
+ <div class="section" id="migrating-from-transform-to-process">
+ <h2>
+ <a class="headerlink" href="#migrating-from-transform-to-process" title="Permalink to this headline">
+ Migrating from transform to process
+ </a>
+ </h2>
+ <div>
+ <p>
+ As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>, <code>transformValues</code>, and <code>flatTransformValues</code>, have
+ been removed. These methods have been replaced with the more versatile <code>process</code> API. This
+ guide provides detailed steps for migrating existing code to use the new <code>process</code> API and
+ explains the benefits of the changes.
+ </p>
+ <h3>Overview of Changes</h3>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ </ul>
+ <p>The <code>process</code> API now serves as a unified replacement for all these methods. It simplifies the
+ API surface while maintaining support for both stateless and stateful operations.</p>
+
+ <h3>Migration Steps</h3>
+
+ <h4>1. Migrate from <code>transform</code> to <code>process</code></h4>
+
+ <pre class="line-numbers"><code class="language-java">// Before
+KStream<String, String> transformedStream = stream.transform(
+ () -> new CustomTransformer(),
+ "state-store"
+);
+
+// After
+KStream<String, String> processedStream = stream.process(
+ () -> new CustomProcessor(),
+ "state-store"
+);</code></pre>
+
+ <h4>2. Migrate from <code>flatTransform</code> to <code>process</code></h4>
+ <pre class="line-numbers"><code class="language-java">// Before
+KStream<String, String> flatTransformedStream = stream.flatTransform(
+ () -> new CustomFlatTransformer(),
+ "state-store"
+);
+
+// After
+KStream<String, String> processedStream = stream.process(
+ () -> new Processor<String, String>() {
Review Comment:
New new `api.Processor` has 4 generic, key/value in/out.
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3243,6 +3244,177 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
// `Processor` interface, see further down below.
.process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
</div>
+ <div class="section" id="migrating-from-transform-to-process">
+ <h2>
+ <a class="headerlink" href="#migrating-from-transform-to-process" title="Permalink to this headline">
+ Migrating from transform to process
+ </a>
+ </h2>
+ <div>
+ <p>
+ As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>, <code>transformValues</code>, and <code>flatTransformValues</code>, have
+ been removed. These methods have been replaced with the more versatile <code>process</code> API. This
+ guide provides detailed steps for migrating existing code to use the new <code>process</code> API and
+ explains the benefits of the changes.
+ </p>
+ <h3>Overview of Changes</h3>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ </ul>
+ <p>The <code>process</code> API now serves as a unified replacement for all these methods. It simplifies the
+ API surface while maintaining support for both stateless and stateful operations.</p>
+
+ <h3>Migration Steps</h3>
+
+ <h4>1. Migrate from <code>transform</code> to <code>process</code></h4>
+
+ <pre class="line-numbers"><code class="language-java">// Before
+KStream<String, String> transformedStream = stream.transform(
+ () -> new CustomTransformer(),
+ "state-store"
+);
+
+// After
+KStream<String, String> processedStream = stream.process(
+ () -> new CustomProcessor(),
+ "state-store"
+);</code></pre>
+
+ <h4>2. Migrate from <code>flatTransform</code> to <code>process</code></h4>
+ <pre class="line-numbers"><code class="language-java">// Before
+KStream<String, String> flatTransformedStream = stream.flatTransform(
+ () -> new CustomFlatTransformer(),
+ "state-store"
+);
+
+// After
+KStream<String, String> processedStream = stream.process(
+ () -> new Processor<String, String>() {
+ @Override
+ public void init(ProcessorContext context) {
+ // Initialization logic here
+ }
+
+ @Override
+ public void process(String key, String value) {
+ Iterable<KeyValue<String, String>> results = customFlatTransformation(key, value);
Review Comment:
I think it would be difficult for user to reason what `customFlatTransformation` is?
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3243,6 +3244,177 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
// `Processor` interface, see further down below.
.process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
</div>
+ <div class="section" id="migrating-from-transform-to-process">
+ <h2>
+ <a class="headerlink" href="#migrating-from-transform-to-process" title="Permalink to this headline">
+ Migrating from transform to process
+ </a>
+ </h2>
+ <div>
+ <p>
+ As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>, <code>transformValues</code>, and <code>flatTransformValues</code>, have
+ been removed. These methods have been replaced with the more versatile <code>process</code> API. This
+ guide provides detailed steps for migrating existing code to use the new <code>process</code> API and
+ explains the benefits of the changes.
+ </p>
+ <h3>Overview of Changes</h3>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ </ul>
+ <p>The <code>process</code> API now serves as a unified replacement for all these methods. It simplifies the
+ API surface while maintaining support for both stateless and stateful operations.</p>
+
+ <h3>Migration Steps</h3>
+
+ <h4>1. Migrate from <code>transform</code> to <code>process</code></h4>
+
+ <pre class="line-numbers"><code class="language-java">// Before
+KStream<String, String> transformedStream = stream.transform(
+ () -> new CustomTransformer(),
+ "state-store"
+);
+
+// After
+KStream<String, String> processedStream = stream.process(
+ () -> new CustomProcessor(),
+ "state-store"
+);</code></pre>
+
+ <h4>2. Migrate from <code>flatTransform</code> to <code>process</code></h4>
+ <pre class="line-numbers"><code class="language-java">// Before
+KStream<String, String> flatTransformedStream = stream.flatTransform(
+ () -> new CustomFlatTransformer(),
+ "state-store"
+);
+
+// After
+KStream<String, String> processedStream = stream.process(
+ () -> new Processor<String, String>() {
+ @Override
+ public void init(ProcessorContext context) {
+ // Initialization logic here
+ }
+
+ @Override
+ public void process(String key, String value) {
+ Iterable<KeyValue<String, String>> results = customFlatTransformation(key, value);
+ for (KeyValue<String, String> result : results) {
+ context.forward(result.key, result.value);
+ }
+ }
+
+ @Override
+ public void close() {
Review Comment:
Can be omitted as it's empty.
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3243,6 +3244,177 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
// `Processor` interface, see further down below.
.process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
</div>
+ <div class="section" id="migrating-from-transform-to-process">
+ <h2>
+ <a class="headerlink" href="#migrating-from-transform-to-process" title="Permalink to this headline">
+ Migrating from transform to process
+ </a>
+ </h2>
+ <div>
+ <p>
+ As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>, <code>transformValues</code>, and <code>flatTransformValues</code>, have
+ been removed. These methods have been replaced with the more versatile <code>process</code> API. This
+ guide provides detailed steps for migrating existing code to use the new <code>process</code> API and
+ explains the benefits of the changes.
+ </p>
+ <h3>Overview of Changes</h3>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ </ul>
+ <p>The <code>process</code> API now serves as a unified replacement for all these methods. It simplifies the
+ API surface while maintaining support for both stateless and stateful operations.</p>
+
+ <h3>Migration Steps</h3>
+
+ <h4>1. Migrate from <code>transform</code> to <code>process</code></h4>
+
+ <pre class="line-numbers"><code class="language-java">// Before
+KStream<String, String> transformedStream = stream.transform(
+ () -> new CustomTransformer(),
+ "state-store"
+);
+
+// After
+KStream<String, String> processedStream = stream.process(
+ () -> new CustomProcessor(),
+ "state-store"
+);</code></pre>
+
+ <h4>2. Migrate from <code>flatTransform</code> to <code>process</code></h4>
+ <pre class="line-numbers"><code class="language-java">// Before
+KStream<String, String> flatTransformedStream = stream.flatTransform(
+ () -> new CustomFlatTransformer(),
+ "state-store"
+);
+
+// After
+KStream<String, String> processedStream = stream.process(
+ () -> new Processor<String, String>() {
+ @Override
+ public void init(ProcessorContext context) {
+ // Initialization logic here
Review Comment:
Might be good to just fill in the code? We need to store a reference to `context`
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3243,6 +3244,177 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
// `Processor` interface, see further down below.
.process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
</div>
+ <div class="section" id="migrating-from-transform-to-process">
+ <h2>
+ <a class="headerlink" href="#migrating-from-transform-to-process" title="Permalink to this headline">
+ Migrating from transform to process
+ </a>
+ </h2>
+ <div>
+ <p>
+ As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>, <code>transformValues</code>, and <code>flatTransformValues</code>, have
+ been removed. These methods have been replaced with the more versatile <code>process</code> API. This
+ guide provides detailed steps for migrating existing code to use the new <code>process</code> API and
+ explains the benefits of the changes.
+ </p>
+ <h3>Overview of Changes</h3>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ </ul>
+ <p>The <code>process</code> API now serves as a unified replacement for all these methods. It simplifies the
+ API surface while maintaining support for both stateless and stateful operations.</p>
+
+ <h3>Migration Steps</h3>
+
+ <h4>1. Migrate from <code>transform</code> to <code>process</code></h4>
+
+ <pre class="line-numbers"><code class="language-java">// Before
+KStream<String, String> transformedStream = stream.transform(
+ () -> new CustomTransformer(),
+ "state-store"
+);
+
+// After
+KStream<String, String> processedStream = stream.process(
+ () -> new CustomProcessor(),
+ "state-store"
+);</code></pre>
+
+ <h4>2. Migrate from <code>flatTransform</code> to <code>process</code></h4>
+ <pre class="line-numbers"><code class="language-java">// Before
+KStream<String, String> flatTransformedStream = stream.flatTransform(
+ () -> new CustomFlatTransformer(),
+ "state-store"
+);
+
+// After
+KStream<String, String> processedStream = stream.process(
+ () -> new Processor<String, String>() {
+ @Override
+ public void init(ProcessorContext context) {
Review Comment:
The new `api.ProcessorContext` has `<KOut, VOut>` generic types.
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3243,6 +3244,177 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
// `Processor` interface, see further down below.
.process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
</div>
+ <div class="section" id="migrating-from-transform-to-process">
+ <h2>
+ <a class="headerlink" href="#migrating-from-transform-to-process" title="Permalink to this headline">
+ Migrating from transform to process
+ </a>
+ </h2>
+ <div>
+ <p>
+ As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>, <code>transformValues</code>, and <code>flatTransformValues</code>, have
+ been removed. These methods have been replaced with the more versatile <code>process</code> API. This
+ guide provides detailed steps for migrating existing code to use the new <code>process</code> API and
+ explains the benefits of the changes.
+ </p>
+ <h3>Overview of Changes</h3>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ </ul>
+ <p>The <code>process</code> API now serves as a unified replacement for all these methods. It simplifies the
+ API surface while maintaining support for both stateless and stateful operations.</p>
+
+ <h3>Migration Steps</h3>
+
+ <h4>1. Migrate from <code>transform</code> to <code>process</code></h4>
+
+ <pre class="line-numbers"><code class="language-java">// Before
+KStream<String, String> transformedStream = stream.transform(
+ () -> new CustomTransformer(),
+ "state-store"
+);
+
+// After
+KStream<String, String> processedStream = stream.process(
+ () -> new CustomProcessor(),
+ "state-store"
+);</code></pre>
+
+ <h4>2. Migrate from <code>flatTransform</code> to <code>process</code></h4>
+ <pre class="line-numbers"><code class="language-java">// Before
+KStream<String, String> flatTransformedStream = stream.flatTransform(
+ () -> new CustomFlatTransformer(),
+ "state-store"
+);
+
+// After
+KStream<String, String> processedStream = stream.process(
+ () -> new Processor<String, String>() {
+ @Override
+ public void init(ProcessorContext context) {
+ // Initialization logic here
+ }
+
+ @Override
+ public void process(String key, String value) {
+ Iterable<KeyValue<String, String>> results = customFlatTransformation(key, value);
+ for (KeyValue<String, String> result : results) {
+ context.forward(result.key, result.value);
+ }
+ }
+
+ @Override
+ public void close() {
+ // Cleanup logic here
+ }
+ },
+ "state-store"
+);</code></pre>
+
+ <h4>3. Migrate from <code>transformValues</code> to <code>process</code></h4>
+
+ <pre class="line-numbers"><code class="language-java">// Before
+KStream<String, String> transformedValuesStream = stream.transformValues(
+ () -> new ValueTransformer<String, String>() {
+ @Override
+ public void init(ProcessorContext context) {
+ // Initialization logic
+ }
+
+ @Override
+ public String transform(String value) {
+ return value.toUpperCase();
+ }
+
+ @Override
+ public void close() {
+ // Cleanup logic
+ }
+ },
+ "state-store"
+);
+
+// After
+KStream<String, String> processedStream = stream.process(
+ () -> new Processor<String, String>() {
+ @Override
+ public void init(ProcessorContext context) {
+ // Initialization logic
+ }
+
+ @Override
+ public void process(String key, String value) {
+ String transformedValue = value.toUpperCase();
+ context.forward(key, transformedValue);
+ }
+
+ @Override
+ public void close() {
+ // Cleanup logic
+ }
+ },
+ "state-store"
+);</code></pre>
+
+ <h4>4. Migrate from <code>flatTransformValues</code> to <code>process</code></h4>
+
+ <pre class="line-numbers"><code class="language-java">// Before
+KStream<String> flatTransformedValuesStream = stream.flatTransformValues(
+ () -> new ValueTransformerWithKey<String, String, Iterable<String>>() {
+ @Override
+ public void init(ProcessorContext context) {
+ // Initialization logic
+ }
+
+ @Override
+ public Iterable<String> transform(String readOnlyKey, String value) {
+ return Arrays.asList(value.split(","));
+ }
+
+ @Override
+ public void close() {
+ // Cleanup logic
+ }
+ },
+ "state-store"
+);
+
+// After
+KStream<String, String> processedStream = stream.process(
+ () -> new Processor<String, String>() {
+ @Override
+ public void init(ProcessorContext context) {
+ // Initialization logic
+ }
+
+ @Override
+ public void process(String key, String value) {
+ for (String transformedValue : value.split(",")) {
+ context.forward(key, transformedValue);
+ }
+ }
+
+ @Override
+ public void close() {
+ // Cleanup logic
+ }
+ },
+ "state-store"
+);</code></pre>
+
+ <h3>Benefits of Migrating to <code>process</code></h3>
+ <ul>
+ <li><strong>Unified API:</strong> Consolidates multiple methods into a single, versatile API.</li>
+ <li><strong>Flexibility:</strong> Simplifies the implementation of custom processing logic.</li>
+ <li><strong>Future-Proof:</strong> Ensures compatibility with the latest Kafka Streams releases.</li>
Review Comment:
I would add "Improved type-safty" that the "new PAPI" offers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on PR #18314:
URL: https://github.com/apache/kafka/pull/18314#issuecomment-2614460950
@mjsax, I should not have expressed myself very well, sorry. By Friday, I mean I had resolved most of the conversations you pointed out, with the exception of the _update migrating/applying section (the first conversation)_ and the `business logic of slang example (the last conversation)`. I marked the others as resolved and updated them in-place in the original Markdown I wrote as draft.
Now, I have split the _Migrating/Applying section_, the first one being only for applying and the second one being only for migrating. See:
First:

Second:

I will rewrite this Markdown draft into HTML now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #18314:
URL: https://github.com/apache/kafka/pull/18314#discussion_r1913737465
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -64,7 +64,7 @@
</li>
</ul>
</li>
- <li><a class="reference internal" href="#applying-processors-and-transformers-processor-api-integration" id="id24">Applying processors and transformers (Processor API integration)</a></li>
+ <li><a class="reference internal" href="#migrating-from-transform-methods-to-processor-api-papi" id="id37">Migrating from transform Methods to Processor API (PAPI)</a></li>
Review Comment:
Would it be better to update the old section to `Applying processors (Processor API integration)` and add a new section `Migrating from transform Methods to Processor API (PAPI)` ?
We would eventually remove the "migration section" (and maybe even back-port to 3.9/3.8 docs, with some adjustments), but keep the updated "PAPI integration".
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3097,615 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
- </ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
- </thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- </tbody>
- </table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <div class="section" id="migrating-from-transform-methods-to-processor-api-papi">
Review Comment:
Why is the indention change?
The last line before is
```
</div>
```
But is one is only
```
<div class="section" id="migrating-from-transform-methods-to-processor-api-papi">
```
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3097,615 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
- </ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
- </thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- </tbody>
- </table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <div class="section" id="migrating-from-transform-methods-to-processor-api-papi">
+ <h2>
+ <a class="headerlink" href="#migrating-from-transform-methods-to-processor-api-papi"
+ title="Permalink to this headline">
+ Migrating from transform Methods to Processor API (PAPI)
+ </a>
+ </h2>
+ <h3>Overview of Changes</h3>
+ <p>
+ As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>, <code>transformValues</code>, and <code>flatTransformValues</code>, have
+ been removed. These methods have been replaced with the more versatile Processor API. This
+ guide provides detailed steps for migrating existing code to use the new Processor API and
+ explains the benefits of the changes.
+ </p>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ </ul>
+ <p>The Processor API now serves as a unified replacement for all these methods. It simplifies the
+ API surface while maintaining support for both stateless and stateful operations.</p>
+ <h3>Migration Process</h3>
+ <p>The migration process consists of:</p>
+ <ol>
+ <li>
+ Replace <code>Transformer</code> with <code>Processor</code> or <code>ValueTransformer</code> with
+ <code>FixedKeyProcessor</code>;
+ </li>
+ <li>
+ Replace record <code>key</code> and <code>value</code> with <code>Record</code> or <code>FixedKeyRecord</code>;
+ </li>
+ <li>
+ Rewrite the <code>transform</code> method of <code>Transformer</code> and <code>ValueTransformer</code> as
+ <code>process</code> or <code>processValues</code>;
+ </li>
+ <li>
+ Use the new <code>Record</code> or <code>FixedKeyRecord</code> as argument of the renamed method;</li>
+ <li>
+ Rewrite the return type of the renamed method to <code>void</code> and forward the record through the context;
+ and finally
+ </li>
+ <li>
+ Change the <code>KStream</code> call of the <code>transform</code> method to <code>process</code> or
+ <code>processValues</code>.
+ </li>
+ </ol>
+ <h3>Migration Examples</h3>
+ <p>
+ To migrate from the deprecated <code>transform</code>, <code>transformValues</code>, <code>flatTransform</code>, and
+ <code>flatTransformValues</code> methods to the Process API (PAPI) in Kafka Streams, follow these examples. The new
+ <code>process</code> and <code>processValues</code> APIs enable a more flexible and reusable approach by requiring
+ implementations of the <code>Processor</code> or <code>FixedKeyProcessor</code> interfaces.
+ </p>
+ <p>Here are examples to help you migrate:</p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Migrating from</th>
+ <th>Migrating to</th>
+ <th>State Type</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a></td>
+ <td><code>transform</code></td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>flatTransform</code></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>transformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>flatTransformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
+ </tbody>
+ </table>
+ <h4>Stateless Examples</h4>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li>
+ <strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g., INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to another processor.
Review Comment:
> The rest (WARN) are forwarded to another processor.
I don't see this in the example? Seem `WARN` is going into some other topic, using a `TopicNameExtractor` ?
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3097,615 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
- </ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
- </thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- </tbody>
- </table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <div class="section" id="migrating-from-transform-methods-to-processor-api-papi">
+ <h2>
+ <a class="headerlink" href="#migrating-from-transform-methods-to-processor-api-papi"
+ title="Permalink to this headline">
+ Migrating from transform Methods to Processor API (PAPI)
+ </a>
+ </h2>
+ <h3>Overview of Changes</h3>
+ <p>
+ As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>, <code>transformValues</code>, and <code>flatTransformValues</code>, have
+ been removed. These methods have been replaced with the more versatile Processor API. This
+ guide provides detailed steps for migrating existing code to use the new Processor API and
+ explains the benefits of the changes.
+ </p>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ </ul>
+ <p>The Processor API now serves as a unified replacement for all these methods. It simplifies the
+ API surface while maintaining support for both stateless and stateful operations.</p>
+ <h3>Migration Process</h3>
+ <p>The migration process consists of:</p>
+ <ol>
+ <li>
+ Replace <code>Transformer</code> with <code>Processor</code> or <code>ValueTransformer</code> with
+ <code>FixedKeyProcessor</code>;
+ </li>
+ <li>
+ Replace record <code>key</code> and <code>value</code> with <code>Record</code> or <code>FixedKeyRecord</code>;
+ </li>
+ <li>
+ Rewrite the <code>transform</code> method of <code>Transformer</code> and <code>ValueTransformer</code> as
+ <code>process</code> or <code>processValues</code>;
+ </li>
+ <li>
+ Use the new <code>Record</code> or <code>FixedKeyRecord</code> as argument of the renamed method;</li>
+ <li>
+ Rewrite the return type of the renamed method to <code>void</code> and forward the record through the context;
+ and finally
+ </li>
+ <li>
+ Change the <code>KStream</code> call of the <code>transform</code> method to <code>process</code> or
+ <code>processValues</code>.
+ </li>
+ </ol>
+ <h3>Migration Examples</h3>
+ <p>
+ To migrate from the deprecated <code>transform</code>, <code>transformValues</code>, <code>flatTransform</code>, and
+ <code>flatTransformValues</code> methods to the Process API (PAPI) in Kafka Streams, follow these examples. The new
+ <code>process</code> and <code>processValues</code> APIs enable a more flexible and reusable approach by requiring
+ implementations of the <code>Processor</code> or <code>FixedKeyProcessor</code> interfaces.
+ </p>
+ <p>Here are examples to help you migrate:</p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Migrating from</th>
+ <th>Migrating to</th>
+ <th>State Type</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a></td>
+ <td><code>transform</code></td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>flatTransform</code></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>transformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>flatTransformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
+ </tbody>
+ </table>
+ <h4>Stateless Examples</h4>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li>
+ <strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g., INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to another processor.
+ </li>
+ <li>
+ <strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks, and
+ INFO logs are stored for basic reporting purposes.
+ </li>
+ </ul>
+ <p>
+ Below, methods <code>categorizeWithFlatTransform</code> and <code>categorizeWithProcess</code> show how you can
+ migrate from <code>flatTransform</code> to <code>process</code>.
+ </p>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithFlatTransform(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.flatTransform(() -> new LogSeverityTransformer())
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new);
Review Comment:
`to(...)` is missing
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3097,615 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
- </ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
- </thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- </tbody>
- </table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <div class="section" id="migrating-from-transform-methods-to-processor-api-papi">
+ <h2>
+ <a class="headerlink" href="#migrating-from-transform-methods-to-processor-api-papi"
+ title="Permalink to this headline">
+ Migrating from transform Methods to Processor API (PAPI)
+ </a>
+ </h2>
+ <h3>Overview of Changes</h3>
+ <p>
+ As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>, <code>transformValues</code>, and <code>flatTransformValues</code>, have
+ been removed. These methods have been replaced with the more versatile Processor API. This
+ guide provides detailed steps for migrating existing code to use the new Processor API and
+ explains the benefits of the changes.
+ </p>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ </ul>
+ <p>The Processor API now serves as a unified replacement for all these methods. It simplifies the
+ API surface while maintaining support for both stateless and stateful operations.</p>
+ <h3>Migration Process</h3>
+ <p>The migration process consists of:</p>
+ <ol>
+ <li>
+ Replace <code>Transformer</code> with <code>Processor</code> or <code>ValueTransformer</code> with
+ <code>FixedKeyProcessor</code>;
+ </li>
+ <li>
+ Replace record <code>key</code> and <code>value</code> with <code>Record</code> or <code>FixedKeyRecord</code>;
+ </li>
+ <li>
+ Rewrite the <code>transform</code> method of <code>Transformer</code> and <code>ValueTransformer</code> as
+ <code>process</code> or <code>processValues</code>;
+ </li>
+ <li>
+ Use the new <code>Record</code> or <code>FixedKeyRecord</code> as argument of the renamed method;</li>
+ <li>
+ Rewrite the return type of the renamed method to <code>void</code> and forward the record through the context;
+ and finally
+ </li>
+ <li>
+ Change the <code>KStream</code> call of the <code>transform</code> method to <code>process</code> or
+ <code>processValues</code>.
+ </li>
+ </ol>
+ <h3>Migration Examples</h3>
+ <p>
+ To migrate from the deprecated <code>transform</code>, <code>transformValues</code>, <code>flatTransform</code>, and
+ <code>flatTransformValues</code> methods to the Process API (PAPI) in Kafka Streams, follow these examples. The new
+ <code>process</code> and <code>processValues</code> APIs enable a more flexible and reusable approach by requiring
+ implementations of the <code>Processor</code> or <code>FixedKeyProcessor</code> interfaces.
+ </p>
+ <p>Here are examples to help you migrate:</p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Migrating from</th>
+ <th>Migrating to</th>
+ <th>State Type</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a></td>
+ <td><code>transform</code></td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>flatTransform</code></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>transformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>flatTransformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
+ </tbody>
+ </table>
+ <h4>Stateless Examples</h4>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li>
+ <strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g., INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to another processor.
+ </li>
+ <li>
+ <strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks, and
+ INFO logs are stored for basic reporting purposes.
+ </li>
+ </ul>
+ <p>
+ Below, methods <code>categorizeWithFlatTransform</code> and <code>categorizeWithProcess</code> show how you can
+ migrate from <code>flatTransform</code> to <code>process</code>.
+ </p>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithFlatTransform(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.flatTransform(() -> new LogSeverityTransformer())
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new);
+ }
- @Override
- void process(Record<PageId, Long> record) {
- // Here you would format and send the alert email.
- //
- // In this specific example, you would be able to include
- // information about the page's ID and its view count
- }
+ private static class LogSeverityTransformer implements Transformer<String, String, Iterable<KeyValue<String, String>>> {
+ @Override
+ public void init(org.apache.kafka.streams.processor.ProcessorContext context) {
+ }
- @Override
- void close() {
- // Any code for clean up would go here, for example tearing down the email client and anything
- // else you created in the init() method
- // This processor instance will not be used again after this call.
- }
+ @Override
+ public Iterable<KeyValue<String, String>> transform(String key, String value) {
+ if (value == null) {
+ return Collections.emptyList(); // Skip null values
+ }
-}</code></pre>
- <div class="admonition tip">
- <p><b>Tip</b></p>
- <p class="last">Even though we do not demonstrate it in this example, a stream processor can access any available state stores by
- calling <code class="docutils literal"><span class="pre">ProcessorContext#getStateStore()</span></code>.
- State stores are only available if they have been connected to the processor, or if they are global stores. While global stores do not need to be connected explicitly, they only allow for read-only access.
- There are two ways to connect state stores to a processor:
- <ul class="simple">
- <li>By passing the name of a store that has already been added via <code class="docutils literal"><span class="pre">Topology#addStateStore()</span></code> to the corresponding <code class="docutils literal"><span class="pre">KStream#process()</span></code> method call.</li>
- <li>Implementing <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the <code class="docutils literal"><span class="pre">ProcessorSupplier</span></code>
- passed to <code class="docutils literal"><span class="pre">KStream#process()</span></code>. In this case there is no need to call <code class="docutils literal"><span class="pre">StreamsBuilder#addStateStore()</span></code>
- beforehand, the store will be automatically added for you. You can also implement <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the
- <code class="docutils literal"><span class="pre">Value*</span></code> or <code class="docutils literal"><span class="pre">*WithKey</span></code> supplier variants, or <code class="docutils literal"><span class="pre">TransformerSupplier</span></code> or any of its variants.
- </li>
- </ul>
- </div>
- <p>Then we can leverage the <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code> processor in the DSL via <code class="docutils literal"><span class="pre">KStream#process</span></code>.</p>
- <pre class="line-numbers"><code class="language-java">KStream<String, GenericRecord> pageViews = ...;
-
-// Send an email notification when the view count of a page reaches one thousand.
-pageViews.groupByKey()
- .count()
- .filter((PageId pageId, Long viewCount) -> viewCount == 1000)
- // PopularPageEmailAlert is your custom processor that implements the
- // `Processor` interface, see further down below.
- .process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
- </div>
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ int colonIndex = value.indexOf(':');
+ String severity = colonIndex > 0 ? value.substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Create appropriate KeyValue pair based on severity
+ return switch (severity) {
+ case "ERROR" -> List.of(new KeyValue<>("ERROR", value));
+ case "WARN" -> List.of(new KeyValue<>("WARN", value));
+ case "INFO" -> Collections.emptyList(); // INFO logs are ignored
+ default -> List.of(new KeyValue<>("UNKNOWN", value));
+ };
+ }
+
+ @Override
+ public void close() {
+ }
+ }
+
+ private static class LogSeverityProcessor implements Processor<String, String, String, String> {
Review Comment:
We could use `ContextualProcessor` instead? Thoughts? (would also us to simplify the code below as we would get the `context` set automatically.
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3097,615 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
- </ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
- </thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- </tbody>
- </table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <div class="section" id="migrating-from-transform-methods-to-processor-api-papi">
+ <h2>
+ <a class="headerlink" href="#migrating-from-transform-methods-to-processor-api-papi"
+ title="Permalink to this headline">
+ Migrating from transform Methods to Processor API (PAPI)
+ </a>
+ </h2>
+ <h3>Overview of Changes</h3>
+ <p>
+ As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>, <code>transformValues</code>, and <code>flatTransformValues</code>, have
+ been removed. These methods have been replaced with the more versatile Processor API. This
+ guide provides detailed steps for migrating existing code to use the new Processor API and
+ explains the benefits of the changes.
+ </p>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ </ul>
+ <p>The Processor API now serves as a unified replacement for all these methods. It simplifies the
+ API surface while maintaining support for both stateless and stateful operations.</p>
+ <h3>Migration Process</h3>
+ <p>The migration process consists of:</p>
+ <ol>
+ <li>
+ Replace <code>Transformer</code> with <code>Processor</code> or <code>ValueTransformer</code> with
+ <code>FixedKeyProcessor</code>;
+ </li>
+ <li>
+ Replace record <code>key</code> and <code>value</code> with <code>Record</code> or <code>FixedKeyRecord</code>;
+ </li>
+ <li>
+ Rewrite the <code>transform</code> method of <code>Transformer</code> and <code>ValueTransformer</code> as
+ <code>process</code> or <code>processValues</code>;
+ </li>
+ <li>
+ Use the new <code>Record</code> or <code>FixedKeyRecord</code> as argument of the renamed method;</li>
+ <li>
+ Rewrite the return type of the renamed method to <code>void</code> and forward the record through the context;
+ and finally
+ </li>
+ <li>
+ Change the <code>KStream</code> call of the <code>transform</code> method to <code>process</code> or
+ <code>processValues</code>.
+ </li>
+ </ol>
+ <h3>Migration Examples</h3>
+ <p>
+ To migrate from the deprecated <code>transform</code>, <code>transformValues</code>, <code>flatTransform</code>, and
+ <code>flatTransformValues</code> methods to the Process API (PAPI) in Kafka Streams, follow these examples. The new
+ <code>process</code> and <code>processValues</code> APIs enable a more flexible and reusable approach by requiring
+ implementations of the <code>Processor</code> or <code>FixedKeyProcessor</code> interfaces.
+ </p>
+ <p>Here are examples to help you migrate:</p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Migrating from</th>
+ <th>Migrating to</th>
+ <th>State Type</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a></td>
+ <td><code>transform</code></td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>flatTransform</code></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>transformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>flatTransformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
+ </tbody>
+ </table>
+ <h4>Stateless Examples</h4>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li>
+ <strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g., INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to another processor.
+ </li>
+ <li>
+ <strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks, and
+ INFO logs are stored for basic reporting purposes.
+ </li>
+ </ul>
+ <p>
+ Below, methods <code>categorizeWithFlatTransform</code> and <code>categorizeWithProcess</code> show how you can
+ migrate from <code>flatTransform</code> to <code>process</code>.
+ </p>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithFlatTransform(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.flatTransform(() -> new LogSeverityTransformer())
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new);
+ }
- @Override
- void process(Record<PageId, Long> record) {
- // Here you would format and send the alert email.
- //
- // In this specific example, you would be able to include
- // information about the page's ID and its view count
- }
+ private static class LogSeverityTransformer implements Transformer<String, String, Iterable<KeyValue<String, String>>> {
+ @Override
+ public void init(org.apache.kafka.streams.processor.ProcessorContext context) {
+ }
- @Override
- void close() {
- // Any code for clean up would go here, for example tearing down the email client and anything
- // else you created in the init() method
- // This processor instance will not be used again after this call.
- }
+ @Override
+ public Iterable<KeyValue<String, String>> transform(String key, String value) {
+ if (value == null) {
+ return Collections.emptyList(); // Skip null values
+ }
-}</code></pre>
- <div class="admonition tip">
- <p><b>Tip</b></p>
- <p class="last">Even though we do not demonstrate it in this example, a stream processor can access any available state stores by
- calling <code class="docutils literal"><span class="pre">ProcessorContext#getStateStore()</span></code>.
- State stores are only available if they have been connected to the processor, or if they are global stores. While global stores do not need to be connected explicitly, they only allow for read-only access.
- There are two ways to connect state stores to a processor:
- <ul class="simple">
- <li>By passing the name of a store that has already been added via <code class="docutils literal"><span class="pre">Topology#addStateStore()</span></code> to the corresponding <code class="docutils literal"><span class="pre">KStream#process()</span></code> method call.</li>
- <li>Implementing <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the <code class="docutils literal"><span class="pre">ProcessorSupplier</span></code>
- passed to <code class="docutils literal"><span class="pre">KStream#process()</span></code>. In this case there is no need to call <code class="docutils literal"><span class="pre">StreamsBuilder#addStateStore()</span></code>
- beforehand, the store will be automatically added for you. You can also implement <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the
- <code class="docutils literal"><span class="pre">Value*</span></code> or <code class="docutils literal"><span class="pre">*WithKey</span></code> supplier variants, or <code class="docutils literal"><span class="pre">TransformerSupplier</span></code> or any of its variants.
- </li>
- </ul>
- </div>
- <p>Then we can leverage the <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code> processor in the DSL via <code class="docutils literal"><span class="pre">KStream#process</span></code>.</p>
- <pre class="line-numbers"><code class="language-java">KStream<String, GenericRecord> pageViews = ...;
-
-// Send an email notification when the view count of a page reaches one thousand.
-pageViews.groupByKey()
- .count()
- .filter((PageId pageId, Long viewCount) -> viewCount == 1000)
- // PopularPageEmailAlert is your custom processor that implements the
- // `Processor` interface, see further down below.
- .process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
- </div>
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ int colonIndex = value.indexOf(':');
+ String severity = colonIndex > 0 ? value.substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Create appropriate KeyValue pair based on severity
+ return switch (severity) {
+ case "ERROR" -> List.of(new KeyValue<>("ERROR", value));
+ case "WARN" -> List.of(new KeyValue<>("WARN", value));
+ case "INFO" -> Collections.emptyList(); // INFO logs are ignored
+ default -> List.of(new KeyValue<>("UNKNOWN", value));
+ };
+ }
+
+ @Override
+ public void close() {
+ }
+ }
+
+ private static class LogSeverityProcessor implements Processor<String, String, String, String> {
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ }
+
+ @Override
+ public void process(final Record<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ final int colonIndex = record.value().indexOf(':');
+ final String severity = colonIndex > 0 ? record.value().substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Route logs based on severity
+ switch (severity) {
+ case "ERROR":
+ context.forward(new Record<>(ERROR_LOGS_TOPIC, record.value(), record.timestamp()));
+ break;
+ case "WARN":
+ context.forward(new Record<>(WARN_LOGS_TOPIC, record.value(), record.timestamp()));
+ break;
+ case "INFO":
+ // INFO logs are ignored
+ break;
+ default:
+ // Forward to an "unknown" topic for logs with unrecognized severities
+ context.forward(new Record<>(UNKNOWN_LOGS_TOPIC, record.value(), record.timestamp()));
+ }
+ }
+ }
+}
+</code></pre>
+ <h5 id="replacing-slang-in-text-messages">Replacing Slang in Text Messages</h5>
+ <ul>
+ <li>
+ <strong>Idea:</strong> A messaging stream contains user-generated content, and you want to replace slang words
+ with their formal equivalents (e.g., "u" becomes "you", "brb" becomes "be
+ right back"). The operation only modifies the message value and keeps the key intact.
+ </li>
+ <li>
+ <strong>Real-World Context:</strong> In customer support chat systems, normalizing text by replacing slang with
+ formal equivalents ensures that automated sentiment analysis tools work accurately and provide reliable
+ insights.
+ </li>
+ </ul>
+ <p>
+ Below, methods <code>replaceWithFlatTransformValues</code> and <code>replaceWithProcessValues</code> show how you
+ can migrate from <code>flatTransformValues</code> to <code>processValues</code>.
+ </p>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessorContext;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
+
+public class ReplacingSlangTextInMessagesExample {
+ private static final Map<String, String> SLANG_DICTIONARY = Map.of(
+ "u", "you",
+ "brb", "be right back",
+ "omg", "oh my god",
+ "btw", "by the way"
+ );
+ private static final String INPUT_MESSAGES_TOPIC = "input-messages-topic";
+ private static final String OUTPUT_MESSAGES_TOPIC = "output-messages-topic";
+
+ public static void replaceWithFlatTransformValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.flatTransformValues(SlangReplacementTransformer::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ public static void replaceWithProcessValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.processValues(SlangReplacementProcessor::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ private static class SlangReplacementTransformer implements ValueTransformer<String, Iterable<String>> {
+ @Override
+ public void init(final org.apache.kafka.streams.processor.ProcessorContext context) {
+ }
+
+ @Override
+ public Iterable<String> transform(final String value) {
+ if (value == null) {
+ return Collections.emptyList(); // Skip null values
+ }
+
+ // Replace slang words in the message
+ final String[] words = value.split("\\s+");
+ return Arrays.asList(
+ Arrays.stream(words)
+ .map(word -> SLANG_DICTIONARY.getOrDefault(word, word))
+ .toArray(String[]::new)
Review Comment:
Why is this returning an `String[]` (if I read it correctly)? -- It should return a `String` instead, so we would need a `reduce()` or something like this? (Did you test test examples (do they compile)?
Or just use the same code as below:
```
final StringBuilder replacedMessage = new StringBuilder();
for (String word : words) {
replacedMessage.append(SLANG_DICTIONARY.getOrDefault(word, word)).append(" ");
}
return replacedMessage.toString();
```
It's confusing if the actually business logic of replacing words is different in both examples; it's better to have it aligned in both.
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3097,615 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
- </ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
- </thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- </tbody>
- </table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <div class="section" id="migrating-from-transform-methods-to-processor-api-papi">
+ <h2>
+ <a class="headerlink" href="#migrating-from-transform-methods-to-processor-api-papi"
+ title="Permalink to this headline">
+ Migrating from transform Methods to Processor API (PAPI)
+ </a>
+ </h2>
+ <h3>Overview of Changes</h3>
+ <p>
+ As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>, <code>transformValues</code>, and <code>flatTransformValues</code>, have
+ been removed. These methods have been replaced with the more versatile Processor API. This
+ guide provides detailed steps for migrating existing code to use the new Processor API and
+ explains the benefits of the changes.
+ </p>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ </ul>
+ <p>The Processor API now serves as a unified replacement for all these methods. It simplifies the
+ API surface while maintaining support for both stateless and stateful operations.</p>
+ <h3>Migration Process</h3>
+ <p>The migration process consists of:</p>
+ <ol>
+ <li>
+ Replace <code>Transformer</code> with <code>Processor</code> or <code>ValueTransformer</code> with
+ <code>FixedKeyProcessor</code>;
+ </li>
+ <li>
+ Replace record <code>key</code> and <code>value</code> with <code>Record</code> or <code>FixedKeyRecord</code>;
+ </li>
+ <li>
+ Rewrite the <code>transform</code> method of <code>Transformer</code> and <code>ValueTransformer</code> as
+ <code>process</code> or <code>processValues</code>;
+ </li>
+ <li>
+ Use the new <code>Record</code> or <code>FixedKeyRecord</code> as argument of the renamed method;</li>
+ <li>
+ Rewrite the return type of the renamed method to <code>void</code> and forward the record through the context;
+ and finally
+ </li>
+ <li>
+ Change the <code>KStream</code> call of the <code>transform</code> method to <code>process</code> or
+ <code>processValues</code>.
+ </li>
+ </ol>
+ <h3>Migration Examples</h3>
+ <p>
+ To migrate from the deprecated <code>transform</code>, <code>transformValues</code>, <code>flatTransform</code>, and
+ <code>flatTransformValues</code> methods to the Process API (PAPI) in Kafka Streams, follow these examples. The new
+ <code>process</code> and <code>processValues</code> APIs enable a more flexible and reusable approach by requiring
+ implementations of the <code>Processor</code> or <code>FixedKeyProcessor</code> interfaces.
+ </p>
+ <p>Here are examples to help you migrate:</p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Migrating from</th>
+ <th>Migrating to</th>
+ <th>State Type</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a></td>
+ <td><code>transform</code></td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>flatTransform</code></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>transformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>flatTransformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
+ </tbody>
+ </table>
+ <h4>Stateless Examples</h4>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li>
+ <strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g., INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to another processor.
+ </li>
+ <li>
+ <strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks, and
+ INFO logs are stored for basic reporting purposes.
+ </li>
+ </ul>
+ <p>
+ Below, methods <code>categorizeWithFlatTransform</code> and <code>categorizeWithProcess</code> show how you can
+ migrate from <code>flatTransform</code> to <code>process</code>.
+ </p>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithFlatTransform(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.flatTransform(() -> new LogSeverityTransformer())
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new);
+ }
- @Override
- void process(Record<PageId, Long> record) {
- // Here you would format and send the alert email.
- //
- // In this specific example, you would be able to include
- // information about the page's ID and its view count
- }
+ private static class LogSeverityTransformer implements Transformer<String, String, Iterable<KeyValue<String, String>>> {
+ @Override
+ public void init(org.apache.kafka.streams.processor.ProcessorContext context) {
+ }
- @Override
- void close() {
- // Any code for clean up would go here, for example tearing down the email client and anything
- // else you created in the init() method
- // This processor instance will not be used again after this call.
- }
+ @Override
+ public Iterable<KeyValue<String, String>> transform(String key, String value) {
+ if (value == null) {
+ return Collections.emptyList(); // Skip null values
+ }
-}</code></pre>
- <div class="admonition tip">
- <p><b>Tip</b></p>
- <p class="last">Even though we do not demonstrate it in this example, a stream processor can access any available state stores by
- calling <code class="docutils literal"><span class="pre">ProcessorContext#getStateStore()</span></code>.
- State stores are only available if they have been connected to the processor, or if they are global stores. While global stores do not need to be connected explicitly, they only allow for read-only access.
- There are two ways to connect state stores to a processor:
- <ul class="simple">
- <li>By passing the name of a store that has already been added via <code class="docutils literal"><span class="pre">Topology#addStateStore()</span></code> to the corresponding <code class="docutils literal"><span class="pre">KStream#process()</span></code> method call.</li>
- <li>Implementing <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the <code class="docutils literal"><span class="pre">ProcessorSupplier</span></code>
- passed to <code class="docutils literal"><span class="pre">KStream#process()</span></code>. In this case there is no need to call <code class="docutils literal"><span class="pre">StreamsBuilder#addStateStore()</span></code>
- beforehand, the store will be automatically added for you. You can also implement <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the
- <code class="docutils literal"><span class="pre">Value*</span></code> or <code class="docutils literal"><span class="pre">*WithKey</span></code> supplier variants, or <code class="docutils literal"><span class="pre">TransformerSupplier</span></code> or any of its variants.
- </li>
- </ul>
- </div>
- <p>Then we can leverage the <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code> processor in the DSL via <code class="docutils literal"><span class="pre">KStream#process</span></code>.</p>
- <pre class="line-numbers"><code class="language-java">KStream<String, GenericRecord> pageViews = ...;
-
-// Send an email notification when the view count of a page reaches one thousand.
-pageViews.groupByKey()
- .count()
- .filter((PageId pageId, Long viewCount) -> viewCount == 1000)
- // PopularPageEmailAlert is your custom processor that implements the
- // `Processor` interface, see further down below.
- .process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
- </div>
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ int colonIndex = value.indexOf(':');
+ String severity = colonIndex > 0 ? value.substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Create appropriate KeyValue pair based on severity
+ return switch (severity) {
+ case "ERROR" -> List.of(new KeyValue<>("ERROR", value));
+ case "WARN" -> List.of(new KeyValue<>("WARN", value));
+ case "INFO" -> Collections.emptyList(); // INFO logs are ignored
+ default -> List.of(new KeyValue<>("UNKNOWN", value));
+ };
+ }
+
+ @Override
+ public void close() {
+ }
+ }
+
+ private static class LogSeverityProcessor implements Processor<String, String, String, String> {
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ }
+
+ @Override
+ public void process(final Record<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ final int colonIndex = record.value().indexOf(':');
+ final String severity = colonIndex > 0 ? record.value().substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Route logs based on severity
+ switch (severity) {
+ case "ERROR":
+ context.forward(new Record<>(ERROR_LOGS_TOPIC, record.value(), record.timestamp()));
+ break;
+ case "WARN":
+ context.forward(new Record<>(WARN_LOGS_TOPIC, record.value(), record.timestamp()));
+ break;
+ case "INFO":
+ // INFO logs are ignored
+ break;
+ default:
+ // Forward to an "unknown" topic for logs with unrecognized severities
+ context.forward(new Record<>(UNKNOWN_LOGS_TOPIC, record.value(), record.timestamp()));
+ }
+ }
+ }
+}
+</code></pre>
+ <h5 id="replacing-slang-in-text-messages">Replacing Slang in Text Messages</h5>
+ <ul>
+ <li>
+ <strong>Idea:</strong> A messaging stream contains user-generated content, and you want to replace slang words
+ with their formal equivalents (e.g., "u" becomes "you", "brb" becomes "be
+ right back"). The operation only modifies the message value and keeps the key intact.
+ </li>
+ <li>
+ <strong>Real-World Context:</strong> In customer support chat systems, normalizing text by replacing slang with
+ formal equivalents ensures that automated sentiment analysis tools work accurately and provide reliable
+ insights.
+ </li>
+ </ul>
+ <p>
+ Below, methods <code>replaceWithFlatTransformValues</code> and <code>replaceWithProcessValues</code> show how you
+ can migrate from <code>flatTransformValues</code> to <code>processValues</code>.
+ </p>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessorContext;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
+
+public class ReplacingSlangTextInMessagesExample {
+ private static final Map<String, String> SLANG_DICTIONARY = Map.of(
+ "u", "you",
+ "brb", "be right back",
+ "omg", "oh my god",
+ "btw", "by the way"
+ );
+ private static final String INPUT_MESSAGES_TOPIC = "input-messages-topic";
+ private static final String OUTPUT_MESSAGES_TOPIC = "output-messages-topic";
+
+ public static void replaceWithFlatTransformValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.flatTransformValues(SlangReplacementTransformer::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ public static void replaceWithProcessValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.processValues(SlangReplacementProcessor::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ private static class SlangReplacementTransformer implements ValueTransformer<String, Iterable<String>> {
+ @Override
+ public void init(final org.apache.kafka.streams.processor.ProcessorContext context) {
+ }
+
+ @Override
+ public Iterable<String> transform(final String value) {
+ if (value == null) {
+ return Collections.emptyList(); // Skip null values
+ }
+
+ // Replace slang words in the message
+ final String[] words = value.split("\\s+");
+ return Arrays.asList(
+ Arrays.stream(words)
+ .map(word -> SLANG_DICTIONARY.getOrDefault(word, word))
+ .toArray(String[]::new)
+ );
+ }
+
+ @Override
+ public void close() {
+ }
+ }
+
+ private static class SlangReplacementProcessor implements FixedKeyProcessor<String, String, String> {
Review Comment:
`ContextualFixedKeyProcessor` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on PR #18314:
URL: https://github.com/apache/kafka/pull/18314#issuecomment-2581326906
Sorry, @mjsax. I've been a little busy these days. I fully agree with finishing it before the code freeze. I should be able to send an update by tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on PR #18314:
URL: https://github.com/apache/kafka/pull/18314#issuecomment-2591093065
@mjsax, I suppose we won't be able to finish before the freeze 😔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on PR #18314:
URL: https://github.com/apache/kafka/pull/18314#issuecomment-2622947501
Thanks for supporting 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on code in PR #18314:
URL: https://github.com/apache/kafka/pull/18314#discussion_r1933121266
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
Review Comment:
Oh, sorry, I had that in mind, but this time it went unnoticed after the Markdown to HTML conversion. I ended up "batch updating".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on code in PR #18314:
URL: https://github.com/apache/kafka/pull/18314#discussion_r1933065851
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
+ </li>
+ <li><code>ProcessorSupplier</code>: A processor supplier that can create one or more <code>Processor</code>
+ instances; and</li>
+ <li><code>FixedKeyProcessorSupplier</code>: A processor supplier that can create one or more
+ <code>FixedKeyProcessor</code> instances.
+ </li>
+ </ul>
+ <h4>Examples</h4>
+ <p>Follow the examples below to learn how to apply <code>process</code> and <code>processValues</code> to your
+ <code>KStream</code>.
+ </p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Operation</th>
+ <th>State Type</th>
+ </tr>
</thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
</tbody>
</table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <h5 id="cumulative-discounts-for-a-loyalty-program"">Cumulative Discounts for a Loyalty Program</h5>
+ <ul>
+ <li><strong>Idea:</strong> A stream of purchase events contains user IDs and transaction amounts. Use a state
+ store
+ to accumulate the total spending of each user. When their total crosses a threshold, apply a discount on
+ their next transaction and update their accumulated total.</li>
+ <li><strong>Real-World Context:</strong> In a retail loyalty program, tracking cumulative customer spending
+ enables
+ dynamic rewards, such as issuing a discount when a customer's total purchases exceed a predefined limit.
+ </li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
Review Comment:
Removed with packages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on code in PR #18314:
URL: https://github.com/apache/kafka/pull/18314#discussion_r1933073009
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
+ </li>
+ <li><code>ProcessorSupplier</code>: A processor supplier that can create one or more <code>Processor</code>
+ instances; and</li>
+ <li><code>FixedKeyProcessorSupplier</code>: A processor supplier that can create one or more
+ <code>FixedKeyProcessor</code> instances.
+ </li>
+ </ul>
+ <h4>Examples</h4>
+ <p>Follow the examples below to learn how to apply <code>process</code> and <code>processValues</code> to your
+ <code>KStream</code>.
+ </p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Operation</th>
+ <th>State Type</th>
+ </tr>
</thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
</tbody>
</table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <h5 id="cumulative-discounts-for-a-loyalty-program"">Cumulative Discounts for a Loyalty Program</h5>
+ <ul>
+ <li><strong>Idea:</strong> A stream of purchase events contains user IDs and transaction amounts. Use a state
+ store
+ to accumulate the total spending of each user. When their total crosses a threshold, apply a discount on
+ their next transaction and update their accumulated total.</li>
+ <li><strong>Real-World Context:</strong> In a retail loyalty program, tracking cumulative customer spending
+ enables
+ dynamic rewards, such as issuing a discount when a customer's total purchases exceed a predefined limit.
+ </li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+
+public class CumulativeDiscountsForALoyaltyProgramExample {
+ private static final double DISCOUNT_THRESHOLD = 100.0;
+ private static final String CUSTOMER_SPENDING_STORE = "customer-spending-store";
+ private static final String DISCOUNT_NOTIFICATION_MESSAGE =
+ "Discount applied! You have received a reward for your purchases.";
+ private static final String DISCOUNT_NOTIFICATIONS_TOPIC = "discount-notifications-topic";
+ private static final String PURCHASE_EVENTS_TOPIC = "purchase-events-topic";
+
+ public static void applyDiscountWithProcess(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ Serdes.String(),
+ Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Processor with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.process(CumulativeDiscountProcessor::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ private static class CumulativeDiscountProcessor implements Processor<String, Double, String, String> {
+ private KeyValueStore<String, Double> spendingStore;
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public void process(final Record<String, Double> record) {
+ if (record.value() == null) {
+ return; // Skip null purchase amounts
+ }
- @Override
- void process(Record<PageId, Long> record) {
- // Here you would format and send the alert email.
- //
- // In this specific example, you would be able to include
- // information about the page's ID and its view count
- }
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(record.key());
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += record.value();
+ spendingStore.put(record.key(), currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(record.key(), currentSpending - DISCOUNT_THRESHOLD);
+ // Send a discount notification
+ context.forward(record.withValue(DISCOUNT_NOTIFICATION_MESSAGE));
+ }
+ }
+ }
+}</code></pre>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li><strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g.,
+ INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to a dedicated topic too.</li>
+ <li><strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity
+ ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks,
+ and
+ INFO logs are stored for basic reporting purposes.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- void close() {
- // Any code for clean up would go here, for example tearing down the email client and anything
- // else you created in the init() method
- // This processor instance will not be used again after this call.
- }
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new)
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
+
+ private static class LogSeverityProcessor extends ContextualProcessor<String, String, String, String> {
+ @Override
+ public void process(final Record<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ final int colonIndex = record.value().indexOf(':');
+ final String severity = colonIndex > 0 ? record.value().substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Route logs based on severity
+ switch (severity) {
+ case "ERROR":
+ context().forward(record.withKey(ERROR_LOGS_TOPIC));
+ break;
+ case "WARN":
+ context().forward(record.withKey(WARN_LOGS_TOPIC));
+ break;
+ case "INFO":
+ // INFO logs are ignored
+ break;
+ default:
+ // Forward to an "unknown" topic for logs with unrecognized severities
+ context().forward(record.withKey(UNKNOWN_LOGS_TOPIC));
+ }
+ }
+ }
}</code></pre>
- <div class="admonition tip">
- <p><b>Tip</b></p>
- <p class="last">Even though we do not demonstrate it in this example, a stream processor can access any available state stores by
- calling <code class="docutils literal"><span class="pre">ProcessorContext#getStateStore()</span></code>.
- State stores are only available if they have been connected to the processor, or if they are global stores. While global stores do not need to be connected explicitly, they only allow for read-only access.
- There are two ways to connect state stores to a processor:
- <ul class="simple">
- <li>By passing the name of a store that has already been added via <code class="docutils literal"><span class="pre">Topology#addStateStore()</span></code> to the corresponding <code class="docutils literal"><span class="pre">KStream#process()</span></code> method call.</li>
- <li>Implementing <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the <code class="docutils literal"><span class="pre">ProcessorSupplier</span></code>
- passed to <code class="docutils literal"><span class="pre">KStream#process()</span></code>. In this case there is no need to call <code class="docutils literal"><span class="pre">StreamsBuilder#addStateStore()</span></code>
- beforehand, the store will be automatically added for you. You can also implement <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the
- <code class="docutils literal"><span class="pre">Value*</span></code> or <code class="docutils literal"><span class="pre">*WithKey</span></code> supplier variants, or <code class="docutils literal"><span class="pre">TransformerSupplier</span></code> or any of its variants.
- </li>
- </ul>
- </div>
- <p>Then we can leverage the <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code> processor in the DSL via <code class="docutils literal"><span class="pre">KStream#process</span></code>.</p>
- <pre class="line-numbers"><code class="language-java">KStream<String, GenericRecord> pageViews = ...;
-
-// Send an email notification when the view count of a page reaches one thousand.
-pageViews.groupByKey()
- .count()
- .filter((PageId pageId, Long viewCount) -> viewCount == 1000)
- // PopularPageEmailAlert is your custom processor that implements the
- // `Processor` interface, see further down below.
- .process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
- </div>
+ <h5 id="traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</h5>
+ <ul>
+ <li><strong>Idea:</strong> A radar monitors cars passing along a road stretch. A system counts the cars for each
+ day, maintaining a cumulative total for the current day in a state store. At the end of the day, the count
+ is
+ emitted and the state is cleared for the next day.</li>
+ <li><strong>Real-World Context:</strong> A car counting system can be useful for determining measures for
+ widening
+ or controlling traffic depending on the number of cars passing through the monitored stretch.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.common.serialization;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessorContext;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.state.Stores;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.format.DateTimeFormatter;
+
+public class TrafficRadarMonitoringCarCountExample {
+ private static final String DAILY_COUNT_STORE = "price-state-store";
+ private static final String DAILY_COUNT_TOPIC = "price-state-topic";
+ private static final String RADAR_COUNT_TOPIC = "car-radar-topic";
+
+ public static void countWithProcessValues(final StreamsBuilder builder) {
+ // Define a state store for tracking daily car counts
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(DAILY_COUNT_STORE),
+ Serdes.String(),
+ Serdes.Long()
+ )
+ );
+ final KStream<Void, String> radarStream = builder.stream(RADAR_COUNT_TOPIC);
+ // Apply the FixedKeyProcessor with the state store
+ radarStream.processValues(DailyCarCountProcessor::new, DAILY_COUNT_STORE)
+ .to(DAILY_COUNT_TOPIC);
+ }
+
+ private static class DailyCarCountProcessor implements FixedKeyProcessor<Void, String, String> {
+ private FixedKeyProcessorContext<Void, String> context;
+ private KeyValueStore<String, Long> stateStore;
+ private static final DateTimeFormatter DATE_FORMATTER =
+ DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(ZoneId.systemDefault());
+
+ @Override
+ public void init(final FixedKeyProcessorContext<Void, String> context) {
+ this.context = context;
+ stateStore = context.getStateStore(DAILY_COUNT_STORE);
+ }
+
+ @Override
+ public void process(final FixedKeyRecord<Void, String> record) {
+ if (record.value() == null) {
+ return; // Skip null events
+ }
+
+ // Derive the current day from the event timestamp
+ final long timestamp = System.currentTimeMillis(); // Use system time for simplicity
+ final String currentDay = DATE_FORMATTER.format(Instant.ofEpochMilli(timestamp));
+ // Retrieve the current count for the day
+ Long dailyCount = stateStore.get(currentDay);
+ if (dailyCount == null) {
+ dailyCount = 0L;
+ }
+ // Increment the count
+ dailyCount++;
+ stateStore.put(currentDay, dailyCount);
+
+ // Emit the current day's count
+ context.forward(record.withValue(String.format("Day: %s, Car Count: %s", currentDay, dailyCount)));
+ }
+ }
+}</code></pre>
+ <h5 id="replacing-slang-in-text-messages">Replacing Slang in Text Messages</h5>
+ <ul>
+ <li><strong>Idea:</strong> A messaging stream contains user-generated content, and you want to replace slang
+ words
+ with their formal equivalents (e.g., "u" becomes "you", "brb" becomes "be
+ right back"). The operation only modifies the message value and keeps the key intact.</li>
+ <li><strong>Real-World Context:</strong> In customer support chat systems, normalizing text by replacing slang
+ with
+ formal equivalents ensures that automated sentiment analysis tools work accurately and provide reliable
+ insights.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualFixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
+
+public class ReplacingSlangTextInMessagesExample {
+ private static final Map<String, String> SLANG_DICTIONARY = Map.of(
+ "u", "you",
+ "brb", "be right back",
+ "omg", "oh my god",
+ "btw", "by the way"
+ );
+ private static final String INPUT_MESSAGES_TOPIC = "input-messages-topic";
+ private static final String OUTPUT_MESSAGES_TOPIC = "output-messages-topic";
+
+ public static void replaceWithProcessValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.processValues(SlangReplacementProcessor::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ private static class SlangReplacementProcessor extends ContextualFixedKeyProcessor<String, String, String> {
+ @Override
+ public void process(final FixedKeyRecord<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+
+ // Replace slang words in the message
+ final String[] words = record.value().split("\\s+");
+ for (final String word : words) {
+ String replacedWord = SLANG_DICTIONARY.getOrDefault(word, word);
+ context().forward(record.withValue(replacedWord));
+ }
+ }
+ }
+}</code></pre>
+ <h4>Keynotes</h4>
+ <ul>
+ <li><strong>Type Safety and Flexibility:</strong> The process and processValues APIs utilize
+ <code>ProcessorContext</code> and <code>Record</code> or <code>FixedKeyRecord</code> objects for better type
+ safety and flexibility of custom processing logic.
+ </li>
+ <li><strong>Clear State and Logic Management:</strong> Implementations for <code>Processor</code> or
+ <code>FixedKeyProcessor</code> should manage state and logic clearly. Use <code>context().forward()</code>
+ for emitting records downstream.
+ </li>
+ <li><strong>Unified API:</strong> Consolidates multiple methods into a single, versatile API.</li>
+ <li><strong>Future-Proof:</strong> Ensures compatibility with the latest Kafka Streams releases.</li>
+ </ul>
+ </div>
+ <div class="section" id="transformers-removal-and-migration-to-processors">
+ <a class="headerlink" href="#transformers-removal-and-migration-to-processors" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id37">Transformers removal and migration to processors</a>
+ </h3>
+ </a>
+ <p>As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>,<br><code>transformValues</code>, <code>flatTransformValues</code>, and
+ <code>process</code> have been removed. These methods have been replaced with the<br>more versatile Processor API.
+ This guide provides detailed steps for migrating existing code to use the new Processor<br>API and explains the
+ benefits of the changes.</p>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ <li><code>KStream#process</code></li>
+ </ul>
+ <p>The Processor API now serves as a unified replacement for all these methods. It simplifies the API surface
+ while<br>maintaining support for both stateless and stateful operations.</p>
+ <h4>Migration Examples</h4>
+ <p>To migrate from the deprecated <code>transform</code>, <code>transformValues</code>, <code>flatTransform</code>, and
+ <code>flatTransformValues</code> methods to the<br>Processor API (PAPI) in Kafka Streams, let's resume the
+ previouss examples. The new <code>process</code> and <code>processValues</code><br>methods enable a more flexible
+ and reusable approach by requiring implementations of the <code>Processor</code>
+ or<br><code>FixedKeyProcessor</code> interfaces.</p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Migrating from</th>
+ <th>Migrating to</th>
+ <th>State Type</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program-removal">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>transform</code></td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity-removal">Categorizing Logs by Severity</a></td>
+ <td><code>flatTransform</code></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count-removal">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>transformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages-removal">Replacing Slang in Text Messages</a></td>
+ <td><code>flatTransformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
+ </tbody>
+ </table>
+ <h5 id="cumulative-discounts-for-a-loyalty-program-removal">Cumulative Discounts for a Loyalty Program</h5>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+
+public class CumulativeDiscountsForALoyaltyProgramExample {
+ private static final double DISCOUNT_THRESHOLD = 100.0;
+ private static final String CUSTOMER_SPENDING_STORE = "customer-spending-store";
+ private static final String DISCOUNT_NOTIFICATION_MESSAGE =
+ "Discount applied! You have received a reward for your purchases.";
+ private static final String DISCOUNT_NOTIFICATIONS_TOPIC = "discount-notifications-topic";
+ private static final String PURCHASE_EVENTS_TOPIC = "purchase-events-topic";
+
+ public static void applyDiscountWithTransform(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ Serdes.String(),
+ Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Transformer with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.transform(CumulativeDiscountTransformer::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
+
+ public static void applyDiscountWithProcess(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ org.apache.kafka.common.serialization.Serdes.String(),
+ org.apache.kafka.common.serialization.Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Processor with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.process(CumulativeDiscountProcessor::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
+
+ private static class CumulativeDiscountTransformer implements Transformer<String, Double, KeyValue<String, String>> {
+ private KeyValueStore<String, Double> spendingStore;
+
+ @Override
+ public void init(final org.apache.kafka.streams.processor.ProcessorContext context) {
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public KeyValue<String, String> transform(final String key, final Double value) {
+ if (value == null) {
+ return null; // Skip null purchase amounts
+ }
+
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(key);
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += value;
+ spendingStore.put(key, currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(key, currentSpending - DISCOUNT_THRESHOLD);
+ // Return a notification message
+ return new KeyValue<>(key, DISCOUNT_NOTIFICATION_MESSAGE);
+ }
+ return null; // No discount, so no output for this record
+ }
+
+ @Override
+ public void close() {
+ }
+ }
+
+ private static class CumulativeDiscountProcessor implements Processor<String, Double, String, String> {
+ private KeyValueStore<String, Double> spendingStore;
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public void process(final Record<String, Double> record) {
+ if (record.value() == null) {
+ return; // Skip null purchase amounts
+ }
+
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(record.key());
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += record.value();
+ spendingStore.put(record.key(), currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(record.key(), currentSpending - DISCOUNT_THRESHOLD);
+ // Send a discount notification
+ context.forward(record.withValue(DISCOUNT_NOTIFICATION_MESSAGE));
+ }
+ }
+ }
+}</code></pre>
+ <h5 id="categorizing-logs-by-severity-removal">Categorizing Logs by Severity</h5>
+ <p>Below, methods <code>categorizeWithFlatTransform</code> and <code>categorizeWithProcess</code> show how you can
+ migrate from <code>flatTransform</code><br>to <code>process</code>.</p>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithFlatTransform(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.flatTransform(LogSeverityTransformer::new)
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
+
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new)
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
+
+ private static class LogSeverityTransformer implements Transformer<String, String, Iterable<KeyValue<String, String>>> {
+ @Override
+ public void init(org.apache.kafka.streams.processor.ProcessorContext context) {
+ }
+
+ @Override
+ public Iterable<KeyValue<String, String>> transform(String key, String value) {
+ if (value == null) {
+ return Collections.emptyList(); // Skip null values
+ }
+
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ int colonIndex = value.indexOf(':');
+ String severity = colonIndex > 0 ? value.substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Create appropriate KeyValue pair based on severity
+ return switch (severity) {
+ case "ERROR" -> List.of(new KeyValue<>("ERROR", value));
+ case "WARN" -> List.of(new KeyValue<>("WARN", value));
+ case "INFO" -> Collections.emptyList(); // INFO logs are ignored
+ default -> List.of(new KeyValue<>("UNKNOWN", value));
+ };
+ }
+
+ @Override
+ public void close() {
+ }
+ }
+
+ private static class LogSeverityProcessor extends ContextualProcessor<String, String, String, String> {
+ @Override
+ public void process(final Record<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ final int colonIndex = record.value().indexOf(':');
+ final String severity = colonIndex > 0 ? record.value().substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Route logs based on severity
+ switch (severity) {
+ case "ERROR":
+ context().forward(record.withKey(ERROR_LOGS_TOPIC));
+ break;
+ case "WARN":
+ context().forward(record.withKey(WARN_LOGS_TOPIC));
+ break;
+ case "INFO":
+ // INFO logs are ignored
+ break;
+ default:
+ // Forward to an "unknown" topic for logs with unrecognized severities
+ context().forward(record.withKey(UNKNOWN_LOGS_TOPIC));
+ }
+ }
+ }
+}</code></pre>
+ <h5 id="traffic-radar-monitoring-car-count-removal">Traffic Radar Monitoring Car Count</h5>
+ <p>Below, methods <code>countWithTransformValues</code> and <code>countWithProcessValues</code> show how you can migrate
+ from <code>transformValues</code><br>to <code>processValues</code>.</p>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.processor.api.FixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessorContext;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.state.Stores;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.format.DateTimeFormatter;
+
+public class TrafficRadarMonitoringCarCountExample {
+ private static final String DAILY_COUNT_STORE = "price-state-store";
+ private static final String DAILY_COUNT_TOPIC = "price-state-topic";
+ private static final String RADAR_COUNT_TOPIC = "car-radar-topic";
+
+ public static void countWithTransformValues(final StreamsBuilder builder) {
+ // Define a state store for tracking daily car counts
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(DAILY_COUNT_STORE),
+ org.apache.kafka.common.serialization.Serdes.String(),
+ org.apache.kafka.common.serialization.Serdes.Long()
+ )
+ );
+ final KStream<Void, String> radarStream = builder.stream(RADAR_COUNT_TOPIC);
+ // Apply the ValueTransformer with the state store
+ radarStream.transformValues(DailyCarCountTransformer::new, DAILY_COUNT_STORE)
+ .to(DAILY_COUNT_TOPIC);
+ }
+
+ public static void countWithProcessValues(final StreamsBuilder builder) {
+ // Define a state store for tracking daily car counts
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(DAILY_COUNT_STORE),
+ org.apache.kafka.common.serialization.Serdes.String(),
+ org.apache.kafka.common.serialization.Serdes.Long()
+ )
+ );
+ final KStream<Void, String> radarStream = builder.stream(RADAR_COUNT_TOPIC);
+ // Apply the FixedKeyProcessor with the state store
+ radarStream.processValues(DailyCarCountProcessor::new, DAILY_COUNT_STORE)
+ .to(DAILY_COUNT_TOPIC);
+ }
+
+ private static class DailyCarCountTransformer implements ValueTransformerWithKey<Void, String, String> {
+ private KeyValueStore<String, Long> stateStore;
+ private static final DateTimeFormatter DATE_FORMATTER =
+ DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(ZoneId.systemDefault());
+
+ @Override
+ public void init(final org.apache.kafka.streams.processor.ProcessorContext context) {
+ // Access the state store
+ stateStore = context.getStateStore(DAILY_COUNT_STORE);
+ }
+
+ @Override
+ public String transform(Void readOnlyKey, String value) {
+ if (value == null) {
+ return null; // Skip null events
+ }
+
+ // Derive the current day from the event timestamp
+ final long timestamp = System.currentTimeMillis(); // Use system time for simplicity
+ final String currentDay = DATE_FORMATTER.format(Instant.ofEpochMilli(timestamp));
+ // Retrieve the current count for the day
+ Long dailyCount = stateStore.get(currentDay);
+ if (dailyCount == null) {
+ dailyCount = 0L;
+ }
+ // Increment the count
+ dailyCount++;
+ stateStore.put(currentDay, dailyCount);
+
+ // Return the current day's count
+ return String.format("Day: %s, Car Count: %s", currentDay, dailyCount);
+ }
+
+ @Override
+ public void close() {
+ }
+ }
+
+ private static class DailyCarCountProcessor implements FixedKeyProcessor<Void, String, String> {
+ private FixedKeyProcessorContext<Void, String> context;
+ private KeyValueStore<String, Long> stateStore;
+ private static final DateTimeFormatter DATE_FORMATTER =
+ DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(ZoneId.systemDefault());
+
+ @Override
+ public void init(final FixedKeyProcessorContext<Void, String> context) {
+ this.context = context;
+ stateStore = context.getStateStore(DAILY_COUNT_STORE);
+ }
+
+ @Override
+ public void process(final FixedKeyRecord<Void, String> record) {
+ if (record.value() == null) {
+ return; // Skip null events
+ }
+
+ // Derive the current day from the event timestamp
+ final long timestamp = System.currentTimeMillis(); // Use system time for simplicity
+ final String currentDay = DATE_FORMATTER.format(Instant.ofEpochMilli(timestamp));
+ // Retrieve the current count for the day
+ Long dailyCount = stateStore.get(currentDay);
+ if (dailyCount == null) {
+ dailyCount = 0L;
+ }
+ // Increment the count
+ dailyCount++;
+ stateStore.put(currentDay, dailyCount);
+
+ // Emit the current day's count
+ context.forward(record.withValue(String.format("Day: %s, Car Count: %s", currentDay, dailyCount)));
+ }
+ }
+}</code></pre>
+ <h5 id="replacing-slang-in-text-messages-removal">Replacing Slang in Text Messages</h5>
+ <p>Below, methods <code>replaceWithFlatTransformValues</code> and <code>replaceWithProcessValues</code> show how you can
+ migrate from<br><code>flatTransformValues</code> to <code>processValues</code>.</p>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualFixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
+
+public class ReplacingSlangTextInMessagesExample {
+ private static final Map<String, String> SLANG_DICTIONARY = Map.of(
+ "u", "you",
+ "brb", "be right back",
+ "omg", "oh my god",
+ "btw", "by the way"
+ );
+ private static final String INPUT_MESSAGES_TOPIC = "input-messages-topic";
+ private static final String OUTPUT_MESSAGES_TOPIC = "output-messages-topic";
+
+ public static void replaceWithFlatTransformValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.flatTransformValues(SlangReplacementTransformer::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ public static void replaceWithProcessValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.processValues(SlangReplacementProcessor::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ private static class SlangReplacementTransformer implements ValueTransformer<String, Iterable<String>> {
+
+ @Override
+ public void init(final org.apache.kafka.streams.processor.ProcessorContext context) {
+ }
+
+ @Override
+ public Iterable<String> transform(final String value) {
+ if (value == null) {
+ return Collections.emptyList(); // Skip null values
+ }
+
+ // Replace slang words in the message
+ final String[] words = value.split("\\s+");
+ return Arrays.asList(
+ Arrays.stream(words)
+ .map(word -> SLANG_DICTIONARY.getOrDefault(word, word))
+ .toArray(String[]::new)
+ );
+ }
+
+ @Override
+ public void close() {
+ }
+ }
+
+ private static class SlangReplacementProcessor extends ContextualFixedKeyProcessor<String, String, String> {
+ @Override
+ public void process(final FixedKeyRecord<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+
+ // Replace slang words in the message
+ final String[] words = record.value().split("\\s+");
+ for (final String word : words) {
+ String replacedWord = SLANG_DICTIONARY.getOrDefault(word, word);
+ context().forward(record.withValue(replacedWord));
+ }
+ }
+ }
+}</code></pre>
+ <h4>Keynotes</h4>
+ <ul>
+ <li><strong>Type Safety and Flexibility:</strong> The process and processValues APIs utilize
+ <code>ProcessorContext</code> and <code>Record</code> or<br><code>FixedKeyRecord</code> objects for better type
+ safety and flexibility of custom processing logic.</li>
+ <li><strong>Clear State and Logic Management:</strong> Implementations for <code>Processor</code> or
+ <code>FixedKeyProcessor</code> should manage state and<br>logic clearly. Use <code>context().forward()</code>
+ for emitting records downstream.</li>
+ <li><strong>Unified API:</strong> Consolidates multiple methods into a single, versatile API.</li>
+ <li><strong>Future-Proof:</strong> Ensures compatibility with the latest Kafka Streams releases.</li>
+ </ul>
+ <h4>Removal of Old <code>process</code> Method</h4>
+ <p>It is worth mentioning that, in addition to the methods mentioned above, the <code>process</code> method (which is
+ not part of the<br>Processor API) has also been removed. The following example shows how to migrate to the new
Review Comment:
Perfect! This is exactly what I wanted to say. Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on code in PR #18314:
URL: https://github.com/apache/kafka/pull/18314#discussion_r1928019115
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3097,615 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
- </ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
- </thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- </tbody>
- </table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <div class="section" id="migrating-from-transform-methods-to-processor-api-papi">
+ <h2>
+ <a class="headerlink" href="#migrating-from-transform-methods-to-processor-api-papi"
+ title="Permalink to this headline">
+ Migrating from transform Methods to Processor API (PAPI)
+ </a>
+ </h2>
+ <h3>Overview of Changes</h3>
+ <p>
+ As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>, <code>transformValues</code>, and <code>flatTransformValues</code>, have
+ been removed. These methods have been replaced with the more versatile Processor API. This
+ guide provides detailed steps for migrating existing code to use the new Processor API and
+ explains the benefits of the changes.
+ </p>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ </ul>
+ <p>The Processor API now serves as a unified replacement for all these methods. It simplifies the
+ API surface while maintaining support for both stateless and stateful operations.</p>
+ <h3>Migration Process</h3>
+ <p>The migration process consists of:</p>
+ <ol>
+ <li>
+ Replace <code>Transformer</code> with <code>Processor</code> or <code>ValueTransformer</code> with
+ <code>FixedKeyProcessor</code>;
+ </li>
+ <li>
+ Replace record <code>key</code> and <code>value</code> with <code>Record</code> or <code>FixedKeyRecord</code>;
+ </li>
+ <li>
+ Rewrite the <code>transform</code> method of <code>Transformer</code> and <code>ValueTransformer</code> as
+ <code>process</code> or <code>processValues</code>;
+ </li>
+ <li>
+ Use the new <code>Record</code> or <code>FixedKeyRecord</code> as argument of the renamed method;</li>
+ <li>
+ Rewrite the return type of the renamed method to <code>void</code> and forward the record through the context;
+ and finally
+ </li>
+ <li>
+ Change the <code>KStream</code> call of the <code>transform</code> method to <code>process</code> or
+ <code>processValues</code>.
+ </li>
+ </ol>
+ <h3>Migration Examples</h3>
+ <p>
+ To migrate from the deprecated <code>transform</code>, <code>transformValues</code>, <code>flatTransform</code>, and
+ <code>flatTransformValues</code> methods to the Process API (PAPI) in Kafka Streams, follow these examples. The new
+ <code>process</code> and <code>processValues</code> APIs enable a more flexible and reusable approach by requiring
+ implementations of the <code>Processor</code> or <code>FixedKeyProcessor</code> interfaces.
+ </p>
+ <p>Here are examples to help you migrate:</p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Migrating from</th>
+ <th>Migrating to</th>
+ <th>State Type</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a></td>
+ <td><code>transform</code></td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>flatTransform</code></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>transformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>flatTransformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
+ </tbody>
+ </table>
+ <h4>Stateless Examples</h4>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li>
+ <strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g., INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to another processor.
+ </li>
+ <li>
+ <strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks, and
+ INFO logs are stored for basic reporting purposes.
+ </li>
+ </ul>
+ <p>
+ Below, methods <code>categorizeWithFlatTransform</code> and <code>categorizeWithProcess</code> show how you can
+ migrate from <code>flatTransform</code> to <code>process</code>.
+ </p>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithFlatTransform(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.flatTransform(() -> new LogSeverityTransformer())
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new);
Review Comment:
Updated in Mardown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #18314:
URL: https://github.com/apache/kafka/pull/18314#issuecomment-2605263355
@fonsdant -- How are things. Any progress?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on PR #18314:
URL: https://github.com/apache/kafka/pull/18314#issuecomment-2566716097
I have wrote a "draft" into Markdown, before to write it into HTML. I hope this helps the review. Now, I will translate it to HTML.
***
# Migrating from transform to process
To migrate from the deprecated transform and flatTransform methods to the process API in Kafka Streams, follow these
steps. The new process and processValues APIs enable a more flexible and reusable approach by requiring implementations
of the Processor or ProcessorWithKey interfaces.
Here are examples to help you migrate:
| Example | Migrating from | Migrating to | Type |
|-------------------------------------------------------------------------------------------|-----------------------|-----------------|-----------|
| [Cumulative Discounts for a Loyalty Program](#cumulative-discounts-for-a-loyalty-program) | `transform` | `process` | Stateful |
| [Categorizing Logs by Severity](#categorizing-logs-by-severity) | `flatTransform` | `process` | Stateless |
| [Traffic Radar Monitoring Car Count](#traffic-radar-monitoring-car-count) | `transformValues` | `processValues` | Stateful |
| [Replacing Slang in Text Messages](#replacing-slang-in-text-messages) | `flatTransformValues` | `processValues` | Stateless |
## Stateless Examples
### Categorizing Logs by Severity
* **Idea:** You have a stream of log messages. Each message contains a severity level (e.g., INFO, WARN, ERROR) in the
value. The processor filters messages, routing ERROR messages to a dedicated topic and discarding INFO messages. The
rest (WARN) are forwarded to another processor.
* **Real-World Context:** In a production monitoring system, categorizing logs by severity ensures ERROR logs are sent
to a critical incident management system, WARN logs are analyzed for potential risks, and INFO logs are stored for
basic reporting purposes.
Below, methods `categorizeWithFlatTransform` and `categorizeWithProcess` show how you can migrate from `flatTransform`
to `process`.
```java
package org.apache.kafka.streams.kstream;
import org.apache.kafka.streams.KeyValue;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.processor.api.Processor;
import org.apache.kafka.streams.processor.api.ProcessorContext;
import org.apache.kafka.streams.processor.api.Record;
import java.util.Collections;
import java.util.List;
public class CategorizingLogsBySeverityExample {
private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
public static void categorizeWithFlatTransform(final StreamsBuilder builder) {
final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
logStream.flatTransform(() -> new LogSeverityTransformer())
.to((key, value, recordContext) -> {
// Determine the target topic dynamically
if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
return UNKNOWN_LOGS_TOPIC;
});
}
public static void categorizeWithProcess(final StreamsBuilder builder) {
final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
logStream.process(LogSeverityProcessor::new);
}
private static class LogSeverityTransformer implements Transformer<String, String, Iterable<KeyValue<String, String>>> {
@Override
public void init(org.apache.kafka.streams.processor.ProcessorContext context) {
}
@Override
public Iterable<KeyValue<String, String>> transform(String key, String value) {
if (value == null) {
return Collections.emptyList(); // Skip null values
}
// Assume the severity is the first word in the log message
// For example: "ERROR: Disk not found" -> "ERROR"
int colonIndex = value.indexOf(':');
String severity = colonIndex > 0 ? value.substring(0, colonIndex).trim() : "UNKNOWN";
// Create appropriate KeyValue pair based on severity
return switch (severity) {
case "ERROR" -> List.of(new KeyValue<>("ERROR", value));
case "WARN" -> List.of(new KeyValue<>("WARN", value));
case "INFO" -> Collections.emptyList(); // INFO logs are ignored
default -> List.of(new KeyValue<>("UNKNOWN", value));
};
}
@Override
public void close() {
}
}
private static class LogSeverityProcessor implements Processor<String, String, String, String> {
private ProcessorContext<String, String> context;
@Override
public void init(final ProcessorContext<String, String> context) {
this.context = context;
}
@Override
public void process(final Record<String, String> record) {
if (record.value() == null) {
return; // Skip null values
}
// Assume the severity is the first word in the log message
// For example: "ERROR: Disk not found" -> "ERROR"
final int colonIndex = record.value().indexOf(':');
final String severity = colonIndex > 0 ? record.value().substring(0, colonIndex).trim() : "UNKNOWN";
// Route logs based on severity
switch (severity) {
case "ERROR":
context.forward(new Record<>(ERROR_LOGS_TOPIC, record.value(), record.timestamp()));
break;
case "WARN":
context.forward(new Record<>(WARN_LOGS_TOPIC, record.value(), record.timestamp()));
break;
case "INFO":
// INFO logs are ignored
break;
default:
// Forward to an "unknown" topic for logs with unrecognized severities
context.forward(new Record<>(UNKNOWN_LOGS_TOPIC, record.value(), record.timestamp()));
}
}
}
}
```
### Replacing Slang in Text Messages
* **Idea:** A messaging stream contains user-generated content, and you want to replace slang words with their formal
equivalents (e.g., "u" becomes "you", "brb" becomes "be right back"). The operation only modifies the message value
and keeps the key intact.
* **Real-World Context:** In customer support chat systems, normalizing text by replacing slang with formal equivalents
ensures that automated sentiment analysis tools work accurately and provide reliable insights.
Below, methods `replaceWithFlatTransformValues` and `replaceWithProcessValues` show how you can migrate from
`flatTransformValues` to `processValues`.
```java
package org.apache.kafka.streams.kstream;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.processor.api.FixedKeyProcessor;
import org.apache.kafka.streams.processor.api.FixedKeyProcessorContext;
import org.apache.kafka.streams.processor.api.FixedKeyRecord;
import java.util.Arrays;
import java.util.Collections;
import java.util.Map;
public class ReplacingSlangTextInMessagesExample {
private static final Map<String, String> SLANG_DICTIONARY = Map.of(
"u", "you",
"brb", "be right back",
"omg", "oh my god",
"btw", "by the way"
);
private static final String INPUT_MESSAGES_TOPIC = "input-messages-topic";
private static final String OUTPUT_MESSAGES_TOPIC = "output-messages-topic";
public static void replaceWithFlatTransformValues(final StreamsBuilder builder) {
KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
messageStream.flatTransformValues(SlangReplacementTransformer::new).to(OUTPUT_MESSAGES_TOPIC);
}
public static void replaceWithProcessValues(final StreamsBuilder builder) {
KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
messageStream.processValues(SlangReplacementProcessor::new).to(OUTPUT_MESSAGES_TOPIC);
}
private static class SlangReplacementTransformer implements ValueTransformer<String, Iterable<String>> {
@Override
public void init(final org.apache.kafka.streams.processor.ProcessorContext context) {
}
@Override
public Iterable<String> transform(final String value) {
if (value == null) {
return Collections.emptyList(); // Skip null values
}
// Replace slang words in the message
final String[] words = value.split("\\s+");
return Arrays.asList(
Arrays.stream(words)
.map(word -> SLANG_DICTIONARY.getOrDefault(word, word))
.toArray(String[]::new)
);
}
@Override
public void close() {
}
}
private static class SlangReplacementProcessor implements FixedKeyProcessor<String, String, String> {
private FixedKeyProcessorContext<String, String> context;
@Override
public void init(final FixedKeyProcessorContext<String, String> context) {
this.context = context;
}
@Override
public void process(final FixedKeyRecord<String, String> record) {
if (record.value() == null) {
return; // Skip null values
}
// Replace slang words in the message
final String[] words = record.value().split("\\s+");
final StringBuilder replacedMessage = new StringBuilder();
for (String word : words) {
replacedMessage.append(SLANG_DICTIONARY.getOrDefault(word, word)).append(" ");
}
context.forward(record.withValue(replacedMessage.toString()));
}
}
}
```
## Stateful Examples
### Cumulative Discounts for a Loyalty Program
* **Idea:** A stream of purchase events contains user IDs and transaction amounts. Use a state store to accumulate the
total spending of each user. When their total crosses a threshold, apply a discount on their next transaction and
update their accumulated total.
* **Real-World Context:** In a retail loyalty program, tracking cumulative customer spending enables dynamic rewards,
such as issuing a discount when a customer’s total purchases exceed a predefined limit.
Below, methods `applyDiscountWithTransform` and `applyDiscountWithProcess` show how you can migrate from `transform` to
`process`.
```java
package org.apache.kafka.streams.kstream;
import org.apache.kafka.streams.KeyValue;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.processor.api.Processor;
import org.apache.kafka.streams.processor.api.ProcessorContext;
import org.apache.kafka.streams.processor.api.Record;
import org.apache.kafka.streams.state.KeyValueStore;
import org.apache.kafka.streams.state.Stores;
public class CumulativeDiscountsForALoyaltyProgramExample {
private static final double DISCOUNT_THRESHOLD = 100.0;
private static final String CUSTOMER_SPENDING_STORE = "customer-spending-store";
private static final String DISCOUNT_NOTIFICATION_MESSAGE =
"Discount applied! You have received a reward for your purchases.";
private static final String DISCOUNT_NOTIFICATIONS_TOPIC = "discount-notifications-topic";
private static final String PURCHASE_EVENTS_TOPIC = "purchase-events-topic";
public static void applyDiscountWithTransform(final StreamsBuilder builder) {
// Define the state store for tracking cumulative spending
builder.addStateStore(
Stores.keyValueStoreBuilder(
Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
org.apache.kafka.common.serialization.Serdes.String(),
org.apache.kafka.common.serialization.Serdes.Double()
)
);
final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
// Apply the Transformer with the state store
final KStream<String, String> notificationStream =
purchaseStream.transform(CumulativeDiscountTransformer::new, CUSTOMER_SPENDING_STORE);
// Send the notifications to the output topic
notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
}
public static void applyDiscountWithProcess(final StreamsBuilder builder) {
// Define the state store for tracking cumulative spending
builder.addStateStore(
Stores.keyValueStoreBuilder(
Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
org.apache.kafka.common.serialization.Serdes.String(),
org.apache.kafka.common.serialization.Serdes.Double()
)
);
final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
// Apply the Processor with the state store
final KStream<String, String> notificationStream =
purchaseStream.process(CumulativeDiscountProcessor::new, CUSTOMER_SPENDING_STORE);
// Send the notifications to the output topic
notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
}
private static class CumulativeDiscountTransformer implements Transformer<String, Double, KeyValue<String, String>> {
private KeyValueStore<String, Double> spendingStore;
@Override
public void init(final org.apache.kafka.streams.processor.ProcessorContext context) {
// Retrieve the state store for cumulative spending
spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
}
@Override
public KeyValue<String, String> transform(final String key, final Double value) {
if (value == null) {
return null; // Skip null purchase amounts
}
// Get the current spending total for the customer
Double currentSpending = spendingStore.get(key);
if (currentSpending == null) {
currentSpending = 0.0;
}
// Update the cumulative spending
currentSpending += value;
spendingStore.put(key, currentSpending);
// Check if the customer qualifies for a discount
if (currentSpending >= DISCOUNT_THRESHOLD) {
// Reset the spending after applying the discount
spendingStore.put(key, currentSpending - DISCOUNT_THRESHOLD);
// Return a notification message
return new KeyValue<>(key, DISCOUNT_NOTIFICATION_MESSAGE);
}
return null; // No discount, so no output for this record
}
@Override
public void close() {
}
}
private static class CumulativeDiscountProcessor implements Processor<String, Double, String, String> {
private KeyValueStore<String, Double> spendingStore;
private ProcessorContext<String, String> context;
@Override
public void init(final ProcessorContext<String, String> context) {
this.context = context;
// Retrieve the state store for cumulative spending
spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
}
@Override
public void process(final Record<String, Double> record) {
if (record.value() == null) {
return; // Skip null purchase amounts
}
// Get the current spending total for the customer
Double currentSpending = spendingStore.get(record.key());
if (currentSpending == null) {
currentSpending = 0.0;
}
// Update the cumulative spending
currentSpending += record.value();
spendingStore.put(record.key(), currentSpending);
// Check if the customer qualifies for a discount
if (currentSpending >= DISCOUNT_THRESHOLD) {
// Reset the spending after applying the discount
spendingStore.put(record.key(), currentSpending - DISCOUNT_THRESHOLD);
// Send a discount notification
context.forward(new Record<>(record.key(), DISCOUNT_NOTIFICATION_MESSAGE, record.timestamp()));
}
}
}
}
```
### Traffic Radar Monitoring Car Count
* **Idea:** A radar monitors cars passing along a road stretch. A system counts the cars for each day, maintaining a
cumulative total for the current day in a state store. At the end of the day, the count is emitted and the state is
cleared for the next day.
* **Real-World Context:** A car counting system can be useful for determining measures for widening or controlling
traffic depending on the number of cars passing through the monitored stretch.
Below, methods `countWithTransformValues` and `countWithProcessValues` show how you can migrate from `transformValues`
to `processValues`.
```java
package org.apache.kafka.streams.kstream;
import org.apache.kafka.streams.processor.api.FixedKeyProcessor;
import org.apache.kafka.streams.processor.api.FixedKeyProcessorContext;
import org.apache.kafka.streams.processor.api.FixedKeyRecord;
import org.apache.kafka.streams.state.KeyValueStore;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.state.Stores;
import java.time.Instant;
import java.time.ZoneId;
import java.time.format.DateTimeFormatter;
public class TrafficRadarMonitoringCarCountExample {
private static final String DAILY_COUNT_STORE = "price-state-store";
private static final String DAILY_COUNT_TOPIC = "price-state-topic";
private static final String RADAR_COUNT_TOPIC = "car-radar-topic";
public static void countWithTransformValues(final StreamsBuilder builder) {
// Define a state store for tracking daily car counts
builder.addStateStore(
Stores.keyValueStoreBuilder(
Stores.inMemoryKeyValueStore(DAILY_COUNT_STORE),
org.apache.kafka.common.serialization.Serdes.String(),
org.apache.kafka.common.serialization.Serdes.Long()
)
);
final KStream<Void, String> radarStream = builder.stream(RADAR_COUNT_TOPIC);
// Apply the ValueTransformer with the state store
radarStream.transformValues(DailyCarCountTransformer::new, DAILY_COUNT_STORE)
.to(DAILY_COUNT_TOPIC);
}
public static void countWithProcessValues(final StreamsBuilder builder) {
// Define a state store for tracking daily car counts
builder.addStateStore(
Stores.keyValueStoreBuilder(
Stores.inMemoryKeyValueStore(DAILY_COUNT_STORE),
org.apache.kafka.common.serialization.Serdes.String(),
org.apache.kafka.common.serialization.Serdes.Long()
)
);
final KStream<Void, String> radarStream = builder.stream(RADAR_COUNT_TOPIC);
// Apply the FixedKeyProcessor with the state store
radarStream.processValues(DailyCarCountProcessor::new, DAILY_COUNT_STORE)
.to(DAILY_COUNT_TOPIC);
}
public static class DailyCarCountTransformer implements ValueTransformer<String, String> {
private KeyValueStore<String, Long> stateStore;
private static final DateTimeFormatter DATE_FORMATTER =
DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(ZoneId.systemDefault());
@Override
public void init(final org.apache.kafka.streams.processor.ProcessorContext context) {
// Access the state store
stateStore = context.getStateStore(DAILY_COUNT_STORE);
}
@Override
public String transform(final String value) {
if (value == null) {
return null; // Skip null events
}
// Derive the current day from the event timestamp
final long timestamp = System.currentTimeMillis(); // Use system time for simplicity
final String currentDay = DATE_FORMATTER.format(Instant.ofEpochMilli(timestamp));
// Retrieve the current count for the day
Long dailyCount = stateStore.get(currentDay);
if (dailyCount == null) {
dailyCount = 0L;
}
// Increment the count
dailyCount++;
stateStore.put(currentDay, dailyCount);
// Return the current day's count
return String.format("Day: %s, Car Count: %s", currentDay, dailyCount);
}
@Override
public void close() {
}
}
private static class DailyCarCountProcessor implements FixedKeyProcessor<Void, String, String> {
private FixedKeyProcessorContext<Void, String> context;
private KeyValueStore<String, Long> stateStore;
private static final DateTimeFormatter DATE_FORMATTER =
DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(ZoneId.systemDefault());
@Override
public void init(final FixedKeyProcessorContext<Void, String> context) {
this.context = context;
stateStore = context.getStateStore(DAILY_COUNT_STORE);
}
@Override
public void process(final FixedKeyRecord<Void, String> record) {
if (record.value() == null) {
return; // Skip null events
}
// Derive the current day from the event timestamp
final long timestamp = System.currentTimeMillis(); // Use system time for simplicity
final String currentDay = DATE_FORMATTER.format(Instant.ofEpochMilli(timestamp));
// Retrieve the current count for the day
Long dailyCount = stateStore.get(currentDay);
if (dailyCount == null) {
dailyCount = 0L;
}
// Increment the count
dailyCount++;
stateStore.put(currentDay, dailyCount);
// Emit the current day's count
context.forward(record.withValue(String.format("Day: %s, Car Count: %s", currentDay, dailyCount)));
}
}
}
```
## Keynotes
* The process and processValues APIs utilize ProcessorContext and Record objects for better type safety and flexibility.
* Implementations for Processor or ProcessorWithKey should manage state and logic clearly. Use context().forward() for
* emitting records downstream.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #18314:
URL: https://github.com/apache/kafka/pull/18314#issuecomment-2565928361
> may I format the code? Some blocks are not indented properly. I understand it might pollute the diff in code changes, so if I may, I would do it as a final commit.
Let's do this in a follow up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #18314:
URL: https://github.com/apache/kafka/pull/18314#issuecomment-2622612299
Thanks for this PR. Really cool to have this migration guide!
Merged to `trunk` and cherry-picked to `4.0` branch.
Feel free to do a "MINOR" follow-up PR for the Javadocs issue you found.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on code in PR #18314:
URL: https://github.com/apache/kafka/pull/18314#discussion_r1928018597
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3097,615 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
- </ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
- </thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- </tbody>
- </table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <div class="section" id="migrating-from-transform-methods-to-processor-api-papi">
+ <h2>
+ <a class="headerlink" href="#migrating-from-transform-methods-to-processor-api-papi"
+ title="Permalink to this headline">
+ Migrating from transform Methods to Processor API (PAPI)
+ </a>
+ </h2>
+ <h3>Overview of Changes</h3>
+ <p>
+ As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>, <code>transformValues</code>, and <code>flatTransformValues</code>, have
+ been removed. These methods have been replaced with the more versatile Processor API. This
+ guide provides detailed steps for migrating existing code to use the new Processor API and
+ explains the benefits of the changes.
+ </p>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ </ul>
+ <p>The Processor API now serves as a unified replacement for all these methods. It simplifies the
+ API surface while maintaining support for both stateless and stateful operations.</p>
+ <h3>Migration Process</h3>
+ <p>The migration process consists of:</p>
+ <ol>
+ <li>
+ Replace <code>Transformer</code> with <code>Processor</code> or <code>ValueTransformer</code> with
+ <code>FixedKeyProcessor</code>;
+ </li>
+ <li>
+ Replace record <code>key</code> and <code>value</code> with <code>Record</code> or <code>FixedKeyRecord</code>;
+ </li>
+ <li>
+ Rewrite the <code>transform</code> method of <code>Transformer</code> and <code>ValueTransformer</code> as
+ <code>process</code> or <code>processValues</code>;
+ </li>
+ <li>
+ Use the new <code>Record</code> or <code>FixedKeyRecord</code> as argument of the renamed method;</li>
+ <li>
+ Rewrite the return type of the renamed method to <code>void</code> and forward the record through the context;
+ and finally
+ </li>
+ <li>
+ Change the <code>KStream</code> call of the <code>transform</code> method to <code>process</code> or
+ <code>processValues</code>.
+ </li>
+ </ol>
+ <h3>Migration Examples</h3>
+ <p>
+ To migrate from the deprecated <code>transform</code>, <code>transformValues</code>, <code>flatTransform</code>, and
+ <code>flatTransformValues</code> methods to the Process API (PAPI) in Kafka Streams, follow these examples. The new
+ <code>process</code> and <code>processValues</code> APIs enable a more flexible and reusable approach by requiring
+ implementations of the <code>Processor</code> or <code>FixedKeyProcessor</code> interfaces.
+ </p>
+ <p>Here are examples to help you migrate:</p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Migrating from</th>
+ <th>Migrating to</th>
+ <th>State Type</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a></td>
+ <td><code>transform</code></td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>flatTransform</code></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>transformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>flatTransformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
+ </tbody>
+ </table>
+ <h4>Stateless Examples</h4>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li>
+ <strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g., INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to another processor.
+ </li>
+ <li>
+ <strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks, and
+ INFO logs are stored for basic reporting purposes.
+ </li>
+ </ul>
+ <p>
+ Below, methods <code>categorizeWithFlatTransform</code> and <code>categorizeWithProcess</code> show how you can
+ migrate from <code>flatTransform</code> to <code>process</code>.
+ </p>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithFlatTransform(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.flatTransform(() -> new LogSeverityTransformer())
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new);
+ }
- @Override
- void process(Record<PageId, Long> record) {
- // Here you would format and send the alert email.
- //
- // In this specific example, you would be able to include
- // information about the page's ID and its view count
- }
+ private static class LogSeverityTransformer implements Transformer<String, String, Iterable<KeyValue<String, String>>> {
+ @Override
+ public void init(org.apache.kafka.streams.processor.ProcessorContext context) {
+ }
- @Override
- void close() {
- // Any code for clean up would go here, for example tearing down the email client and anything
- // else you created in the init() method
- // This processor instance will not be used again after this call.
- }
+ @Override
+ public Iterable<KeyValue<String, String>> transform(String key, String value) {
+ if (value == null) {
+ return Collections.emptyList(); // Skip null values
+ }
-}</code></pre>
- <div class="admonition tip">
- <p><b>Tip</b></p>
- <p class="last">Even though we do not demonstrate it in this example, a stream processor can access any available state stores by
- calling <code class="docutils literal"><span class="pre">ProcessorContext#getStateStore()</span></code>.
- State stores are only available if they have been connected to the processor, or if they are global stores. While global stores do not need to be connected explicitly, they only allow for read-only access.
- There are two ways to connect state stores to a processor:
- <ul class="simple">
- <li>By passing the name of a store that has already been added via <code class="docutils literal"><span class="pre">Topology#addStateStore()</span></code> to the corresponding <code class="docutils literal"><span class="pre">KStream#process()</span></code> method call.</li>
- <li>Implementing <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the <code class="docutils literal"><span class="pre">ProcessorSupplier</span></code>
- passed to <code class="docutils literal"><span class="pre">KStream#process()</span></code>. In this case there is no need to call <code class="docutils literal"><span class="pre">StreamsBuilder#addStateStore()</span></code>
- beforehand, the store will be automatically added for you. You can also implement <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the
- <code class="docutils literal"><span class="pre">Value*</span></code> or <code class="docutils literal"><span class="pre">*WithKey</span></code> supplier variants, or <code class="docutils literal"><span class="pre">TransformerSupplier</span></code> or any of its variants.
- </li>
- </ul>
- </div>
- <p>Then we can leverage the <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code> processor in the DSL via <code class="docutils literal"><span class="pre">KStream#process</span></code>.</p>
- <pre class="line-numbers"><code class="language-java">KStream<String, GenericRecord> pageViews = ...;
-
-// Send an email notification when the view count of a page reaches one thousand.
-pageViews.groupByKey()
- .count()
- .filter((PageId pageId, Long viewCount) -> viewCount == 1000)
- // PopularPageEmailAlert is your custom processor that implements the
- // `Processor` interface, see further down below.
- .process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
- </div>
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ int colonIndex = value.indexOf(':');
+ String severity = colonIndex > 0 ? value.substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Create appropriate KeyValue pair based on severity
+ return switch (severity) {
+ case "ERROR" -> List.of(new KeyValue<>("ERROR", value));
+ case "WARN" -> List.of(new KeyValue<>("WARN", value));
+ case "INFO" -> Collections.emptyList(); // INFO logs are ignored
+ default -> List.of(new KeyValue<>("UNKNOWN", value));
+ };
+ }
+
+ @Override
+ public void close() {
+ }
+ }
+
+ private static class LogSeverityProcessor implements Processor<String, String, String, String> {
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ }
+
+ @Override
+ public void process(final Record<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ final int colonIndex = record.value().indexOf(':');
+ final String severity = colonIndex > 0 ? record.value().substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Route logs based on severity
+ switch (severity) {
+ case "ERROR":
+ context.forward(new Record<>(ERROR_LOGS_TOPIC, record.value(), record.timestamp()));
+ break;
+ case "WARN":
+ context.forward(new Record<>(WARN_LOGS_TOPIC, record.value(), record.timestamp()));
+ break;
+ case "INFO":
+ // INFO logs are ignored
+ break;
+ default:
+ // Forward to an "unknown" topic for logs with unrecognized severities
+ context.forward(new Record<>(UNKNOWN_LOGS_TOPIC, record.value(), record.timestamp()));
+ }
+ }
+ }
+}
+</code></pre>
+ <h5 id="replacing-slang-in-text-messages">Replacing Slang in Text Messages</h5>
+ <ul>
+ <li>
+ <strong>Idea:</strong> A messaging stream contains user-generated content, and you want to replace slang words
+ with their formal equivalents (e.g., "u" becomes "you", "brb" becomes "be
+ right back"). The operation only modifies the message value and keeps the key intact.
+ </li>
+ <li>
+ <strong>Real-World Context:</strong> In customer support chat systems, normalizing text by replacing slang with
+ formal equivalents ensures that automated sentiment analysis tools work accurately and provide reliable
+ insights.
+ </li>
+ </ul>
+ <p>
+ Below, methods <code>replaceWithFlatTransformValues</code> and <code>replaceWithProcessValues</code> show how you
+ can migrate from <code>flatTransformValues</code> to <code>processValues</code>.
+ </p>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessorContext;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
+
+public class ReplacingSlangTextInMessagesExample {
+ private static final Map<String, String> SLANG_DICTIONARY = Map.of(
+ "u", "you",
+ "brb", "be right back",
+ "omg", "oh my god",
+ "btw", "by the way"
+ );
+ private static final String INPUT_MESSAGES_TOPIC = "input-messages-topic";
+ private static final String OUTPUT_MESSAGES_TOPIC = "output-messages-topic";
+
+ public static void replaceWithFlatTransformValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.flatTransformValues(SlangReplacementTransformer::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ public static void replaceWithProcessValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.processValues(SlangReplacementProcessor::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ private static class SlangReplacementTransformer implements ValueTransformer<String, Iterable<String>> {
+ @Override
+ public void init(final org.apache.kafka.streams.processor.ProcessorContext context) {
+ }
+
+ @Override
+ public Iterable<String> transform(final String value) {
+ if (value == null) {
+ return Collections.emptyList(); // Skip null values
+ }
+
+ // Replace slang words in the message
+ final String[] words = value.split("\\s+");
+ return Arrays.asList(
+ Arrays.stream(words)
+ .map(word -> SLANG_DICTIONARY.getOrDefault(word, word))
+ .toArray(String[]::new)
+ );
+ }
+
+ @Override
+ public void close() {
+ }
+ }
+
+ private static class SlangReplacementProcessor implements FixedKeyProcessor<String, String, String> {
Review Comment:
Updated in Mardown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on PR #18314:
URL: https://github.com/apache/kafka/pull/18314#issuecomment-2620494379
@mjsax, I have pushed some updates. All that's missing are the descriptions for ContextualProcessors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on code in PR #18314:
URL: https://github.com/apache/kafka/pull/18314#discussion_r1933066974
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
+ </li>
+ <li><code>ProcessorSupplier</code>: A processor supplier that can create one or more <code>Processor</code>
+ instances; and</li>
+ <li><code>FixedKeyProcessorSupplier</code>: A processor supplier that can create one or more
+ <code>FixedKeyProcessor</code> instances.
+ </li>
+ </ul>
+ <h4>Examples</h4>
+ <p>Follow the examples below to learn how to apply <code>process</code> and <code>processValues</code> to your
+ <code>KStream</code>.
+ </p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Operation</th>
+ <th>State Type</th>
+ </tr>
</thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
</tbody>
</table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <h5 id="cumulative-discounts-for-a-loyalty-program"">Cumulative Discounts for a Loyalty Program</h5>
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #18314:
URL: https://github.com/apache/kafka/pull/18314#discussion_r1933254272
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
Review Comment:
Good idea. Let's do this in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on PR #18314:
URL: https://github.com/apache/kafka/pull/18314#issuecomment-2585901473
@mjsax, how about using the `PopularPageEmailAlert` of the _Applying processors and transformers (Processor API integration)_ section?
```java
package org.apache.kafka.streams.kstream;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.processor.AbstractProcessor;
import org.apache.kafka.streams.processor.api.Processor;
import org.apache.kafka.streams.processor.api.ProcessorContext;
import org.apache.kafka.streams.processor.api.Record;
public class PopularPageEmailAlertExample {
private static final String ALERTS_EMAIL = "alerts@yourcompany.com";
private static final String PAGE_VIEWS_TOPIC = "page-views-topic";
public static void alertWithOldProcess(StreamsBuilder builder) {
KStream<String, Long> pageViews = builder.stream(PAGE_VIEWS_TOPIC);
// Filter pages with exactly 1000 views and process them using the old API
pageViews.filter((pageId, viewCount) -> viewCount == 1000)
.process(PopularPageEmailAlertOld::new);
}
public static void alertWithNewProcess(StreamsBuilder builder) {
KStream<String, Long> pageViews = builder.stream(PAGE_VIEWS_TOPIC);
// Filter pages with exactly 1000 views and process them using the new API
pageViews.filter((pageId, viewCount) -> viewCount == 1000)
.process(PopularPageEmailAlertNew::new);
}
private static class PopularPageEmailAlertOld extends AbstractProcessor<String, Long> {
@Override
public void init(org.apache.kafka.streams.processor.ProcessorContext context) {
super.init(context);
System.out.println("Initialized email client for: " + ALERTS_EMAIL);
}
@Override
public void process(String key, Long value) {
if (value == null) return;
if (value == 1000) {
// Send an email alert
System.out.printf("ALERT (Old API): Page %s has reached 1000 views. Sending email to %s%n", key, ALERTS_EMAIL);
}
}
@Override
public void close() {
System.out.println("Tearing down email client for: " + ALERTS_EMAIL);
}
}
private static class PopularPageEmailAlertNew implements Processor<String, Long, Void, Void> {
@Override
public void init(ProcessorContext<Void, Void> context) {
System.out.println("Initialized email client for: " + ALERTS_EMAIL);
}
@Override
public void process(Record<String, Long> record) {
if (record.value() == null) return;
if (record.value() == 1000) {
// Send an email alert
System.out.printf("ALERT (New API): Page %s has reached 1000 views. Sending email to %s%n", record.key(), ALERTS_EMAIL);
}
}
@Override
public void close() {
System.out.println("Tearing down email client for: " + ALERTS_EMAIL);
}
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #18314:
URL: https://github.com/apache/kafka/pull/18314#issuecomment-2580965258
@fonsdant -- Any updates on this PR. Would be great to finish this before code freeze (Jan/15).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on PR #18314:
URL: https://github.com/apache/kafka/pull/18314#issuecomment-2608700567
@mjsax, I forgot to mention that I am updating the Markdown. So you could, for example, check the addition of the old process example. I'll try to send new patches for the slang example by tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on code in PR #18314:
URL: https://github.com/apache/kafka/pull/18314#discussion_r1933062865
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
+ </li>
+ <li><code>ProcessorSupplier</code>: A processor supplier that can create one or more <code>Processor</code>
+ instances; and</li>
+ <li><code>FixedKeyProcessorSupplier</code>: A processor supplier that can create one or more
+ <code>FixedKeyProcessor</code> instances.
+ </li>
+ </ul>
+ <h4>Examples</h4>
+ <p>Follow the examples below to learn how to apply <code>process</code> and <code>processValues</code> to your
+ <code>KStream</code>.
+ </p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Operation</th>
+ <th>State Type</th>
+ </tr>
</thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
</tbody>
</table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <h5 id="cumulative-discounts-for-a-loyalty-program"">Cumulative Discounts for a Loyalty Program</h5>
+ <ul>
+ <li><strong>Idea:</strong> A stream of purchase events contains user IDs and transaction amounts. Use a state
+ store
+ to accumulate the total spending of each user. When their total crosses a threshold, apply a discount on
+ their next transaction and update their accumulated total.</li>
+ <li><strong>Real-World Context:</strong> In a retail loyalty program, tracking cumulative customer spending
+ enables
+ dynamic rewards, such as issuing a discount when a customer's total purchases exceed a predefined limit.
+ </li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+
+public class CumulativeDiscountsForALoyaltyProgramExample {
+ private static final double DISCOUNT_THRESHOLD = 100.0;
+ private static final String CUSTOMER_SPENDING_STORE = "customer-spending-store";
+ private static final String DISCOUNT_NOTIFICATION_MESSAGE =
+ "Discount applied! You have received a reward for your purchases.";
+ private static final String DISCOUNT_NOTIFICATIONS_TOPIC = "discount-notifications-topic";
+ private static final String PURCHASE_EVENTS_TOPIC = "purchase-events-topic";
+
+ public static void applyDiscountWithProcess(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ Serdes.String(),
+ Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Processor with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.process(CumulativeDiscountProcessor::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ private static class CumulativeDiscountProcessor implements Processor<String, Double, String, String> {
+ private KeyValueStore<String, Double> spendingStore;
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public void process(final Record<String, Double> record) {
+ if (record.value() == null) {
+ return; // Skip null purchase amounts
+ }
- @Override
- void process(Record<PageId, Long> record) {
- // Here you would format and send the alert email.
- //
- // In this specific example, you would be able to include
- // information about the page's ID and its view count
- }
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(record.key());
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += record.value();
+ spendingStore.put(record.key(), currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(record.key(), currentSpending - DISCOUNT_THRESHOLD);
+ // Send a discount notification
+ context.forward(record.withValue(DISCOUNT_NOTIFICATION_MESSAGE));
+ }
+ }
+ }
+}</code></pre>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li><strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g.,
+ INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to a dedicated topic too.</li>
+ <li><strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity
+ ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks,
+ and
+ INFO logs are stored for basic reporting purposes.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- void close() {
- // Any code for clean up would go here, for example tearing down the email client and anything
- // else you created in the init() method
- // This processor instance will not be used again after this call.
- }
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new)
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
+
+ private static class LogSeverityProcessor extends ContextualProcessor<String, String, String, String> {
+ @Override
+ public void process(final Record<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ final int colonIndex = record.value().indexOf(':');
+ final String severity = colonIndex > 0 ? record.value().substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Route logs based on severity
+ switch (severity) {
+ case "ERROR":
+ context().forward(record.withKey(ERROR_LOGS_TOPIC));
+ break;
+ case "WARN":
+ context().forward(record.withKey(WARN_LOGS_TOPIC));
+ break;
+ case "INFO":
+ // INFO logs are ignored
+ break;
+ default:
+ // Forward to an "unknown" topic for logs with unrecognized severities
+ context().forward(record.withKey(UNKNOWN_LOGS_TOPIC));
+ }
+ }
+ }
}</code></pre>
- <div class="admonition tip">
- <p><b>Tip</b></p>
- <p class="last">Even though we do not demonstrate it in this example, a stream processor can access any available state stores by
- calling <code class="docutils literal"><span class="pre">ProcessorContext#getStateStore()</span></code>.
- State stores are only available if they have been connected to the processor, or if they are global stores. While global stores do not need to be connected explicitly, they only allow for read-only access.
- There are two ways to connect state stores to a processor:
- <ul class="simple">
- <li>By passing the name of a store that has already been added via <code class="docutils literal"><span class="pre">Topology#addStateStore()</span></code> to the corresponding <code class="docutils literal"><span class="pre">KStream#process()</span></code> method call.</li>
- <li>Implementing <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the <code class="docutils literal"><span class="pre">ProcessorSupplier</span></code>
- passed to <code class="docutils literal"><span class="pre">KStream#process()</span></code>. In this case there is no need to call <code class="docutils literal"><span class="pre">StreamsBuilder#addStateStore()</span></code>
- beforehand, the store will be automatically added for you. You can also implement <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the
- <code class="docutils literal"><span class="pre">Value*</span></code> or <code class="docutils literal"><span class="pre">*WithKey</span></code> supplier variants, or <code class="docutils literal"><span class="pre">TransformerSupplier</span></code> or any of its variants.
- </li>
- </ul>
- </div>
- <p>Then we can leverage the <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code> processor in the DSL via <code class="docutils literal"><span class="pre">KStream#process</span></code>.</p>
- <pre class="line-numbers"><code class="language-java">KStream<String, GenericRecord> pageViews = ...;
-
-// Send an email notification when the view count of a page reaches one thousand.
-pageViews.groupByKey()
- .count()
- .filter((PageId pageId, Long viewCount) -> viewCount == 1000)
- // PopularPageEmailAlert is your custom processor that implements the
- // `Processor` interface, see further down below.
- .process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
- </div>
+ <h5 id="traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</h5>
+ <ul>
+ <li><strong>Idea:</strong> A radar monitors cars passing along a road stretch. A system counts the cars for each
+ day, maintaining a cumulative total for the current day in a state store. At the end of the day, the count
+ is
+ emitted and the state is cleared for the next day.</li>
+ <li><strong>Real-World Context:</strong> A car counting system can be useful for determining measures for
+ widening
+ or controlling traffic depending on the number of cars passing through the monitored stretch.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.common.serialization;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessorContext;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.state.Stores;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.format.DateTimeFormatter;
+
+public class TrafficRadarMonitoringCarCountExample {
+ private static final String DAILY_COUNT_STORE = "price-state-store";
+ private static final String DAILY_COUNT_TOPIC = "price-state-topic";
+ private static final String RADAR_COUNT_TOPIC = "car-radar-topic";
+
+ public static void countWithProcessValues(final StreamsBuilder builder) {
+ // Define a state store for tracking daily car counts
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(DAILY_COUNT_STORE),
+ Serdes.String(),
+ Serdes.Long()
+ )
+ );
+ final KStream<Void, String> radarStream = builder.stream(RADAR_COUNT_TOPIC);
+ // Apply the FixedKeyProcessor with the state store
+ radarStream.processValues(DailyCarCountProcessor::new, DAILY_COUNT_STORE)
+ .to(DAILY_COUNT_TOPIC);
+ }
+
+ private static class DailyCarCountProcessor implements FixedKeyProcessor<Void, String, String> {
+ private FixedKeyProcessorContext<Void, String> context;
+ private KeyValueStore<String, Long> stateStore;
+ private static final DateTimeFormatter DATE_FORMATTER =
+ DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(ZoneId.systemDefault());
+
+ @Override
+ public void init(final FixedKeyProcessorContext<Void, String> context) {
+ this.context = context;
+ stateStore = context.getStateStore(DAILY_COUNT_STORE);
+ }
+
+ @Override
+ public void process(final FixedKeyRecord<Void, String> record) {
+ if (record.value() == null) {
+ return; // Skip null events
+ }
+
+ // Derive the current day from the event timestamp
+ final long timestamp = System.currentTimeMillis(); // Use system time for simplicity
+ final String currentDay = DATE_FORMATTER.format(Instant.ofEpochMilli(timestamp));
+ // Retrieve the current count for the day
+ Long dailyCount = stateStore.get(currentDay);
+ if (dailyCount == null) {
+ dailyCount = 0L;
+ }
+ // Increment the count
+ dailyCount++;
+ stateStore.put(currentDay, dailyCount);
+
+ // Emit the current day's count
+ context.forward(record.withValue(String.format("Day: %s, Car Count: %s", currentDay, dailyCount)));
+ }
+ }
+}</code></pre>
+ <h5 id="replacing-slang-in-text-messages">Replacing Slang in Text Messages</h5>
+ <ul>
+ <li><strong>Idea:</strong> A messaging stream contains user-generated content, and you want to replace slang
+ words
+ with their formal equivalents (e.g., "u" becomes "you", "brb" becomes "be
+ right back"). The operation only modifies the message value and keeps the key intact.</li>
+ <li><strong>Real-World Context:</strong> In customer support chat systems, normalizing text by replacing slang
+ with
+ formal equivalents ensures that automated sentiment analysis tools work accurately and provide reliable
+ insights.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualFixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
+
+public class ReplacingSlangTextInMessagesExample {
+ private static final Map<String, String> SLANG_DICTIONARY = Map.of(
+ "u", "you",
+ "brb", "be right back",
+ "omg", "oh my god",
+ "btw", "by the way"
+ );
+ private static final String INPUT_MESSAGES_TOPIC = "input-messages-topic";
+ private static final String OUTPUT_MESSAGES_TOPIC = "output-messages-topic";
+
+ public static void replaceWithProcessValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.processValues(SlangReplacementProcessor::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ private static class SlangReplacementProcessor extends ContextualFixedKeyProcessor<String, String, String> {
+ @Override
+ public void process(final FixedKeyRecord<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+
+ // Replace slang words in the message
+ final String[] words = record.value().split("\\s+");
+ for (final String word : words) {
+ String replacedWord = SLANG_DICTIONARY.getOrDefault(word, word);
+ context().forward(record.withValue(replacedWord));
+ }
+ }
+ }
+}</code></pre>
+ <h4>Keynotes</h4>
+ <ul>
+ <li><strong>Type Safety and Flexibility:</strong> The process and processValues APIs utilize
+ <code>ProcessorContext</code> and <code>Record</code> or <code>FixedKeyRecord</code> objects for better type
+ safety and flexibility of custom processing logic.
+ </li>
+ <li><strong>Clear State and Logic Management:</strong> Implementations for <code>Processor</code> or
+ <code>FixedKeyProcessor</code> should manage state and logic clearly. Use <code>context().forward()</code>
+ for emitting records downstream.
+ </li>
+ <li><strong>Unified API:</strong> Consolidates multiple methods into a single, versatile API.</li>
+ <li><strong>Future-Proof:</strong> Ensures compatibility with the latest Kafka Streams releases.</li>
+ </ul>
+ </div>
+ <div class="section" id="transformers-removal-and-migration-to-processors">
+ <a class="headerlink" href="#transformers-removal-and-migration-to-processors" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id37">Transformers removal and migration to processors</a>
+ </h3>
+ </a>
+ <p>As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>,<br><code>transformValues</code>, <code>flatTransformValues</code>, and
+ <code>process</code> have been removed. These methods have been replaced with the<br>more versatile Processor API.
+ This guide provides detailed steps for migrating existing code to use the new Processor<br>API and explains the
+ benefits of the changes.</p>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ <li><code>KStream#process</code></li>
+ </ul>
+ <p>The Processor API now serves as a unified replacement for all these methods. It simplifies the API surface
+ while<br>maintaining support for both stateless and stateful operations.</p>
+ <h4>Migration Examples</h4>
+ <p>To migrate from the deprecated <code>transform</code>, <code>transformValues</code>, <code>flatTransform</code>, and
+ <code>flatTransformValues</code> methods to the<br>Processor API (PAPI) in Kafka Streams, let's resume the
+ previouss examples. The new <code>process</code> and <code>processValues</code><br>methods enable a more flexible
+ and reusable approach by requiring implementations of the <code>Processor</code>
+ or<br><code>FixedKeyProcessor</code> interfaces.</p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Migrating from</th>
+ <th>Migrating to</th>
+ <th>State Type</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program-removal">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>transform</code></td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity-removal">Categorizing Logs by Severity</a></td>
+ <td><code>flatTransform</code></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count-removal">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>transformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages-removal">Replacing Slang in Text Messages</a></td>
+ <td><code>flatTransformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
+ </tbody>
+ </table>
+ <h5 id="cumulative-discounts-for-a-loyalty-program-removal">Cumulative Discounts for a Loyalty Program</h5>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+
+public class CumulativeDiscountsForALoyaltyProgramExample {
+ private static final double DISCOUNT_THRESHOLD = 100.0;
+ private static final String CUSTOMER_SPENDING_STORE = "customer-spending-store";
+ private static final String DISCOUNT_NOTIFICATION_MESSAGE =
+ "Discount applied! You have received a reward for your purchases.";
+ private static final String DISCOUNT_NOTIFICATIONS_TOPIC = "discount-notifications-topic";
+ private static final String PURCHASE_EVENTS_TOPIC = "purchase-events-topic";
+
+ public static void applyDiscountWithTransform(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ Serdes.String(),
+ Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Transformer with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.transform(CumulativeDiscountTransformer::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
+
+ public static void applyDiscountWithProcess(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ org.apache.kafka.common.serialization.Serdes.String(),
+ org.apache.kafka.common.serialization.Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Processor with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.process(CumulativeDiscountProcessor::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
+
+ private static class CumulativeDiscountTransformer implements Transformer<String, Double, KeyValue<String, String>> {
+ private KeyValueStore<String, Double> spendingStore;
+
+ @Override
+ public void init(final org.apache.kafka.streams.processor.ProcessorContext context) {
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public KeyValue<String, String> transform(final String key, final Double value) {
+ if (value == null) {
+ return null; // Skip null purchase amounts
+ }
+
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(key);
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += value;
+ spendingStore.put(key, currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(key, currentSpending - DISCOUNT_THRESHOLD);
+ // Return a notification message
+ return new KeyValue<>(key, DISCOUNT_NOTIFICATION_MESSAGE);
+ }
+ return null; // No discount, so no output for this record
+ }
+
+ @Override
+ public void close() {
+ }
+ }
+
+ private static class CumulativeDiscountProcessor implements Processor<String, Double, String, String> {
+ private KeyValueStore<String, Double> spendingStore;
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public void process(final Record<String, Double> record) {
+ if (record.value() == null) {
+ return; // Skip null purchase amounts
+ }
+
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(record.key());
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += record.value();
+ spendingStore.put(record.key(), currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(record.key(), currentSpending - DISCOUNT_THRESHOLD);
+ // Send a discount notification
+ context.forward(record.withValue(DISCOUNT_NOTIFICATION_MESSAGE));
+ }
+ }
+ }
+}</code></pre>
+ <h5 id="categorizing-logs-by-severity-removal">Categorizing Logs by Severity</h5>
+ <p>Below, methods <code>categorizeWithFlatTransform</code> and <code>categorizeWithProcess</code> show how you can
+ migrate from <code>flatTransform</code><br>to <code>process</code>.</p>
Review Comment:
Replaced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #18314:
URL: https://github.com/apache/kafka/pull/18314#discussion_r1933254272
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
Review Comment:
Good idea. Let's do this in a separate PR. Of course, we should still make an update here, too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax merged PR #18314:
URL: https://github.com/apache/kafka/pull/18314
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on code in PR #18314:
URL: https://github.com/apache/kafka/pull/18314#discussion_r1933125039
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
Review Comment:
Oh, so I think we have an opportunity to update the javadocs. I have took these descriptions from there.
https://github.com/apache/kafka/blob/f18457f2b8f7adc5fb8ea2c46dcf38973aaf4667/streams/src/main/java/org/apache/kafka/streams/processor/api/ContextualProcessor.java#L21
https://github.com/apache/kafka/blob/f18457f2b8f7adc5fb8ea2c46dcf38973aaf4667/streams/src/main/java/org/apache/kafka/streams/processor/api/ContextualFixedKeyProcessor.java#L22
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on PR #18314:
URL: https://github.com/apache/kafka/pull/18314#issuecomment-2585000776
@mjsax, I removed the "migration process" section to simplify the guide, unified the method reference, added the remaining `to`, fixed the `flatTransformValues`-`processValues` behavior, replaced `new Record` with `withValue` and `withKey`, replaced `ValueTransformer` with `ValueTransformerWithKey`, and sorted the examples. The only thing left to do is to create an example for removing `process`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #18314:
URL: https://github.com/apache/kafka/pull/18314#issuecomment-2591103697
No worries. Let's just try to finish asap, and if need be, we update the docs after the release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on code in PR #18314:
URL: https://github.com/apache/kafka/pull/18314#discussion_r1928018450
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3097,615 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
- </ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
- </thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- </tbody>
- </table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <div class="section" id="migrating-from-transform-methods-to-processor-api-papi">
+ <h2>
+ <a class="headerlink" href="#migrating-from-transform-methods-to-processor-api-papi"
+ title="Permalink to this headline">
+ Migrating from transform Methods to Processor API (PAPI)
+ </a>
+ </h2>
+ <h3>Overview of Changes</h3>
+ <p>
+ As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>, <code>transformValues</code>, and <code>flatTransformValues</code>, have
+ been removed. These methods have been replaced with the more versatile Processor API. This
+ guide provides detailed steps for migrating existing code to use the new Processor API and
+ explains the benefits of the changes.
+ </p>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ </ul>
+ <p>The Processor API now serves as a unified replacement for all these methods. It simplifies the
+ API surface while maintaining support for both stateless and stateful operations.</p>
+ <h3>Migration Process</h3>
+ <p>The migration process consists of:</p>
+ <ol>
+ <li>
+ Replace <code>Transformer</code> with <code>Processor</code> or <code>ValueTransformer</code> with
+ <code>FixedKeyProcessor</code>;
+ </li>
+ <li>
+ Replace record <code>key</code> and <code>value</code> with <code>Record</code> or <code>FixedKeyRecord</code>;
+ </li>
+ <li>
+ Rewrite the <code>transform</code> method of <code>Transformer</code> and <code>ValueTransformer</code> as
+ <code>process</code> or <code>processValues</code>;
+ </li>
+ <li>
+ Use the new <code>Record</code> or <code>FixedKeyRecord</code> as argument of the renamed method;</li>
+ <li>
+ Rewrite the return type of the renamed method to <code>void</code> and forward the record through the context;
+ and finally
+ </li>
+ <li>
+ Change the <code>KStream</code> call of the <code>transform</code> method to <code>process</code> or
+ <code>processValues</code>.
+ </li>
+ </ol>
+ <h3>Migration Examples</h3>
+ <p>
+ To migrate from the deprecated <code>transform</code>, <code>transformValues</code>, <code>flatTransform</code>, and
+ <code>flatTransformValues</code> methods to the Process API (PAPI) in Kafka Streams, follow these examples. The new
+ <code>process</code> and <code>processValues</code> APIs enable a more flexible and reusable approach by requiring
+ implementations of the <code>Processor</code> or <code>FixedKeyProcessor</code> interfaces.
+ </p>
+ <p>Here are examples to help you migrate:</p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Migrating from</th>
+ <th>Migrating to</th>
+ <th>State Type</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a></td>
+ <td><code>transform</code></td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>flatTransform</code></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>transformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>flatTransformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
+ </tbody>
+ </table>
+ <h4>Stateless Examples</h4>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li>
+ <strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g., INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to another processor.
Review Comment:
Updated in Mardown.
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3097,615 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
- </ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
- </thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- </tbody>
- </table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <div class="section" id="migrating-from-transform-methods-to-processor-api-papi">
+ <h2>
+ <a class="headerlink" href="#migrating-from-transform-methods-to-processor-api-papi"
+ title="Permalink to this headline">
+ Migrating from transform Methods to Processor API (PAPI)
+ </a>
+ </h2>
+ <h3>Overview of Changes</h3>
+ <p>
+ As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>, <code>transformValues</code>, and <code>flatTransformValues</code>, have
+ been removed. These methods have been replaced with the more versatile Processor API. This
+ guide provides detailed steps for migrating existing code to use the new Processor API and
+ explains the benefits of the changes.
+ </p>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ </ul>
+ <p>The Processor API now serves as a unified replacement for all these methods. It simplifies the
+ API surface while maintaining support for both stateless and stateful operations.</p>
+ <h3>Migration Process</h3>
+ <p>The migration process consists of:</p>
+ <ol>
+ <li>
+ Replace <code>Transformer</code> with <code>Processor</code> or <code>ValueTransformer</code> with
+ <code>FixedKeyProcessor</code>;
+ </li>
+ <li>
+ Replace record <code>key</code> and <code>value</code> with <code>Record</code> or <code>FixedKeyRecord</code>;
+ </li>
+ <li>
+ Rewrite the <code>transform</code> method of <code>Transformer</code> and <code>ValueTransformer</code> as
+ <code>process</code> or <code>processValues</code>;
+ </li>
+ <li>
+ Use the new <code>Record</code> or <code>FixedKeyRecord</code> as argument of the renamed method;</li>
+ <li>
+ Rewrite the return type of the renamed method to <code>void</code> and forward the record through the context;
+ and finally
+ </li>
+ <li>
+ Change the <code>KStream</code> call of the <code>transform</code> method to <code>process</code> or
+ <code>processValues</code>.
+ </li>
+ </ol>
+ <h3>Migration Examples</h3>
+ <p>
+ To migrate from the deprecated <code>transform</code>, <code>transformValues</code>, <code>flatTransform</code>, and
+ <code>flatTransformValues</code> methods to the Process API (PAPI) in Kafka Streams, follow these examples. The new
+ <code>process</code> and <code>processValues</code> APIs enable a more flexible and reusable approach by requiring
+ implementations of the <code>Processor</code> or <code>FixedKeyProcessor</code> interfaces.
+ </p>
+ <p>Here are examples to help you migrate:</p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Migrating from</th>
+ <th>Migrating to</th>
+ <th>State Type</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a></td>
+ <td><code>transform</code></td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>flatTransform</code></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>transformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>flatTransformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
+ </tbody>
+ </table>
+ <h4>Stateless Examples</h4>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li>
+ <strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g., INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to another processor.
+ </li>
+ <li>
+ <strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks, and
+ INFO logs are stored for basic reporting purposes.
+ </li>
+ </ul>
+ <p>
+ Below, methods <code>categorizeWithFlatTransform</code> and <code>categorizeWithProcess</code> show how you can
+ migrate from <code>flatTransform</code> to <code>process</code>.
+ </p>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithFlatTransform(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.flatTransform(() -> new LogSeverityTransformer())
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new);
+ }
- @Override
- void process(Record<PageId, Long> record) {
- // Here you would format and send the alert email.
- //
- // In this specific example, you would be able to include
- // information about the page's ID and its view count
- }
+ private static class LogSeverityTransformer implements Transformer<String, String, Iterable<KeyValue<String, String>>> {
+ @Override
+ public void init(org.apache.kafka.streams.processor.ProcessorContext context) {
+ }
- @Override
- void close() {
- // Any code for clean up would go here, for example tearing down the email client and anything
- // else you created in the init() method
- // This processor instance will not be used again after this call.
- }
+ @Override
+ public Iterable<KeyValue<String, String>> transform(String key, String value) {
+ if (value == null) {
+ return Collections.emptyList(); // Skip null values
+ }
-}</code></pre>
- <div class="admonition tip">
- <p><b>Tip</b></p>
- <p class="last">Even though we do not demonstrate it in this example, a stream processor can access any available state stores by
- calling <code class="docutils literal"><span class="pre">ProcessorContext#getStateStore()</span></code>.
- State stores are only available if they have been connected to the processor, or if they are global stores. While global stores do not need to be connected explicitly, they only allow for read-only access.
- There are two ways to connect state stores to a processor:
- <ul class="simple">
- <li>By passing the name of a store that has already been added via <code class="docutils literal"><span class="pre">Topology#addStateStore()</span></code> to the corresponding <code class="docutils literal"><span class="pre">KStream#process()</span></code> method call.</li>
- <li>Implementing <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the <code class="docutils literal"><span class="pre">ProcessorSupplier</span></code>
- passed to <code class="docutils literal"><span class="pre">KStream#process()</span></code>. In this case there is no need to call <code class="docutils literal"><span class="pre">StreamsBuilder#addStateStore()</span></code>
- beforehand, the store will be automatically added for you. You can also implement <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the
- <code class="docutils literal"><span class="pre">Value*</span></code> or <code class="docutils literal"><span class="pre">*WithKey</span></code> supplier variants, or <code class="docutils literal"><span class="pre">TransformerSupplier</span></code> or any of its variants.
- </li>
- </ul>
- </div>
- <p>Then we can leverage the <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code> processor in the DSL via <code class="docutils literal"><span class="pre">KStream#process</span></code>.</p>
- <pre class="line-numbers"><code class="language-java">KStream<String, GenericRecord> pageViews = ...;
-
-// Send an email notification when the view count of a page reaches one thousand.
-pageViews.groupByKey()
- .count()
- .filter((PageId pageId, Long viewCount) -> viewCount == 1000)
- // PopularPageEmailAlert is your custom processor that implements the
- // `Processor` interface, see further down below.
- .process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
- </div>
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ int colonIndex = value.indexOf(':');
+ String severity = colonIndex > 0 ? value.substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Create appropriate KeyValue pair based on severity
+ return switch (severity) {
+ case "ERROR" -> List.of(new KeyValue<>("ERROR", value));
+ case "WARN" -> List.of(new KeyValue<>("WARN", value));
+ case "INFO" -> Collections.emptyList(); // INFO logs are ignored
+ default -> List.of(new KeyValue<>("UNKNOWN", value));
+ };
+ }
+
+ @Override
+ public void close() {
+ }
+ }
+
+ private static class LogSeverityProcessor implements Processor<String, String, String, String> {
Review Comment:
Updated in Mardown
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on code in PR #18314:
URL: https://github.com/apache/kafka/pull/18314#discussion_r1928003404
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3097,615 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
- </ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
- </thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- </tbody>
- </table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <div class="section" id="migrating-from-transform-methods-to-processor-api-papi">
+ <h2>
+ <a class="headerlink" href="#migrating-from-transform-methods-to-processor-api-papi"
+ title="Permalink to this headline">
+ Migrating from transform Methods to Processor API (PAPI)
+ </a>
+ </h2>
+ <h3>Overview of Changes</h3>
+ <p>
+ As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>, <code>transformValues</code>, and <code>flatTransformValues</code>, have
+ been removed. These methods have been replaced with the more versatile Processor API. This
+ guide provides detailed steps for migrating existing code to use the new Processor API and
+ explains the benefits of the changes.
+ </p>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ </ul>
+ <p>The Processor API now serves as a unified replacement for all these methods. It simplifies the
+ API surface while maintaining support for both stateless and stateful operations.</p>
+ <h3>Migration Process</h3>
+ <p>The migration process consists of:</p>
+ <ol>
+ <li>
+ Replace <code>Transformer</code> with <code>Processor</code> or <code>ValueTransformer</code> with
+ <code>FixedKeyProcessor</code>;
+ </li>
+ <li>
+ Replace record <code>key</code> and <code>value</code> with <code>Record</code> or <code>FixedKeyRecord</code>;
+ </li>
+ <li>
+ Rewrite the <code>transform</code> method of <code>Transformer</code> and <code>ValueTransformer</code> as
+ <code>process</code> or <code>processValues</code>;
+ </li>
+ <li>
+ Use the new <code>Record</code> or <code>FixedKeyRecord</code> as argument of the renamed method;</li>
+ <li>
+ Rewrite the return type of the renamed method to <code>void</code> and forward the record through the context;
+ and finally
+ </li>
+ <li>
+ Change the <code>KStream</code> call of the <code>transform</code> method to <code>process</code> or
+ <code>processValues</code>.
+ </li>
+ </ol>
+ <h3>Migration Examples</h3>
+ <p>
+ To migrate from the deprecated <code>transform</code>, <code>transformValues</code>, <code>flatTransform</code>, and
+ <code>flatTransformValues</code> methods to the Process API (PAPI) in Kafka Streams, follow these examples. The new
+ <code>process</code> and <code>processValues</code> APIs enable a more flexible and reusable approach by requiring
+ implementations of the <code>Processor</code> or <code>FixedKeyProcessor</code> interfaces.
+ </p>
+ <p>Here are examples to help you migrate:</p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Migrating from</th>
+ <th>Migrating to</th>
+ <th>State Type</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a></td>
+ <td><code>transform</code></td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>flatTransform</code></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>transformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>flatTransformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
+ </tbody>
+ </table>
+ <h4>Stateless Examples</h4>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li>
+ <strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g., INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to another processor.
+ </li>
+ <li>
+ <strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks, and
+ INFO logs are stored for basic reporting purposes.
+ </li>
+ </ul>
+ <p>
+ Below, methods <code>categorizeWithFlatTransform</code> and <code>categorizeWithProcess</code> show how you can
+ migrate from <code>flatTransform</code> to <code>process</code>.
+ </p>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithFlatTransform(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.flatTransform(() -> new LogSeverityTransformer())
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new);
+ }
- @Override
- void process(Record<PageId, Long> record) {
- // Here you would format and send the alert email.
- //
- // In this specific example, you would be able to include
- // information about the page's ID and its view count
- }
+ private static class LogSeverityTransformer implements Transformer<String, String, Iterable<KeyValue<String, String>>> {
+ @Override
+ public void init(org.apache.kafka.streams.processor.ProcessorContext context) {
+ }
- @Override
- void close() {
- // Any code for clean up would go here, for example tearing down the email client and anything
- // else you created in the init() method
- // This processor instance will not be used again after this call.
- }
+ @Override
+ public Iterable<KeyValue<String, String>> transform(String key, String value) {
+ if (value == null) {
+ return Collections.emptyList(); // Skip null values
+ }
-}</code></pre>
- <div class="admonition tip">
- <p><b>Tip</b></p>
- <p class="last">Even though we do not demonstrate it in this example, a stream processor can access any available state stores by
- calling <code class="docutils literal"><span class="pre">ProcessorContext#getStateStore()</span></code>.
- State stores are only available if they have been connected to the processor, or if they are global stores. While global stores do not need to be connected explicitly, they only allow for read-only access.
- There are two ways to connect state stores to a processor:
- <ul class="simple">
- <li>By passing the name of a store that has already been added via <code class="docutils literal"><span class="pre">Topology#addStateStore()</span></code> to the corresponding <code class="docutils literal"><span class="pre">KStream#process()</span></code> method call.</li>
- <li>Implementing <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the <code class="docutils literal"><span class="pre">ProcessorSupplier</span></code>
- passed to <code class="docutils literal"><span class="pre">KStream#process()</span></code>. In this case there is no need to call <code class="docutils literal"><span class="pre">StreamsBuilder#addStateStore()</span></code>
- beforehand, the store will be automatically added for you. You can also implement <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the
- <code class="docutils literal"><span class="pre">Value*</span></code> or <code class="docutils literal"><span class="pre">*WithKey</span></code> supplier variants, or <code class="docutils literal"><span class="pre">TransformerSupplier</span></code> or any of its variants.
- </li>
- </ul>
- </div>
- <p>Then we can leverage the <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code> processor in the DSL via <code class="docutils literal"><span class="pre">KStream#process</span></code>.</p>
- <pre class="line-numbers"><code class="language-java">KStream<String, GenericRecord> pageViews = ...;
-
-// Send an email notification when the view count of a page reaches one thousand.
-pageViews.groupByKey()
- .count()
- .filter((PageId pageId, Long viewCount) -> viewCount == 1000)
- // PopularPageEmailAlert is your custom processor that implements the
- // `Processor` interface, see further down below.
- .process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
- </div>
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ int colonIndex = value.indexOf(':');
+ String severity = colonIndex > 0 ? value.substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Create appropriate KeyValue pair based on severity
+ return switch (severity) {
+ case "ERROR" -> List.of(new KeyValue<>("ERROR", value));
+ case "WARN" -> List.of(new KeyValue<>("WARN", value));
+ case "INFO" -> Collections.emptyList(); // INFO logs are ignored
+ default -> List.of(new KeyValue<>("UNKNOWN", value));
+ };
+ }
+
+ @Override
+ public void close() {
+ }
+ }
+
+ private static class LogSeverityProcessor implements Processor<String, String, String, String> {
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ }
+
+ @Override
+ public void process(final Record<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ final int colonIndex = record.value().indexOf(':');
+ final String severity = colonIndex > 0 ? record.value().substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Route logs based on severity
+ switch (severity) {
+ case "ERROR":
+ context.forward(new Record<>(ERROR_LOGS_TOPIC, record.value(), record.timestamp()));
+ break;
+ case "WARN":
+ context.forward(new Record<>(WARN_LOGS_TOPIC, record.value(), record.timestamp()));
+ break;
+ case "INFO":
+ // INFO logs are ignored
+ break;
+ default:
+ // Forward to an "unknown" topic for logs with unrecognized severities
+ context.forward(new Record<>(UNKNOWN_LOGS_TOPIC, record.value(), record.timestamp()));
+ }
+ }
+ }
+}
+</code></pre>
+ <h5 id="replacing-slang-in-text-messages">Replacing Slang in Text Messages</h5>
+ <ul>
+ <li>
+ <strong>Idea:</strong> A messaging stream contains user-generated content, and you want to replace slang words
+ with their formal equivalents (e.g., "u" becomes "you", "brb" becomes "be
+ right back"). The operation only modifies the message value and keeps the key intact.
+ </li>
+ <li>
+ <strong>Real-World Context:</strong> In customer support chat systems, normalizing text by replacing slang with
+ formal equivalents ensures that automated sentiment analysis tools work accurately and provide reliable
+ insights.
+ </li>
+ </ul>
+ <p>
+ Below, methods <code>replaceWithFlatTransformValues</code> and <code>replaceWithProcessValues</code> show how you
+ can migrate from <code>flatTransformValues</code> to <code>processValues</code>.
+ </p>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessorContext;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
+
+public class ReplacingSlangTextInMessagesExample {
+ private static final Map<String, String> SLANG_DICTIONARY = Map.of(
+ "u", "you",
+ "brb", "be right back",
+ "omg", "oh my god",
+ "btw", "by the way"
+ );
+ private static final String INPUT_MESSAGES_TOPIC = "input-messages-topic";
+ private static final String OUTPUT_MESSAGES_TOPIC = "output-messages-topic";
+
+ public static void replaceWithFlatTransformValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.flatTransformValues(SlangReplacementTransformer::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ public static void replaceWithProcessValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.processValues(SlangReplacementProcessor::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ private static class SlangReplacementTransformer implements ValueTransformer<String, Iterable<String>> {
+ @Override
+ public void init(final org.apache.kafka.streams.processor.ProcessorContext context) {
+ }
+
+ @Override
+ public Iterable<String> transform(final String value) {
+ if (value == null) {
+ return Collections.emptyList(); // Skip null values
+ }
+
+ // Replace slang words in the message
+ final String[] words = value.split("\\s+");
+ return Arrays.asList(
+ Arrays.stream(words)
+ .map(word -> SLANG_DICTIONARY.getOrDefault(word, word))
+ .toArray(String[]::new)
Review Comment:
I think it works as expected and the cause of the confusion might be the return types of `ValueTransformer#transform` and `FixedKeyProcessor#process`. While `transform` returns an iterable, `process` is void. So we even have to change the logic inside both to achieve the same goal in this example. Even so, when reading from the output topic, we will have each element of either the `iterable` or `forward` record values iteration. I wrote the test below to check this. We can see that every `readKeyValue` call from the output topic returns one of these elements.
```java
package org.example;
import java.util.List;
import java.util.Properties;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.common.serialization.Serde;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.streams.KeyValue;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.kstream.Consumed;
import org.apache.kafka.streams.test.TestRecord;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
public class ReplacingSlangTextInMessagesExampleTest {
private final static String FLAT_TRANSFORM_INPUT_TOPIC = "flat-transform-input-topic";
private final static String FLAT_TRANSFORM_OUTPUT_TOPIC = "flat-transform-output-topic";
private final static String PROCESS_INPUT_TOPIC = "process-input-topic";
private final static String PROCESS_OUTPUT_TOPIC = "process-output-topic";
private TestInputTopic<String, String> flatTransformInputTopic;
private TestInputTopic<String, String> processInputTopic;
private TestOutputTopic<String, String> flatTransformOutputTopic;
private TestOutputTopic<String, String> processOutputTopic;
private TopologyTestDriver flatTransformTestDriver;
private TopologyTestDriver processTestDriver;
@BeforeEach
public void setup() {
Properties props = new Properties();
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.put(StreamsConfig.ACCEPTABLE_RECOVERY_LAG_CONFIG, 0);
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test-app");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
props.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.StringSerde.class);
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.StringSerde.class);
props.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);
props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE_V2);
props.put(StreamsConfig.STATESTORE_CACHE_MAX_BYTES_CONFIG, 0);
final StreamsBuilder flatTransformBuilder = new StreamsBuilder();
flatTransformBuilder.stream(FLAT_TRANSFORM_INPUT_TOPIC, Consumed.with(Serdes.String(), Serdes.String()))
.flatTransformValues(SlangReplacementTransformer::new)
.to(FLAT_TRANSFORM_OUTPUT_TOPIC);
final Topology flatTransformTopology = flatTransformBuilder.build();
flatTransformTestDriver = new TopologyTestDriver(flatTransformTopology, props);
Serde<String> stringSerde = Serdes.String();
flatTransformInputTopic = flatTransformTestDriver.createInputTopic(FLAT_TRANSFORM_INPUT_TOPIC, stringSerde.serializer(), stringSerde.serializer());
flatTransformOutputTopic = flatTransformTestDriver.createOutputTopic(FLAT_TRANSFORM_OUTPUT_TOPIC, stringSerde.deserializer(), stringSerde.deserializer());
final StreamsBuilder processBuilder = new StreamsBuilder();
processBuilder.<String, String>stream(PROCESS_INPUT_TOPIC)
.processValues(SlangReplacementProcessor::new)
.to(PROCESS_OUTPUT_TOPIC);
final Topology processTopology = processBuilder.build();
processTestDriver = new TopologyTestDriver(processTopology, props);
processInputTopic = processTestDriver.createInputTopic(PROCESS_INPUT_TOPIC, stringSerde.serializer(), stringSerde.serializer());
processOutputTopic = processTestDriver.createOutputTopic(PROCESS_OUTPUT_TOPIC, stringSerde.deserializer(), stringSerde.deserializer());
}
@AfterEach
public void tearDown() {
flatTransformTestDriver.close();
processTestDriver.close();
}
@Test
public void test1() {
final KeyValue<String, String> input = KeyValue.pair("key1", "u brb omg this is crazy");
final KeyValue<String, List<String>> output = KeyValue.pair("key1", List.of("you", "be right back", "oh my god", "this", "is", "crazy"));
flatTransformInputTopic.pipeInput(input.key, input.value);
processInputTopic.pipeInput(input.key, input.value);
for (final String expression : output.value) {
final KeyValue<String, String> flatTransformRecord = flatTransformOutputTopic.readKeyValue();
final KeyValue<String, String> processRecord = processOutputTopic.readKeyValue();
assertEquals(output.key, flatTransformRecord.key);
assertEquals(output.key, processRecord.key);
assertEquals(expression, flatTransformRecord.value);
assertEquals(expression, processRecord.value);
}
assertTrue(flatTransformOutputTopic.isEmpty());
assertTrue(processOutputTopic.isEmpty());
}
@Test
public void test2() {
final TestRecord<String, String> input = new TestRecord<>("key2", "btw did u see that?");
final TestRecord<String, List<String>> output = new TestRecord<>("key2", List.of("by the way", "did", "you", "see", "that?"));
flatTransformInputTopic.pipeInput(input);
processInputTopic.pipeInput(input);
for (final String expression : output.value()) {
final TestRecord<String, String> flatTransformRecord = flatTransformOutputTopic.readRecord();
final TestRecord<String, String> processRecord = processOutputTopic.readRecord();
assertEquals(output.key(), flatTransformRecord.key());
assertEquals(output.key(), processRecord.key());
assertEquals(expression, flatTransformRecord.value());
assertEquals(expression, processRecord.value());
}
assertTrue(flatTransformOutputTopic.isEmpty());
assertTrue(processOutputTopic.isEmpty());
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #18314:
URL: https://github.com/apache/kafka/pull/18314#issuecomment-2614033630
I did not see any push into the PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on code in PR #18314:
URL: https://github.com/apache/kafka/pull/18314#discussion_r1933065386
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
+ </li>
+ <li><code>ProcessorSupplier</code>: A processor supplier that can create one or more <code>Processor</code>
+ instances; and</li>
+ <li><code>FixedKeyProcessorSupplier</code>: A processor supplier that can create one or more
+ <code>FixedKeyProcessor</code> instances.
+ </li>
+ </ul>
+ <h4>Examples</h4>
+ <p>Follow the examples below to learn how to apply <code>process</code> and <code>processValues</code> to your
+ <code>KStream</code>.
+ </p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Operation</th>
+ <th>State Type</th>
+ </tr>
</thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
</tbody>
</table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <h5 id="cumulative-discounts-for-a-loyalty-program"">Cumulative Discounts for a Loyalty Program</h5>
+ <ul>
+ <li><strong>Idea:</strong> A stream of purchase events contains user IDs and transaction amounts. Use a state
+ store
+ to accumulate the total spending of each user. When their total crosses a threshold, apply a discount on
+ their next transaction and update their accumulated total.</li>
+ <li><strong>Real-World Context:</strong> In a retail loyalty program, tracking cumulative customer spending
+ enables
+ dynamic rewards, such as issuing a discount when a customer's total purchases exceed a predefined limit.
+ </li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+
+public class CumulativeDiscountsForALoyaltyProgramExample {
+ private static final double DISCOUNT_THRESHOLD = 100.0;
+ private static final String CUSTOMER_SPENDING_STORE = "customer-spending-store";
+ private static final String DISCOUNT_NOTIFICATION_MESSAGE =
+ "Discount applied! You have received a reward for your purchases.";
+ private static final String DISCOUNT_NOTIFICATIONS_TOPIC = "discount-notifications-topic";
+ private static final String PURCHASE_EVENTS_TOPIC = "purchase-events-topic";
+
+ public static void applyDiscountWithProcess(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ Serdes.String(),
+ Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Processor with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.process(CumulativeDiscountProcessor::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ private static class CumulativeDiscountProcessor implements Processor<String, Double, String, String> {
+ private KeyValueStore<String, Double> spendingStore;
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public void process(final Record<String, Double> record) {
+ if (record.value() == null) {
+ return; // Skip null purchase amounts
+ }
- @Override
- void process(Record<PageId, Long> record) {
- // Here you would format and send the alert email.
- //
- // In this specific example, you would be able to include
- // information about the page's ID and its view count
- }
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(record.key());
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += record.value();
+ spendingStore.put(record.key(), currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(record.key(), currentSpending - DISCOUNT_THRESHOLD);
+ // Send a discount notification
+ context.forward(record.withValue(DISCOUNT_NOTIFICATION_MESSAGE));
+ }
+ }
+ }
+}</code></pre>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li><strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g.,
+ INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to a dedicated topic too.</li>
+ <li><strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity
+ ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks,
+ and
+ INFO logs are stored for basic reporting purposes.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- void close() {
- // Any code for clean up would go here, for example tearing down the email client and anything
- // else you created in the init() method
- // This processor instance will not be used again after this call.
- }
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new)
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
+
+ private static class LogSeverityProcessor extends ContextualProcessor<String, String, String, String> {
+ @Override
+ public void process(final Record<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ final int colonIndex = record.value().indexOf(':');
+ final String severity = colonIndex > 0 ? record.value().substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Route logs based on severity
+ switch (severity) {
+ case "ERROR":
+ context().forward(record.withKey(ERROR_LOGS_TOPIC));
+ break;
+ case "WARN":
+ context().forward(record.withKey(WARN_LOGS_TOPIC));
+ break;
+ case "INFO":
+ // INFO logs are ignored
+ break;
+ default:
+ // Forward to an "unknown" topic for logs with unrecognized severities
+ context().forward(record.withKey(UNKNOWN_LOGS_TOPIC));
+ }
+ }
+ }
}</code></pre>
- <div class="admonition tip">
- <p><b>Tip</b></p>
- <p class="last">Even though we do not demonstrate it in this example, a stream processor can access any available state stores by
- calling <code class="docutils literal"><span class="pre">ProcessorContext#getStateStore()</span></code>.
- State stores are only available if they have been connected to the processor, or if they are global stores. While global stores do not need to be connected explicitly, they only allow for read-only access.
- There are two ways to connect state stores to a processor:
- <ul class="simple">
- <li>By passing the name of a store that has already been added via <code class="docutils literal"><span class="pre">Topology#addStateStore()</span></code> to the corresponding <code class="docutils literal"><span class="pre">KStream#process()</span></code> method call.</li>
- <li>Implementing <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the <code class="docutils literal"><span class="pre">ProcessorSupplier</span></code>
- passed to <code class="docutils literal"><span class="pre">KStream#process()</span></code>. In this case there is no need to call <code class="docutils literal"><span class="pre">StreamsBuilder#addStateStore()</span></code>
- beforehand, the store will be automatically added for you. You can also implement <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the
- <code class="docutils literal"><span class="pre">Value*</span></code> or <code class="docutils literal"><span class="pre">*WithKey</span></code> supplier variants, or <code class="docutils literal"><span class="pre">TransformerSupplier</span></code> or any of its variants.
- </li>
- </ul>
- </div>
- <p>Then we can leverage the <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code> processor in the DSL via <code class="docutils literal"><span class="pre">KStream#process</span></code>.</p>
- <pre class="line-numbers"><code class="language-java">KStream<String, GenericRecord> pageViews = ...;
-
-// Send an email notification when the view count of a page reaches one thousand.
-pageViews.groupByKey()
- .count()
- .filter((PageId pageId, Long viewCount) -> viewCount == 1000)
- // PopularPageEmailAlert is your custom processor that implements the
- // `Processor` interface, see further down below.
- .process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
- </div>
+ <h5 id="traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</h5>
+ <ul>
+ <li><strong>Idea:</strong> A radar monitors cars passing along a road stretch. A system counts the cars for each
+ day, maintaining a cumulative total for the current day in a state store. At the end of the day, the count
+ is
+ emitted and the state is cleared for the next day.</li>
+ <li><strong>Real-World Context:</strong> A car counting system can be useful for determining measures for
+ widening
+ or controlling traffic depending on the number of cars passing through the monitored stretch.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.common.serialization;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessorContext;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.state.Stores;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.format.DateTimeFormatter;
+
+public class TrafficRadarMonitoringCarCountExample {
+ private static final String DAILY_COUNT_STORE = "price-state-store";
+ private static final String DAILY_COUNT_TOPIC = "price-state-topic";
+ private static final String RADAR_COUNT_TOPIC = "car-radar-topic";
+
+ public static void countWithProcessValues(final StreamsBuilder builder) {
+ // Define a state store for tracking daily car counts
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(DAILY_COUNT_STORE),
+ Serdes.String(),
+ Serdes.Long()
+ )
+ );
+ final KStream<Void, String> radarStream = builder.stream(RADAR_COUNT_TOPIC);
+ // Apply the FixedKeyProcessor with the state store
+ radarStream.processValues(DailyCarCountProcessor::new, DAILY_COUNT_STORE)
+ .to(DAILY_COUNT_TOPIC);
+ }
+
+ private static class DailyCarCountProcessor implements FixedKeyProcessor<Void, String, String> {
+ private FixedKeyProcessorContext<Void, String> context;
+ private KeyValueStore<String, Long> stateStore;
+ private static final DateTimeFormatter DATE_FORMATTER =
+ DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(ZoneId.systemDefault());
+
+ @Override
+ public void init(final FixedKeyProcessorContext<Void, String> context) {
+ this.context = context;
+ stateStore = context.getStateStore(DAILY_COUNT_STORE);
+ }
+
+ @Override
+ public void process(final FixedKeyRecord<Void, String> record) {
+ if (record.value() == null) {
+ return; // Skip null events
+ }
+
+ // Derive the current day from the event timestamp
+ final long timestamp = System.currentTimeMillis(); // Use system time for simplicity
+ final String currentDay = DATE_FORMATTER.format(Instant.ofEpochMilli(timestamp));
+ // Retrieve the current count for the day
+ Long dailyCount = stateStore.get(currentDay);
+ if (dailyCount == null) {
+ dailyCount = 0L;
+ }
+ // Increment the count
+ dailyCount++;
+ stateStore.put(currentDay, dailyCount);
+
+ // Emit the current day's count
+ context.forward(record.withValue(String.format("Day: %s, Car Count: %s", currentDay, dailyCount)));
+ }
+ }
+}</code></pre>
+ <h5 id="replacing-slang-in-text-messages">Replacing Slang in Text Messages</h5>
+ <ul>
+ <li><strong>Idea:</strong> A messaging stream contains user-generated content, and you want to replace slang
+ words
+ with their formal equivalents (e.g., "u" becomes "you", "brb" becomes "be
+ right back"). The operation only modifies the message value and keeps the key intact.</li>
+ <li><strong>Real-World Context:</strong> In customer support chat systems, normalizing text by replacing slang
+ with
+ formal equivalents ensures that automated sentiment analysis tools work accurately and provide reliable
+ insights.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualFixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
+
+public class ReplacingSlangTextInMessagesExample {
+ private static final Map<String, String> SLANG_DICTIONARY = Map.of(
+ "u", "you",
+ "brb", "be right back",
+ "omg", "oh my god",
+ "btw", "by the way"
+ );
+ private static final String INPUT_MESSAGES_TOPIC = "input-messages-topic";
+ private static final String OUTPUT_MESSAGES_TOPIC = "output-messages-topic";
+
+ public static void replaceWithProcessValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.processValues(SlangReplacementProcessor::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ private static class SlangReplacementProcessor extends ContextualFixedKeyProcessor<String, String, String> {
+ @Override
+ public void process(final FixedKeyRecord<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+
+ // Replace slang words in the message
+ final String[] words = record.value().split("\\s+");
+ for (final String word : words) {
+ String replacedWord = SLANG_DICTIONARY.getOrDefault(word, word);
+ context().forward(record.withValue(replacedWord));
+ }
+ }
+ }
+}</code></pre>
+ <h4>Keynotes</h4>
+ <ul>
+ <li><strong>Type Safety and Flexibility:</strong> The process and processValues APIs utilize
+ <code>ProcessorContext</code> and <code>Record</code> or <code>FixedKeyRecord</code> objects for better type
+ safety and flexibility of custom processing logic.
+ </li>
+ <li><strong>Clear State and Logic Management:</strong> Implementations for <code>Processor</code> or
+ <code>FixedKeyProcessor</code> should manage state and logic clearly. Use <code>context().forward()</code>
+ for emitting records downstream.
+ </li>
+ <li><strong>Unified API:</strong> Consolidates multiple methods into a single, versatile API.</li>
+ <li><strong>Future-Proof:</strong> Ensures compatibility with the latest Kafka Streams releases.</li>
+ </ul>
+ </div>
+ <div class="section" id="transformers-removal-and-migration-to-processors">
+ <a class="headerlink" href="#transformers-removal-and-migration-to-processors" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id37">Transformers removal and migration to processors</a>
+ </h3>
+ </a>
+ <p>As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>,<br><code>transformValues</code>, <code>flatTransformValues</code>, and
+ <code>process</code> have been removed. These methods have been replaced with the<br>more versatile Processor API.
+ This guide provides detailed steps for migrating existing code to use the new Processor<br>API and explains the
+ benefits of the changes.</p>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ <li><code>KStream#process</code></li>
+ </ul>
+ <p>The Processor API now serves as a unified replacement for all these methods. It simplifies the API surface
+ while<br>maintaining support for both stateless and stateful operations.</p>
+ <h4>Migration Examples</h4>
+ <p>To migrate from the deprecated <code>transform</code>, <code>transformValues</code>, <code>flatTransform</code>, and
+ <code>flatTransformValues</code> methods to the<br>Processor API (PAPI) in Kafka Streams, let's resume the
+ previouss examples. The new <code>process</code> and <code>processValues</code><br>methods enable a more flexible
+ and reusable approach by requiring implementations of the <code>Processor</code>
+ or<br><code>FixedKeyProcessor</code> interfaces.</p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Migrating from</th>
+ <th>Migrating to</th>
+ <th>State Type</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program-removal">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>transform</code></td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity-removal">Categorizing Logs by Severity</a></td>
+ <td><code>flatTransform</code></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count-removal">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>transformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages-removal">Replacing Slang in Text Messages</a></td>
+ <td><code>flatTransformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
+ </tbody>
+ </table>
+ <h5 id="cumulative-discounts-for-a-loyalty-program-removal">Cumulative Discounts for a Loyalty Program</h5>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on code in PR #18314:
URL: https://github.com/apache/kafka/pull/18314#discussion_r1933062713
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
+ </li>
+ <li><code>ProcessorSupplier</code>: A processor supplier that can create one or more <code>Processor</code>
+ instances; and</li>
+ <li><code>FixedKeyProcessorSupplier</code>: A processor supplier that can create one or more
+ <code>FixedKeyProcessor</code> instances.
+ </li>
+ </ul>
+ <h4>Examples</h4>
+ <p>Follow the examples below to learn how to apply <code>process</code> and <code>processValues</code> to your
+ <code>KStream</code>.
+ </p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Operation</th>
+ <th>State Type</th>
+ </tr>
</thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
</tbody>
</table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <h5 id="cumulative-discounts-for-a-loyalty-program"">Cumulative Discounts for a Loyalty Program</h5>
+ <ul>
+ <li><strong>Idea:</strong> A stream of purchase events contains user IDs and transaction amounts. Use a state
+ store
+ to accumulate the total spending of each user. When their total crosses a threshold, apply a discount on
+ their next transaction and update their accumulated total.</li>
+ <li><strong>Real-World Context:</strong> In a retail loyalty program, tracking cumulative customer spending
+ enables
+ dynamic rewards, such as issuing a discount when a customer's total purchases exceed a predefined limit.
+ </li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+
+public class CumulativeDiscountsForALoyaltyProgramExample {
+ private static final double DISCOUNT_THRESHOLD = 100.0;
+ private static final String CUSTOMER_SPENDING_STORE = "customer-spending-store";
+ private static final String DISCOUNT_NOTIFICATION_MESSAGE =
+ "Discount applied! You have received a reward for your purchases.";
+ private static final String DISCOUNT_NOTIFICATIONS_TOPIC = "discount-notifications-topic";
+ private static final String PURCHASE_EVENTS_TOPIC = "purchase-events-topic";
+
+ public static void applyDiscountWithProcess(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ Serdes.String(),
+ Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Processor with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.process(CumulativeDiscountProcessor::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ private static class CumulativeDiscountProcessor implements Processor<String, Double, String, String> {
+ private KeyValueStore<String, Double> spendingStore;
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public void process(final Record<String, Double> record) {
+ if (record.value() == null) {
+ return; // Skip null purchase amounts
+ }
- @Override
- void process(Record<PageId, Long> record) {
- // Here you would format and send the alert email.
- //
- // In this specific example, you would be able to include
- // information about the page's ID and its view count
- }
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(record.key());
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += record.value();
+ spendingStore.put(record.key(), currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(record.key(), currentSpending - DISCOUNT_THRESHOLD);
+ // Send a discount notification
+ context.forward(record.withValue(DISCOUNT_NOTIFICATION_MESSAGE));
+ }
+ }
+ }
+}</code></pre>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li><strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g.,
+ INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to a dedicated topic too.</li>
+ <li><strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity
+ ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks,
+ and
+ INFO logs are stored for basic reporting purposes.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- void close() {
- // Any code for clean up would go here, for example tearing down the email client and anything
- // else you created in the init() method
- // This processor instance will not be used again after this call.
- }
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new)
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
+
+ private static class LogSeverityProcessor extends ContextualProcessor<String, String, String, String> {
+ @Override
+ public void process(final Record<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ final int colonIndex = record.value().indexOf(':');
+ final String severity = colonIndex > 0 ? record.value().substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Route logs based on severity
+ switch (severity) {
+ case "ERROR":
+ context().forward(record.withKey(ERROR_LOGS_TOPIC));
+ break;
+ case "WARN":
+ context().forward(record.withKey(WARN_LOGS_TOPIC));
+ break;
+ case "INFO":
+ // INFO logs are ignored
+ break;
+ default:
+ // Forward to an "unknown" topic for logs with unrecognized severities
+ context().forward(record.withKey(UNKNOWN_LOGS_TOPIC));
+ }
+ }
+ }
}</code></pre>
- <div class="admonition tip">
- <p><b>Tip</b></p>
- <p class="last">Even though we do not demonstrate it in this example, a stream processor can access any available state stores by
- calling <code class="docutils literal"><span class="pre">ProcessorContext#getStateStore()</span></code>.
- State stores are only available if they have been connected to the processor, or if they are global stores. While global stores do not need to be connected explicitly, they only allow for read-only access.
- There are two ways to connect state stores to a processor:
- <ul class="simple">
- <li>By passing the name of a store that has already been added via <code class="docutils literal"><span class="pre">Topology#addStateStore()</span></code> to the corresponding <code class="docutils literal"><span class="pre">KStream#process()</span></code> method call.</li>
- <li>Implementing <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the <code class="docutils literal"><span class="pre">ProcessorSupplier</span></code>
- passed to <code class="docutils literal"><span class="pre">KStream#process()</span></code>. In this case there is no need to call <code class="docutils literal"><span class="pre">StreamsBuilder#addStateStore()</span></code>
- beforehand, the store will be automatically added for you. You can also implement <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the
- <code class="docutils literal"><span class="pre">Value*</span></code> or <code class="docutils literal"><span class="pre">*WithKey</span></code> supplier variants, or <code class="docutils literal"><span class="pre">TransformerSupplier</span></code> or any of its variants.
- </li>
- </ul>
- </div>
- <p>Then we can leverage the <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code> processor in the DSL via <code class="docutils literal"><span class="pre">KStream#process</span></code>.</p>
- <pre class="line-numbers"><code class="language-java">KStream<String, GenericRecord> pageViews = ...;
-
-// Send an email notification when the view count of a page reaches one thousand.
-pageViews.groupByKey()
- .count()
- .filter((PageId pageId, Long viewCount) -> viewCount == 1000)
- // PopularPageEmailAlert is your custom processor that implements the
- // `Processor` interface, see further down below.
- .process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
- </div>
+ <h5 id="traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</h5>
+ <ul>
+ <li><strong>Idea:</strong> A radar monitors cars passing along a road stretch. A system counts the cars for each
+ day, maintaining a cumulative total for the current day in a state store. At the end of the day, the count
+ is
+ emitted and the state is cleared for the next day.</li>
+ <li><strong>Real-World Context:</strong> A car counting system can be useful for determining measures for
+ widening
+ or controlling traffic depending on the number of cars passing through the monitored stretch.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.common.serialization;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessorContext;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.state.Stores;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.format.DateTimeFormatter;
+
+public class TrafficRadarMonitoringCarCountExample {
+ private static final String DAILY_COUNT_STORE = "price-state-store";
+ private static final String DAILY_COUNT_TOPIC = "price-state-topic";
+ private static final String RADAR_COUNT_TOPIC = "car-radar-topic";
+
+ public static void countWithProcessValues(final StreamsBuilder builder) {
+ // Define a state store for tracking daily car counts
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(DAILY_COUNT_STORE),
+ Serdes.String(),
+ Serdes.Long()
+ )
+ );
+ final KStream<Void, String> radarStream = builder.stream(RADAR_COUNT_TOPIC);
+ // Apply the FixedKeyProcessor with the state store
+ radarStream.processValues(DailyCarCountProcessor::new, DAILY_COUNT_STORE)
+ .to(DAILY_COUNT_TOPIC);
+ }
+
+ private static class DailyCarCountProcessor implements FixedKeyProcessor<Void, String, String> {
+ private FixedKeyProcessorContext<Void, String> context;
+ private KeyValueStore<String, Long> stateStore;
+ private static final DateTimeFormatter DATE_FORMATTER =
+ DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(ZoneId.systemDefault());
+
+ @Override
+ public void init(final FixedKeyProcessorContext<Void, String> context) {
+ this.context = context;
+ stateStore = context.getStateStore(DAILY_COUNT_STORE);
+ }
+
+ @Override
+ public void process(final FixedKeyRecord<Void, String> record) {
+ if (record.value() == null) {
+ return; // Skip null events
+ }
+
+ // Derive the current day from the event timestamp
+ final long timestamp = System.currentTimeMillis(); // Use system time for simplicity
+ final String currentDay = DATE_FORMATTER.format(Instant.ofEpochMilli(timestamp));
+ // Retrieve the current count for the day
+ Long dailyCount = stateStore.get(currentDay);
+ if (dailyCount == null) {
+ dailyCount = 0L;
+ }
+ // Increment the count
+ dailyCount++;
+ stateStore.put(currentDay, dailyCount);
+
+ // Emit the current day's count
+ context.forward(record.withValue(String.format("Day: %s, Car Count: %s", currentDay, dailyCount)));
+ }
+ }
+}</code></pre>
+ <h5 id="replacing-slang-in-text-messages">Replacing Slang in Text Messages</h5>
+ <ul>
+ <li><strong>Idea:</strong> A messaging stream contains user-generated content, and you want to replace slang
+ words
+ with their formal equivalents (e.g., "u" becomes "you", "brb" becomes "be
+ right back"). The operation only modifies the message value and keeps the key intact.</li>
+ <li><strong>Real-World Context:</strong> In customer support chat systems, normalizing text by replacing slang
+ with
+ formal equivalents ensures that automated sentiment analysis tools work accurately and provide reliable
+ insights.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualFixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
+
+public class ReplacingSlangTextInMessagesExample {
+ private static final Map<String, String> SLANG_DICTIONARY = Map.of(
+ "u", "you",
+ "brb", "be right back",
+ "omg", "oh my god",
+ "btw", "by the way"
+ );
+ private static final String INPUT_MESSAGES_TOPIC = "input-messages-topic";
+ private static final String OUTPUT_MESSAGES_TOPIC = "output-messages-topic";
+
+ public static void replaceWithProcessValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.processValues(SlangReplacementProcessor::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ private static class SlangReplacementProcessor extends ContextualFixedKeyProcessor<String, String, String> {
+ @Override
+ public void process(final FixedKeyRecord<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+
+ // Replace slang words in the message
+ final String[] words = record.value().split("\\s+");
+ for (final String word : words) {
+ String replacedWord = SLANG_DICTIONARY.getOrDefault(word, word);
+ context().forward(record.withValue(replacedWord));
+ }
+ }
+ }
+}</code></pre>
+ <h4>Keynotes</h4>
+ <ul>
+ <li><strong>Type Safety and Flexibility:</strong> The process and processValues APIs utilize
+ <code>ProcessorContext</code> and <code>Record</code> or <code>FixedKeyRecord</code> objects for better type
+ safety and flexibility of custom processing logic.
+ </li>
+ <li><strong>Clear State and Logic Management:</strong> Implementations for <code>Processor</code> or
+ <code>FixedKeyProcessor</code> should manage state and logic clearly. Use <code>context().forward()</code>
+ for emitting records downstream.
+ </li>
+ <li><strong>Unified API:</strong> Consolidates multiple methods into a single, versatile API.</li>
+ <li><strong>Future-Proof:</strong> Ensures compatibility with the latest Kafka Streams releases.</li>
+ </ul>
+ </div>
+ <div class="section" id="transformers-removal-and-migration-to-processors">
+ <a class="headerlink" href="#transformers-removal-and-migration-to-processors" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id37">Transformers removal and migration to processors</a>
+ </h3>
+ </a>
+ <p>As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>,<br><code>transformValues</code>, <code>flatTransformValues</code>, and
Review Comment:
Replaced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on PR #18314:
URL: https://github.com/apache/kafka/pull/18314#issuecomment-2564397291
@mjsax, can I format the code? Some blocks are not indented correctly. I understand it might pollute the diff in code changes, so if I may, I would do it as a final commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on code in PR #18314:
URL: https://github.com/apache/kafka/pull/18314#discussion_r1933125223
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
Review Comment:
Commented above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #18314:
URL: https://github.com/apache/kafka/pull/18314#discussion_r1931385336
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
Review Comment:
Seems you did not change lot here, but the diff is quite big. It's ok to leave as-is, but in general, try to limit change to what is necessary, avoiding unrelated reformatting to keep the diff smaller, and thus the PR easier to review.
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
Review Comment:
```and provides default no-op implementation of
<code>Processor#close</code>.
```
This was correct for the old PAPI. In the new PAPI, the `api.Processor` interface has already an empty default impl for `close()` -- Thus, `api.ContextualProcessor` does not (need to) provide an empty default impl any longer.
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
Review Comment:
Same as above.
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
+ </li>
+ <li><code>ProcessorSupplier</code>: A processor supplier that can create one or more <code>Processor</code>
+ instances; and</li>
+ <li><code>FixedKeyProcessorSupplier</code>: A processor supplier that can create one or more
+ <code>FixedKeyProcessor</code> instances.
+ </li>
+ </ul>
+ <h4>Examples</h4>
+ <p>Follow the examples below to learn how to apply <code>process</code> and <code>processValues</code> to your
+ <code>KStream</code>.
+ </p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Operation</th>
+ <th>State Type</th>
+ </tr>
</thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
Review Comment:
Stateless first?
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
+ </li>
+ <li><code>ProcessorSupplier</code>: A processor supplier that can create one or more <code>Processor</code>
+ instances; and</li>
+ <li><code>FixedKeyProcessorSupplier</code>: A processor supplier that can create one or more
+ <code>FixedKeyProcessor</code> instances.
+ </li>
+ </ul>
+ <h4>Examples</h4>
+ <p>Follow the examples below to learn how to apply <code>process</code> and <code>processValues</code> to your
+ <code>KStream</code>.
+ </p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Operation</th>
+ <th>State Type</th>
+ </tr>
</thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a>
Review Comment:
Should we start with the simpler stateless example?
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
+ </li>
+ <li><code>ProcessorSupplier</code>: A processor supplier that can create one or more <code>Processor</code>
+ instances; and</li>
+ <li><code>FixedKeyProcessorSupplier</code>: A processor supplier that can create one or more
+ <code>FixedKeyProcessor</code> instances.
+ </li>
+ </ul>
+ <h4>Examples</h4>
+ <p>Follow the examples below to learn how to apply <code>process</code> and <code>processValues</code> to your
+ <code>KStream</code>.
+ </p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Operation</th>
+ <th>State Type</th>
+ </tr>
</thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
</tbody>
</table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <h5 id="cumulative-discounts-for-a-loyalty-program"">Cumulative Discounts for a Loyalty Program</h5>
+ <ul>
+ <li><strong>Idea:</strong> A stream of purchase events contains user IDs and transaction amounts. Use a state
+ store
+ to accumulate the total spending of each user. When their total crosses a threshold, apply a discount on
+ their next transaction and update their accumulated total.</li>
+ <li><strong>Real-World Context:</strong> In a retail loyalty program, tracking cumulative customer spending
+ enables
+ dynamic rewards, such as issuing a discount when a customer's total purchases exceed a predefined limit.
+ </li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
Review Comment:
Should we omit imports to keep the example more concise?
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
+ </li>
+ <li><code>ProcessorSupplier</code>: A processor supplier that can create one or more <code>Processor</code>
+ instances; and</li>
+ <li><code>FixedKeyProcessorSupplier</code>: A processor supplier that can create one or more
+ <code>FixedKeyProcessor</code> instances.
+ </li>
+ </ul>
+ <h4>Examples</h4>
+ <p>Follow the examples below to learn how to apply <code>process</code> and <code>processValues</code> to your
+ <code>KStream</code>.
+ </p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Operation</th>
+ <th>State Type</th>
+ </tr>
</thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
</tbody>
</table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <h5 id="cumulative-discounts-for-a-loyalty-program"">Cumulative Discounts for a Loyalty Program</h5>
Review Comment:
```suggestion
<h5 id="cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</h5>
```
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
+ </li>
+ <li><code>ProcessorSupplier</code>: A processor supplier that can create one or more <code>Processor</code>
+ instances; and</li>
+ <li><code>FixedKeyProcessorSupplier</code>: A processor supplier that can create one or more
+ <code>FixedKeyProcessor</code> instances.
+ </li>
+ </ul>
+ <h4>Examples</h4>
+ <p>Follow the examples below to learn how to apply <code>process</code> and <code>processValues</code> to your
+ <code>KStream</code>.
+ </p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Operation</th>
+ <th>State Type</th>
+ </tr>
</thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
</tbody>
</table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <h5 id="cumulative-discounts-for-a-loyalty-program"">Cumulative Discounts for a Loyalty Program</h5>
+ <ul>
+ <li><strong>Idea:</strong> A stream of purchase events contains user IDs and transaction amounts. Use a state
+ store
+ to accumulate the total spending of each user. When their total crosses a threshold, apply a discount on
+ their next transaction and update their accumulated total.</li>
+ <li><strong>Real-World Context:</strong> In a retail loyalty program, tracking cumulative customer spending
+ enables
+ dynamic rewards, such as issuing a discount when a customer's total purchases exceed a predefined limit.
+ </li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+
+public class CumulativeDiscountsForALoyaltyProgramExample {
+ private static final double DISCOUNT_THRESHOLD = 100.0;
+ private static final String CUSTOMER_SPENDING_STORE = "customer-spending-store";
+ private static final String DISCOUNT_NOTIFICATION_MESSAGE =
+ "Discount applied! You have received a reward for your purchases.";
+ private static final String DISCOUNT_NOTIFICATIONS_TOPIC = "discount-notifications-topic";
+ private static final String PURCHASE_EVENTS_TOPIC = "purchase-events-topic";
+
+ public static void applyDiscountWithProcess(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ Serdes.String(),
+ Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Processor with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.process(CumulativeDiscountProcessor::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ private static class CumulativeDiscountProcessor implements Processor<String, Double, String, String> {
+ private KeyValueStore<String, Double> spendingStore;
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public void process(final Record<String, Double> record) {
+ if (record.value() == null) {
+ return; // Skip null purchase amounts
+ }
- @Override
- void process(Record<PageId, Long> record) {
- // Here you would format and send the alert email.
- //
- // In this specific example, you would be able to include
- // information about the page's ID and its view count
- }
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(record.key());
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += record.value();
+ spendingStore.put(record.key(), currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(record.key(), currentSpending - DISCOUNT_THRESHOLD);
+ // Send a discount notification
+ context.forward(record.withValue(DISCOUNT_NOTIFICATION_MESSAGE));
+ }
+ }
+ }
+}</code></pre>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li><strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g.,
+ INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to a dedicated topic too.</li>
+ <li><strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity
+ ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks,
+ and
+ INFO logs are stored for basic reporting purposes.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- void close() {
- // Any code for clean up would go here, for example tearing down the email client and anything
- // else you created in the init() method
- // This processor instance will not be used again after this call.
- }
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new)
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
+
+ private static class LogSeverityProcessor extends ContextualProcessor<String, String, String, String> {
+ @Override
+ public void process(final Record<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ final int colonIndex = record.value().indexOf(':');
+ final String severity = colonIndex > 0 ? record.value().substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Route logs based on severity
+ switch (severity) {
+ case "ERROR":
+ context().forward(record.withKey(ERROR_LOGS_TOPIC));
+ break;
+ case "WARN":
+ context().forward(record.withKey(WARN_LOGS_TOPIC));
+ break;
+ case "INFO":
+ // INFO logs are ignored
+ break;
+ default:
+ // Forward to an "unknown" topic for logs with unrecognized severities
+ context().forward(record.withKey(UNKNOWN_LOGS_TOPIC));
+ }
+ }
+ }
}</code></pre>
- <div class="admonition tip">
- <p><b>Tip</b></p>
- <p class="last">Even though we do not demonstrate it in this example, a stream processor can access any available state stores by
- calling <code class="docutils literal"><span class="pre">ProcessorContext#getStateStore()</span></code>.
- State stores are only available if they have been connected to the processor, or if they are global stores. While global stores do not need to be connected explicitly, they only allow for read-only access.
- There are two ways to connect state stores to a processor:
- <ul class="simple">
- <li>By passing the name of a store that has already been added via <code class="docutils literal"><span class="pre">Topology#addStateStore()</span></code> to the corresponding <code class="docutils literal"><span class="pre">KStream#process()</span></code> method call.</li>
- <li>Implementing <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the <code class="docutils literal"><span class="pre">ProcessorSupplier</span></code>
- passed to <code class="docutils literal"><span class="pre">KStream#process()</span></code>. In this case there is no need to call <code class="docutils literal"><span class="pre">StreamsBuilder#addStateStore()</span></code>
- beforehand, the store will be automatically added for you. You can also implement <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the
- <code class="docutils literal"><span class="pre">Value*</span></code> or <code class="docutils literal"><span class="pre">*WithKey</span></code> supplier variants, or <code class="docutils literal"><span class="pre">TransformerSupplier</span></code> or any of its variants.
- </li>
- </ul>
- </div>
- <p>Then we can leverage the <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code> processor in the DSL via <code class="docutils literal"><span class="pre">KStream#process</span></code>.</p>
- <pre class="line-numbers"><code class="language-java">KStream<String, GenericRecord> pageViews = ...;
-
-// Send an email notification when the view count of a page reaches one thousand.
-pageViews.groupByKey()
- .count()
- .filter((PageId pageId, Long viewCount) -> viewCount == 1000)
- // PopularPageEmailAlert is your custom processor that implements the
- // `Processor` interface, see further down below.
- .process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
- </div>
+ <h5 id="traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</h5>
+ <ul>
+ <li><strong>Idea:</strong> A radar monitors cars passing along a road stretch. A system counts the cars for each
+ day, maintaining a cumulative total for the current day in a state store. At the end of the day, the count
+ is
+ emitted and the state is cleared for the next day.</li>
+ <li><strong>Real-World Context:</strong> A car counting system can be useful for determining measures for
+ widening
+ or controlling traffic depending on the number of cars passing through the monitored stretch.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.common.serialization;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessorContext;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.state.Stores;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.format.DateTimeFormatter;
+
+public class TrafficRadarMonitoringCarCountExample {
+ private static final String DAILY_COUNT_STORE = "price-state-store";
+ private static final String DAILY_COUNT_TOPIC = "price-state-topic";
+ private static final String RADAR_COUNT_TOPIC = "car-radar-topic";
+
+ public static void countWithProcessValues(final StreamsBuilder builder) {
+ // Define a state store for tracking daily car counts
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(DAILY_COUNT_STORE),
+ Serdes.String(),
+ Serdes.Long()
+ )
+ );
+ final KStream<Void, String> radarStream = builder.stream(RADAR_COUNT_TOPIC);
+ // Apply the FixedKeyProcessor with the state store
+ radarStream.processValues(DailyCarCountProcessor::new, DAILY_COUNT_STORE)
+ .to(DAILY_COUNT_TOPIC);
+ }
+
+ private static class DailyCarCountProcessor implements FixedKeyProcessor<Void, String, String> {
+ private FixedKeyProcessorContext<Void, String> context;
+ private KeyValueStore<String, Long> stateStore;
+ private static final DateTimeFormatter DATE_FORMATTER =
+ DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(ZoneId.systemDefault());
+
+ @Override
+ public void init(final FixedKeyProcessorContext<Void, String> context) {
+ this.context = context;
+ stateStore = context.getStateStore(DAILY_COUNT_STORE);
+ }
+
+ @Override
+ public void process(final FixedKeyRecord<Void, String> record) {
+ if (record.value() == null) {
+ return; // Skip null events
+ }
+
+ // Derive the current day from the event timestamp
+ final long timestamp = System.currentTimeMillis(); // Use system time for simplicity
+ final String currentDay = DATE_FORMATTER.format(Instant.ofEpochMilli(timestamp));
+ // Retrieve the current count for the day
+ Long dailyCount = stateStore.get(currentDay);
+ if (dailyCount == null) {
+ dailyCount = 0L;
+ }
+ // Increment the count
+ dailyCount++;
+ stateStore.put(currentDay, dailyCount);
+
+ // Emit the current day's count
+ context.forward(record.withValue(String.format("Day: %s, Car Count: %s", currentDay, dailyCount)));
+ }
+ }
+}</code></pre>
+ <h5 id="replacing-slang-in-text-messages">Replacing Slang in Text Messages</h5>
+ <ul>
+ <li><strong>Idea:</strong> A messaging stream contains user-generated content, and you want to replace slang
+ words
+ with their formal equivalents (e.g., "u" becomes "you", "brb" becomes "be
+ right back"). The operation only modifies the message value and keeps the key intact.</li>
+ <li><strong>Real-World Context:</strong> In customer support chat systems, normalizing text by replacing slang
+ with
+ formal equivalents ensures that automated sentiment analysis tools work accurately and provide reliable
+ insights.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualFixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
+
+public class ReplacingSlangTextInMessagesExample {
+ private static final Map<String, String> SLANG_DICTIONARY = Map.of(
+ "u", "you",
+ "brb", "be right back",
+ "omg", "oh my god",
+ "btw", "by the way"
+ );
+ private static final String INPUT_MESSAGES_TOPIC = "input-messages-topic";
+ private static final String OUTPUT_MESSAGES_TOPIC = "output-messages-topic";
+
+ public static void replaceWithProcessValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.processValues(SlangReplacementProcessor::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ private static class SlangReplacementProcessor extends ContextualFixedKeyProcessor<String, String, String> {
+ @Override
+ public void process(final FixedKeyRecord<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+
+ // Replace slang words in the message
+ final String[] words = record.value().split("\\s+");
+ for (final String word : words) {
+ String replacedWord = SLANG_DICTIONARY.getOrDefault(word, word);
+ context().forward(record.withValue(replacedWord));
+ }
+ }
+ }
+}</code></pre>
+ <h4>Keynotes</h4>
+ <ul>
+ <li><strong>Type Safety and Flexibility:</strong> The process and processValues APIs utilize
+ <code>ProcessorContext</code> and <code>Record</code> or <code>FixedKeyRecord</code> objects for better type
+ safety and flexibility of custom processing logic.
+ </li>
+ <li><strong>Clear State and Logic Management:</strong> Implementations for <code>Processor</code> or
+ <code>FixedKeyProcessor</code> should manage state and logic clearly. Use <code>context().forward()</code>
+ for emitting records downstream.
+ </li>
+ <li><strong>Unified API:</strong> Consolidates multiple methods into a single, versatile API.</li>
+ <li><strong>Future-Proof:</strong> Ensures compatibility with the latest Kafka Streams releases.</li>
+ </ul>
+ </div>
+ <div class="section" id="transformers-removal-and-migration-to-processors">
+ <a class="headerlink" href="#transformers-removal-and-migration-to-processors" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id37">Transformers removal and migration to processors</a>
+ </h3>
+ </a>
+ <p>As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>,<br><code>transformValues</code>, <code>flatTransformValues</code>, and
+ <code>process</code> have been removed. These methods have been replaced with the<br>more versatile Processor API.
+ This guide provides detailed steps for migrating existing code to use the new Processor<br>API and explains the
+ benefits of the changes.</p>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ <li><code>KStream#process</code></li>
+ </ul>
+ <p>The Processor API now serves as a unified replacement for all these methods. It simplifies the API surface
+ while<br>maintaining support for both stateless and stateful operations.</p>
+ <h4>Migration Examples</h4>
+ <p>To migrate from the deprecated <code>transform</code>, <code>transformValues</code>, <code>flatTransform</code>, and
+ <code>flatTransformValues</code> methods to the<br>Processor API (PAPI) in Kafka Streams, let's resume the
+ previouss examples. The new <code>process</code> and <code>processValues</code><br>methods enable a more flexible
+ and reusable approach by requiring implementations of the <code>Processor</code>
+ or<br><code>FixedKeyProcessor</code> interfaces.</p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Migrating from</th>
+ <th>Migrating to</th>
+ <th>State Type</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program-removal">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>transform</code></td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity-removal">Categorizing Logs by Severity</a></td>
+ <td><code>flatTransform</code></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count-removal">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>transformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
Review Comment:
same question
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
+ </li>
+ <li><code>ProcessorSupplier</code>: A processor supplier that can create one or more <code>Processor</code>
+ instances; and</li>
+ <li><code>FixedKeyProcessorSupplier</code>: A processor supplier that can create one or more
+ <code>FixedKeyProcessor</code> instances.
+ </li>
+ </ul>
+ <h4>Examples</h4>
+ <p>Follow the examples below to learn how to apply <code>process</code> and <code>processValues</code> to your
+ <code>KStream</code>.
+ </p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Operation</th>
+ <th>State Type</th>
+ </tr>
</thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
</tbody>
</table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <h5 id="cumulative-discounts-for-a-loyalty-program"">Cumulative Discounts for a Loyalty Program</h5>
+ <ul>
+ <li><strong>Idea:</strong> A stream of purchase events contains user IDs and transaction amounts. Use a state
+ store
+ to accumulate the total spending of each user. When their total crosses a threshold, apply a discount on
+ their next transaction and update their accumulated total.</li>
+ <li><strong>Real-World Context:</strong> In a retail loyalty program, tracking cumulative customer spending
+ enables
+ dynamic rewards, such as issuing a discount when a customer's total purchases exceed a predefined limit.
+ </li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+
+public class CumulativeDiscountsForALoyaltyProgramExample {
+ private static final double DISCOUNT_THRESHOLD = 100.0;
+ private static final String CUSTOMER_SPENDING_STORE = "customer-spending-store";
+ private static final String DISCOUNT_NOTIFICATION_MESSAGE =
+ "Discount applied! You have received a reward for your purchases.";
+ private static final String DISCOUNT_NOTIFICATIONS_TOPIC = "discount-notifications-topic";
+ private static final String PURCHASE_EVENTS_TOPIC = "purchase-events-topic";
+
+ public static void applyDiscountWithProcess(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ Serdes.String(),
+ Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Processor with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.process(CumulativeDiscountProcessor::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ private static class CumulativeDiscountProcessor implements Processor<String, Double, String, String> {
+ private KeyValueStore<String, Double> spendingStore;
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public void process(final Record<String, Double> record) {
+ if (record.value() == null) {
+ return; // Skip null purchase amounts
+ }
- @Override
- void process(Record<PageId, Long> record) {
- // Here you would format and send the alert email.
- //
- // In this specific example, you would be able to include
- // information about the page's ID and its view count
- }
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(record.key());
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += record.value();
+ spendingStore.put(record.key(), currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(record.key(), currentSpending - DISCOUNT_THRESHOLD);
+ // Send a discount notification
+ context.forward(record.withValue(DISCOUNT_NOTIFICATION_MESSAGE));
+ }
+ }
+ }
+}</code></pre>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li><strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g.,
+ INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to a dedicated topic too.</li>
+ <li><strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity
+ ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks,
+ and
+ INFO logs are stored for basic reporting purposes.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- void close() {
- // Any code for clean up would go here, for example tearing down the email client and anything
- // else you created in the init() method
- // This processor instance will not be used again after this call.
- }
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new)
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
+
+ private static class LogSeverityProcessor extends ContextualProcessor<String, String, String, String> {
+ @Override
+ public void process(final Record<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ final int colonIndex = record.value().indexOf(':');
+ final String severity = colonIndex > 0 ? record.value().substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Route logs based on severity
+ switch (severity) {
+ case "ERROR":
+ context().forward(record.withKey(ERROR_LOGS_TOPIC));
+ break;
+ case "WARN":
+ context().forward(record.withKey(WARN_LOGS_TOPIC));
+ break;
+ case "INFO":
+ // INFO logs are ignored
+ break;
+ default:
+ // Forward to an "unknown" topic for logs with unrecognized severities
+ context().forward(record.withKey(UNKNOWN_LOGS_TOPIC));
+ }
+ }
+ }
}</code></pre>
- <div class="admonition tip">
- <p><b>Tip</b></p>
- <p class="last">Even though we do not demonstrate it in this example, a stream processor can access any available state stores by
- calling <code class="docutils literal"><span class="pre">ProcessorContext#getStateStore()</span></code>.
- State stores are only available if they have been connected to the processor, or if they are global stores. While global stores do not need to be connected explicitly, they only allow for read-only access.
- There are two ways to connect state stores to a processor:
- <ul class="simple">
- <li>By passing the name of a store that has already been added via <code class="docutils literal"><span class="pre">Topology#addStateStore()</span></code> to the corresponding <code class="docutils literal"><span class="pre">KStream#process()</span></code> method call.</li>
- <li>Implementing <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the <code class="docutils literal"><span class="pre">ProcessorSupplier</span></code>
- passed to <code class="docutils literal"><span class="pre">KStream#process()</span></code>. In this case there is no need to call <code class="docutils literal"><span class="pre">StreamsBuilder#addStateStore()</span></code>
- beforehand, the store will be automatically added for you. You can also implement <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the
- <code class="docutils literal"><span class="pre">Value*</span></code> or <code class="docutils literal"><span class="pre">*WithKey</span></code> supplier variants, or <code class="docutils literal"><span class="pre">TransformerSupplier</span></code> or any of its variants.
- </li>
- </ul>
- </div>
- <p>Then we can leverage the <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code> processor in the DSL via <code class="docutils literal"><span class="pre">KStream#process</span></code>.</p>
- <pre class="line-numbers"><code class="language-java">KStream<String, GenericRecord> pageViews = ...;
-
-// Send an email notification when the view count of a page reaches one thousand.
-pageViews.groupByKey()
- .count()
- .filter((PageId pageId, Long viewCount) -> viewCount == 1000)
- // PopularPageEmailAlert is your custom processor that implements the
- // `Processor` interface, see further down below.
- .process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
- </div>
+ <h5 id="traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</h5>
+ <ul>
+ <li><strong>Idea:</strong> A radar monitors cars passing along a road stretch. A system counts the cars for each
+ day, maintaining a cumulative total for the current day in a state store. At the end of the day, the count
+ is
+ emitted and the state is cleared for the next day.</li>
+ <li><strong>Real-World Context:</strong> A car counting system can be useful for determining measures for
+ widening
+ or controlling traffic depending on the number of cars passing through the monitored stretch.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.common.serialization;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessorContext;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.state.Stores;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.format.DateTimeFormatter;
+
+public class TrafficRadarMonitoringCarCountExample {
+ private static final String DAILY_COUNT_STORE = "price-state-store";
+ private static final String DAILY_COUNT_TOPIC = "price-state-topic";
+ private static final String RADAR_COUNT_TOPIC = "car-radar-topic";
+
+ public static void countWithProcessValues(final StreamsBuilder builder) {
+ // Define a state store for tracking daily car counts
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(DAILY_COUNT_STORE),
+ Serdes.String(),
+ Serdes.Long()
+ )
+ );
+ final KStream<Void, String> radarStream = builder.stream(RADAR_COUNT_TOPIC);
+ // Apply the FixedKeyProcessor with the state store
+ radarStream.processValues(DailyCarCountProcessor::new, DAILY_COUNT_STORE)
+ .to(DAILY_COUNT_TOPIC);
+ }
+
+ private static class DailyCarCountProcessor implements FixedKeyProcessor<Void, String, String> {
+ private FixedKeyProcessorContext<Void, String> context;
+ private KeyValueStore<String, Long> stateStore;
+ private static final DateTimeFormatter DATE_FORMATTER =
+ DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(ZoneId.systemDefault());
+
+ @Override
+ public void init(final FixedKeyProcessorContext<Void, String> context) {
+ this.context = context;
+ stateStore = context.getStateStore(DAILY_COUNT_STORE);
+ }
+
+ @Override
+ public void process(final FixedKeyRecord<Void, String> record) {
+ if (record.value() == null) {
+ return; // Skip null events
+ }
+
+ // Derive the current day from the event timestamp
+ final long timestamp = System.currentTimeMillis(); // Use system time for simplicity
+ final String currentDay = DATE_FORMATTER.format(Instant.ofEpochMilli(timestamp));
+ // Retrieve the current count for the day
+ Long dailyCount = stateStore.get(currentDay);
+ if (dailyCount == null) {
+ dailyCount = 0L;
+ }
+ // Increment the count
+ dailyCount++;
+ stateStore.put(currentDay, dailyCount);
+
+ // Emit the current day's count
+ context.forward(record.withValue(String.format("Day: %s, Car Count: %s", currentDay, dailyCount)));
+ }
+ }
+}</code></pre>
+ <h5 id="replacing-slang-in-text-messages">Replacing Slang in Text Messages</h5>
+ <ul>
+ <li><strong>Idea:</strong> A messaging stream contains user-generated content, and you want to replace slang
+ words
+ with their formal equivalents (e.g., "u" becomes "you", "brb" becomes "be
+ right back"). The operation only modifies the message value and keeps the key intact.</li>
+ <li><strong>Real-World Context:</strong> In customer support chat systems, normalizing text by replacing slang
+ with
+ formal equivalents ensures that automated sentiment analysis tools work accurately and provide reliable
+ insights.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualFixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
+
+public class ReplacingSlangTextInMessagesExample {
+ private static final Map<String, String> SLANG_DICTIONARY = Map.of(
+ "u", "you",
+ "brb", "be right back",
+ "omg", "oh my god",
+ "btw", "by the way"
+ );
+ private static final String INPUT_MESSAGES_TOPIC = "input-messages-topic";
+ private static final String OUTPUT_MESSAGES_TOPIC = "output-messages-topic";
+
+ public static void replaceWithProcessValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.processValues(SlangReplacementProcessor::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ private static class SlangReplacementProcessor extends ContextualFixedKeyProcessor<String, String, String> {
+ @Override
+ public void process(final FixedKeyRecord<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+
+ // Replace slang words in the message
+ final String[] words = record.value().split("\\s+");
+ for (final String word : words) {
+ String replacedWord = SLANG_DICTIONARY.getOrDefault(word, word);
+ context().forward(record.withValue(replacedWord));
+ }
+ }
+ }
+}</code></pre>
+ <h4>Keynotes</h4>
+ <ul>
+ <li><strong>Type Safety and Flexibility:</strong> The process and processValues APIs utilize
+ <code>ProcessorContext</code> and <code>Record</code> or <code>FixedKeyRecord</code> objects for better type
+ safety and flexibility of custom processing logic.
+ </li>
+ <li><strong>Clear State and Logic Management:</strong> Implementations for <code>Processor</code> or
+ <code>FixedKeyProcessor</code> should manage state and logic clearly. Use <code>context().forward()</code>
+ for emitting records downstream.
+ </li>
+ <li><strong>Unified API:</strong> Consolidates multiple methods into a single, versatile API.</li>
+ <li><strong>Future-Proof:</strong> Ensures compatibility with the latest Kafka Streams releases.</li>
+ </ul>
+ </div>
+ <div class="section" id="transformers-removal-and-migration-to-processors">
+ <a class="headerlink" href="#transformers-removal-and-migration-to-processors" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id37">Transformers removal and migration to processors</a>
+ </h3>
+ </a>
+ <p>As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>,<br><code>transformValues</code>, <code>flatTransformValues</code>, and
Review Comment:
```suggestion
<code>flatTransform</code>, <code>transformValues</code>, <code>flatTransformValues</code>, and
```
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
+ </li>
+ <li><code>ProcessorSupplier</code>: A processor supplier that can create one or more <code>Processor</code>
+ instances; and</li>
+ <li><code>FixedKeyProcessorSupplier</code>: A processor supplier that can create one or more
+ <code>FixedKeyProcessor</code> instances.
+ </li>
+ </ul>
+ <h4>Examples</h4>
+ <p>Follow the examples below to learn how to apply <code>process</code> and <code>processValues</code> to your
+ <code>KStream</code>.
+ </p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Operation</th>
+ <th>State Type</th>
+ </tr>
</thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
</tbody>
</table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <h5 id="cumulative-discounts-for-a-loyalty-program"">Cumulative Discounts for a Loyalty Program</h5>
+ <ul>
+ <li><strong>Idea:</strong> A stream of purchase events contains user IDs and transaction amounts. Use a state
+ store
+ to accumulate the total spending of each user. When their total crosses a threshold, apply a discount on
+ their next transaction and update their accumulated total.</li>
+ <li><strong>Real-World Context:</strong> In a retail loyalty program, tracking cumulative customer spending
+ enables
+ dynamic rewards, such as issuing a discount when a customer's total purchases exceed a predefined limit.
+ </li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+
+public class CumulativeDiscountsForALoyaltyProgramExample {
+ private static final double DISCOUNT_THRESHOLD = 100.0;
+ private static final String CUSTOMER_SPENDING_STORE = "customer-spending-store";
+ private static final String DISCOUNT_NOTIFICATION_MESSAGE =
+ "Discount applied! You have received a reward for your purchases.";
+ private static final String DISCOUNT_NOTIFICATIONS_TOPIC = "discount-notifications-topic";
+ private static final String PURCHASE_EVENTS_TOPIC = "purchase-events-topic";
+
+ public static void applyDiscountWithProcess(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ Serdes.String(),
+ Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Processor with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.process(CumulativeDiscountProcessor::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ private static class CumulativeDiscountProcessor implements Processor<String, Double, String, String> {
+ private KeyValueStore<String, Double> spendingStore;
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public void process(final Record<String, Double> record) {
+ if (record.value() == null) {
+ return; // Skip null purchase amounts
+ }
- @Override
- void process(Record<PageId, Long> record) {
- // Here you would format and send the alert email.
- //
- // In this specific example, you would be able to include
- // information about the page's ID and its view count
- }
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(record.key());
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += record.value();
+ spendingStore.put(record.key(), currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(record.key(), currentSpending - DISCOUNT_THRESHOLD);
+ // Send a discount notification
+ context.forward(record.withValue(DISCOUNT_NOTIFICATION_MESSAGE));
+ }
+ }
+ }
+}</code></pre>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li><strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g.,
+ INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to a dedicated topic too.</li>
+ <li><strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity
+ ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks,
+ and
+ INFO logs are stored for basic reporting purposes.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- void close() {
- // Any code for clean up would go here, for example tearing down the email client and anything
- // else you created in the init() method
- // This processor instance will not be used again after this call.
- }
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new)
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
+
+ private static class LogSeverityProcessor extends ContextualProcessor<String, String, String, String> {
+ @Override
+ public void process(final Record<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ final int colonIndex = record.value().indexOf(':');
+ final String severity = colonIndex > 0 ? record.value().substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Route logs based on severity
+ switch (severity) {
+ case "ERROR":
+ context().forward(record.withKey(ERROR_LOGS_TOPIC));
+ break;
+ case "WARN":
+ context().forward(record.withKey(WARN_LOGS_TOPIC));
+ break;
+ case "INFO":
+ // INFO logs are ignored
+ break;
+ default:
+ // Forward to an "unknown" topic for logs with unrecognized severities
+ context().forward(record.withKey(UNKNOWN_LOGS_TOPIC));
+ }
+ }
+ }
}</code></pre>
- <div class="admonition tip">
- <p><b>Tip</b></p>
- <p class="last">Even though we do not demonstrate it in this example, a stream processor can access any available state stores by
- calling <code class="docutils literal"><span class="pre">ProcessorContext#getStateStore()</span></code>.
- State stores are only available if they have been connected to the processor, or if they are global stores. While global stores do not need to be connected explicitly, they only allow for read-only access.
- There are two ways to connect state stores to a processor:
- <ul class="simple">
- <li>By passing the name of a store that has already been added via <code class="docutils literal"><span class="pre">Topology#addStateStore()</span></code> to the corresponding <code class="docutils literal"><span class="pre">KStream#process()</span></code> method call.</li>
- <li>Implementing <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the <code class="docutils literal"><span class="pre">ProcessorSupplier</span></code>
- passed to <code class="docutils literal"><span class="pre">KStream#process()</span></code>. In this case there is no need to call <code class="docutils literal"><span class="pre">StreamsBuilder#addStateStore()</span></code>
- beforehand, the store will be automatically added for you. You can also implement <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the
- <code class="docutils literal"><span class="pre">Value*</span></code> or <code class="docutils literal"><span class="pre">*WithKey</span></code> supplier variants, or <code class="docutils literal"><span class="pre">TransformerSupplier</span></code> or any of its variants.
- </li>
- </ul>
- </div>
- <p>Then we can leverage the <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code> processor in the DSL via <code class="docutils literal"><span class="pre">KStream#process</span></code>.</p>
- <pre class="line-numbers"><code class="language-java">KStream<String, GenericRecord> pageViews = ...;
-
-// Send an email notification when the view count of a page reaches one thousand.
-pageViews.groupByKey()
- .count()
- .filter((PageId pageId, Long viewCount) -> viewCount == 1000)
- // PopularPageEmailAlert is your custom processor that implements the
- // `Processor` interface, see further down below.
- .process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
- </div>
+ <h5 id="traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</h5>
+ <ul>
+ <li><strong>Idea:</strong> A radar monitors cars passing along a road stretch. A system counts the cars for each
+ day, maintaining a cumulative total for the current day in a state store. At the end of the day, the count
+ is
+ emitted and the state is cleared for the next day.</li>
+ <li><strong>Real-World Context:</strong> A car counting system can be useful for determining measures for
+ widening
+ or controlling traffic depending on the number of cars passing through the monitored stretch.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.common.serialization;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessorContext;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.state.Stores;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.format.DateTimeFormatter;
+
+public class TrafficRadarMonitoringCarCountExample {
+ private static final String DAILY_COUNT_STORE = "price-state-store";
+ private static final String DAILY_COUNT_TOPIC = "price-state-topic";
+ private static final String RADAR_COUNT_TOPIC = "car-radar-topic";
+
+ public static void countWithProcessValues(final StreamsBuilder builder) {
+ // Define a state store for tracking daily car counts
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(DAILY_COUNT_STORE),
+ Serdes.String(),
+ Serdes.Long()
+ )
+ );
+ final KStream<Void, String> radarStream = builder.stream(RADAR_COUNT_TOPIC);
+ // Apply the FixedKeyProcessor with the state store
+ radarStream.processValues(DailyCarCountProcessor::new, DAILY_COUNT_STORE)
+ .to(DAILY_COUNT_TOPIC);
+ }
+
+ private static class DailyCarCountProcessor implements FixedKeyProcessor<Void, String, String> {
+ private FixedKeyProcessorContext<Void, String> context;
+ private KeyValueStore<String, Long> stateStore;
+ private static final DateTimeFormatter DATE_FORMATTER =
+ DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(ZoneId.systemDefault());
+
+ @Override
+ public void init(final FixedKeyProcessorContext<Void, String> context) {
+ this.context = context;
+ stateStore = context.getStateStore(DAILY_COUNT_STORE);
+ }
+
+ @Override
+ public void process(final FixedKeyRecord<Void, String> record) {
+ if (record.value() == null) {
+ return; // Skip null events
+ }
+
+ // Derive the current day from the event timestamp
+ final long timestamp = System.currentTimeMillis(); // Use system time for simplicity
+ final String currentDay = DATE_FORMATTER.format(Instant.ofEpochMilli(timestamp));
+ // Retrieve the current count for the day
+ Long dailyCount = stateStore.get(currentDay);
+ if (dailyCount == null) {
+ dailyCount = 0L;
+ }
+ // Increment the count
+ dailyCount++;
+ stateStore.put(currentDay, dailyCount);
+
+ // Emit the current day's count
+ context.forward(record.withValue(String.format("Day: %s, Car Count: %s", currentDay, dailyCount)));
+ }
+ }
+}</code></pre>
+ <h5 id="replacing-slang-in-text-messages">Replacing Slang in Text Messages</h5>
+ <ul>
+ <li><strong>Idea:</strong> A messaging stream contains user-generated content, and you want to replace slang
+ words
+ with their formal equivalents (e.g., "u" becomes "you", "brb" becomes "be
+ right back"). The operation only modifies the message value and keeps the key intact.</li>
+ <li><strong>Real-World Context:</strong> In customer support chat systems, normalizing text by replacing slang
+ with
+ formal equivalents ensures that automated sentiment analysis tools work accurately and provide reliable
+ insights.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualFixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
+
+public class ReplacingSlangTextInMessagesExample {
+ private static final Map<String, String> SLANG_DICTIONARY = Map.of(
+ "u", "you",
+ "brb", "be right back",
+ "omg", "oh my god",
+ "btw", "by the way"
+ );
+ private static final String INPUT_MESSAGES_TOPIC = "input-messages-topic";
+ private static final String OUTPUT_MESSAGES_TOPIC = "output-messages-topic";
+
+ public static void replaceWithProcessValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.processValues(SlangReplacementProcessor::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ private static class SlangReplacementProcessor extends ContextualFixedKeyProcessor<String, String, String> {
+ @Override
+ public void process(final FixedKeyRecord<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+
+ // Replace slang words in the message
+ final String[] words = record.value().split("\\s+");
+ for (final String word : words) {
+ String replacedWord = SLANG_DICTIONARY.getOrDefault(word, word);
+ context().forward(record.withValue(replacedWord));
+ }
+ }
+ }
+}</code></pre>
+ <h4>Keynotes</h4>
+ <ul>
+ <li><strong>Type Safety and Flexibility:</strong> The process and processValues APIs utilize
+ <code>ProcessorContext</code> and <code>Record</code> or <code>FixedKeyRecord</code> objects for better type
+ safety and flexibility of custom processing logic.
+ </li>
+ <li><strong>Clear State and Logic Management:</strong> Implementations for <code>Processor</code> or
+ <code>FixedKeyProcessor</code> should manage state and logic clearly. Use <code>context().forward()</code>
+ for emitting records downstream.
+ </li>
+ <li><strong>Unified API:</strong> Consolidates multiple methods into a single, versatile API.</li>
+ <li><strong>Future-Proof:</strong> Ensures compatibility with the latest Kafka Streams releases.</li>
+ </ul>
+ </div>
+ <div class="section" id="transformers-removal-and-migration-to-processors">
+ <a class="headerlink" href="#transformers-removal-and-migration-to-processors" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id37">Transformers removal and migration to processors</a>
+ </h3>
+ </a>
+ <p>As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>,<br><code>transformValues</code>, <code>flatTransformValues</code>, and
+ <code>process</code> have been removed. These methods have been replaced with the<br>more versatile Processor API.
+ This guide provides detailed steps for migrating existing code to use the new Processor<br>API and explains the
+ benefits of the changes.</p>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ <li><code>KStream#process</code></li>
+ </ul>
+ <p>The Processor API now serves as a unified replacement for all these methods. It simplifies the API surface
+ while<br>maintaining support for both stateless and stateful operations.</p>
+ <h4>Migration Examples</h4>
+ <p>To migrate from the deprecated <code>transform</code>, <code>transformValues</code>, <code>flatTransform</code>, and
+ <code>flatTransformValues</code> methods to the<br>Processor API (PAPI) in Kafka Streams, let's resume the
+ previouss examples. The new <code>process</code> and <code>processValues</code><br>methods enable a more flexible
+ and reusable approach by requiring implementations of the <code>Processor</code>
+ or<br><code>FixedKeyProcessor</code> interfaces.</p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Migrating from</th>
+ <th>Migrating to</th>
+ <th>State Type</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program-removal">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>transform</code></td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity-removal">Categorizing Logs by Severity</a></td>
+ <td><code>flatTransform</code></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count-removal">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>transformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages-removal">Replacing Slang in Text Messages</a></td>
+ <td><code>flatTransformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
+ </tbody>
+ </table>
+ <h5 id="cumulative-discounts-for-a-loyalty-program-removal">Cumulative Discounts for a Loyalty Program</h5>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
Review Comment:
Remove `package` and imports ?
Especially the package is "weird" as KS does not contain the code of the example, so using the actually KS package of the DSL seems not to be ideal, but could be confusing.
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
+ </li>
+ <li><code>ProcessorSupplier</code>: A processor supplier that can create one or more <code>Processor</code>
+ instances; and</li>
+ <li><code>FixedKeyProcessorSupplier</code>: A processor supplier that can create one or more
+ <code>FixedKeyProcessor</code> instances.
+ </li>
+ </ul>
+ <h4>Examples</h4>
+ <p>Follow the examples below to learn how to apply <code>process</code> and <code>processValues</code> to your
+ <code>KStream</code>.
+ </p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Operation</th>
+ <th>State Type</th>
+ </tr>
</thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
</tbody>
</table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <h5 id="cumulative-discounts-for-a-loyalty-program"">Cumulative Discounts for a Loyalty Program</h5>
+ <ul>
+ <li><strong>Idea:</strong> A stream of purchase events contains user IDs and transaction amounts. Use a state
+ store
+ to accumulate the total spending of each user. When their total crosses a threshold, apply a discount on
+ their next transaction and update their accumulated total.</li>
+ <li><strong>Real-World Context:</strong> In a retail loyalty program, tracking cumulative customer spending
+ enables
+ dynamic rewards, such as issuing a discount when a customer's total purchases exceed a predefined limit.
+ </li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+
+public class CumulativeDiscountsForALoyaltyProgramExample {
+ private static final double DISCOUNT_THRESHOLD = 100.0;
+ private static final String CUSTOMER_SPENDING_STORE = "customer-spending-store";
+ private static final String DISCOUNT_NOTIFICATION_MESSAGE =
+ "Discount applied! You have received a reward for your purchases.";
+ private static final String DISCOUNT_NOTIFICATIONS_TOPIC = "discount-notifications-topic";
+ private static final String PURCHASE_EVENTS_TOPIC = "purchase-events-topic";
+
+ public static void applyDiscountWithProcess(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ Serdes.String(),
+ Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Processor with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.process(CumulativeDiscountProcessor::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ private static class CumulativeDiscountProcessor implements Processor<String, Double, String, String> {
+ private KeyValueStore<String, Double> spendingStore;
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public void process(final Record<String, Double> record) {
+ if (record.value() == null) {
+ return; // Skip null purchase amounts
+ }
- @Override
- void process(Record<PageId, Long> record) {
- // Here you would format and send the alert email.
- //
- // In this specific example, you would be able to include
- // information about the page's ID and its view count
- }
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(record.key());
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += record.value();
+ spendingStore.put(record.key(), currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(record.key(), currentSpending - DISCOUNT_THRESHOLD);
+ // Send a discount notification
+ context.forward(record.withValue(DISCOUNT_NOTIFICATION_MESSAGE));
+ }
+ }
+ }
+}</code></pre>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li><strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g.,
+ INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to a dedicated topic too.</li>
+ <li><strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity
+ ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks,
+ and
+ INFO logs are stored for basic reporting purposes.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- void close() {
- // Any code for clean up would go here, for example tearing down the email client and anything
- // else you created in the init() method
- // This processor instance will not be used again after this call.
- }
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
Review Comment:
Same question as above.
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
+ </li>
+ <li><code>ProcessorSupplier</code>: A processor supplier that can create one or more <code>Processor</code>
+ instances; and</li>
+ <li><code>FixedKeyProcessorSupplier</code>: A processor supplier that can create one or more
+ <code>FixedKeyProcessor</code> instances.
+ </li>
+ </ul>
+ <h4>Examples</h4>
+ <p>Follow the examples below to learn how to apply <code>process</code> and <code>processValues</code> to your
+ <code>KStream</code>.
+ </p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Operation</th>
+ <th>State Type</th>
+ </tr>
</thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
</tbody>
</table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <h5 id="cumulative-discounts-for-a-loyalty-program"">Cumulative Discounts for a Loyalty Program</h5>
+ <ul>
+ <li><strong>Idea:</strong> A stream of purchase events contains user IDs and transaction amounts. Use a state
+ store
+ to accumulate the total spending of each user. When their total crosses a threshold, apply a discount on
+ their next transaction and update their accumulated total.</li>
+ <li><strong>Real-World Context:</strong> In a retail loyalty program, tracking cumulative customer spending
+ enables
+ dynamic rewards, such as issuing a discount when a customer's total purchases exceed a predefined limit.
+ </li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+
+public class CumulativeDiscountsForALoyaltyProgramExample {
+ private static final double DISCOUNT_THRESHOLD = 100.0;
+ private static final String CUSTOMER_SPENDING_STORE = "customer-spending-store";
+ private static final String DISCOUNT_NOTIFICATION_MESSAGE =
+ "Discount applied! You have received a reward for your purchases.";
+ private static final String DISCOUNT_NOTIFICATIONS_TOPIC = "discount-notifications-topic";
+ private static final String PURCHASE_EVENTS_TOPIC = "purchase-events-topic";
+
+ public static void applyDiscountWithProcess(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ Serdes.String(),
+ Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Processor with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.process(CumulativeDiscountProcessor::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ private static class CumulativeDiscountProcessor implements Processor<String, Double, String, String> {
+ private KeyValueStore<String, Double> spendingStore;
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public void process(final Record<String, Double> record) {
+ if (record.value() == null) {
+ return; // Skip null purchase amounts
+ }
- @Override
- void process(Record<PageId, Long> record) {
- // Here you would format and send the alert email.
- //
- // In this specific example, you would be able to include
- // information about the page's ID and its view count
- }
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(record.key());
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += record.value();
+ spendingStore.put(record.key(), currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(record.key(), currentSpending - DISCOUNT_THRESHOLD);
+ // Send a discount notification
+ context.forward(record.withValue(DISCOUNT_NOTIFICATION_MESSAGE));
+ }
+ }
+ }
+}</code></pre>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li><strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g.,
+ INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to a dedicated topic too.</li>
+ <li><strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity
+ ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks,
+ and
+ INFO logs are stored for basic reporting purposes.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- void close() {
- // Any code for clean up would go here, for example tearing down the email client and anything
- // else you created in the init() method
- // This processor instance will not be used again after this call.
- }
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new)
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
+
+ private static class LogSeverityProcessor extends ContextualProcessor<String, String, String, String> {
+ @Override
+ public void process(final Record<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ final int colonIndex = record.value().indexOf(':');
+ final String severity = colonIndex > 0 ? record.value().substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Route logs based on severity
+ switch (severity) {
+ case "ERROR":
+ context().forward(record.withKey(ERROR_LOGS_TOPIC));
+ break;
+ case "WARN":
+ context().forward(record.withKey(WARN_LOGS_TOPIC));
+ break;
+ case "INFO":
+ // INFO logs are ignored
+ break;
+ default:
+ // Forward to an "unknown" topic for logs with unrecognized severities
+ context().forward(record.withKey(UNKNOWN_LOGS_TOPIC));
+ }
+ }
+ }
}</code></pre>
- <div class="admonition tip">
- <p><b>Tip</b></p>
- <p class="last">Even though we do not demonstrate it in this example, a stream processor can access any available state stores by
- calling <code class="docutils literal"><span class="pre">ProcessorContext#getStateStore()</span></code>.
- State stores are only available if they have been connected to the processor, or if they are global stores. While global stores do not need to be connected explicitly, they only allow for read-only access.
- There are two ways to connect state stores to a processor:
- <ul class="simple">
- <li>By passing the name of a store that has already been added via <code class="docutils literal"><span class="pre">Topology#addStateStore()</span></code> to the corresponding <code class="docutils literal"><span class="pre">KStream#process()</span></code> method call.</li>
- <li>Implementing <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the <code class="docutils literal"><span class="pre">ProcessorSupplier</span></code>
- passed to <code class="docutils literal"><span class="pre">KStream#process()</span></code>. In this case there is no need to call <code class="docutils literal"><span class="pre">StreamsBuilder#addStateStore()</span></code>
- beforehand, the store will be automatically added for you. You can also implement <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the
- <code class="docutils literal"><span class="pre">Value*</span></code> or <code class="docutils literal"><span class="pre">*WithKey</span></code> supplier variants, or <code class="docutils literal"><span class="pre">TransformerSupplier</span></code> or any of its variants.
- </li>
- </ul>
- </div>
- <p>Then we can leverage the <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code> processor in the DSL via <code class="docutils literal"><span class="pre">KStream#process</span></code>.</p>
- <pre class="line-numbers"><code class="language-java">KStream<String, GenericRecord> pageViews = ...;
-
-// Send an email notification when the view count of a page reaches one thousand.
-pageViews.groupByKey()
- .count()
- .filter((PageId pageId, Long viewCount) -> viewCount == 1000)
- // PopularPageEmailAlert is your custom processor that implements the
- // `Processor` interface, see further down below.
- .process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
- </div>
+ <h5 id="traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</h5>
+ <ul>
+ <li><strong>Idea:</strong> A radar monitors cars passing along a road stretch. A system counts the cars for each
+ day, maintaining a cumulative total for the current day in a state store. At the end of the day, the count
+ is
+ emitted and the state is cleared for the next day.</li>
+ <li><strong>Real-World Context:</strong> A car counting system can be useful for determining measures for
+ widening
+ or controlling traffic depending on the number of cars passing through the monitored stretch.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.common.serialization;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessorContext;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.state.Stores;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.format.DateTimeFormatter;
Review Comment:
Remove imports?
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
+ </li>
+ <li><code>ProcessorSupplier</code>: A processor supplier that can create one or more <code>Processor</code>
+ instances; and</li>
+ <li><code>FixedKeyProcessorSupplier</code>: A processor supplier that can create one or more
+ <code>FixedKeyProcessor</code> instances.
+ </li>
+ </ul>
+ <h4>Examples</h4>
+ <p>Follow the examples below to learn how to apply <code>process</code> and <code>processValues</code> to your
+ <code>KStream</code>.
+ </p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Operation</th>
+ <th>State Type</th>
+ </tr>
</thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
</tbody>
</table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <h5 id="cumulative-discounts-for-a-loyalty-program"">Cumulative Discounts for a Loyalty Program</h5>
+ <ul>
+ <li><strong>Idea:</strong> A stream of purchase events contains user IDs and transaction amounts. Use a state
+ store
+ to accumulate the total spending of each user. When their total crosses a threshold, apply a discount on
+ their next transaction and update their accumulated total.</li>
+ <li><strong>Real-World Context:</strong> In a retail loyalty program, tracking cumulative customer spending
+ enables
+ dynamic rewards, such as issuing a discount when a customer's total purchases exceed a predefined limit.
+ </li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+
+public class CumulativeDiscountsForALoyaltyProgramExample {
+ private static final double DISCOUNT_THRESHOLD = 100.0;
+ private static final String CUSTOMER_SPENDING_STORE = "customer-spending-store";
+ private static final String DISCOUNT_NOTIFICATION_MESSAGE =
+ "Discount applied! You have received a reward for your purchases.";
+ private static final String DISCOUNT_NOTIFICATIONS_TOPIC = "discount-notifications-topic";
+ private static final String PURCHASE_EVENTS_TOPIC = "purchase-events-topic";
+
+ public static void applyDiscountWithProcess(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ Serdes.String(),
+ Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Processor with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.process(CumulativeDiscountProcessor::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ private static class CumulativeDiscountProcessor implements Processor<String, Double, String, String> {
+ private KeyValueStore<String, Double> spendingStore;
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public void process(final Record<String, Double> record) {
+ if (record.value() == null) {
+ return; // Skip null purchase amounts
+ }
- @Override
- void process(Record<PageId, Long> record) {
- // Here you would format and send the alert email.
- //
- // In this specific example, you would be able to include
- // information about the page's ID and its view count
- }
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(record.key());
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += record.value();
+ spendingStore.put(record.key(), currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(record.key(), currentSpending - DISCOUNT_THRESHOLD);
+ // Send a discount notification
+ context.forward(record.withValue(DISCOUNT_NOTIFICATION_MESSAGE));
+ }
+ }
+ }
+}</code></pre>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li><strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g.,
+ INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to a dedicated topic too.</li>
+ <li><strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity
+ ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks,
+ and
+ INFO logs are stored for basic reporting purposes.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- void close() {
- // Any code for clean up would go here, for example tearing down the email client and anything
- // else you created in the init() method
- // This processor instance will not be used again after this call.
- }
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new)
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
+
+ private static class LogSeverityProcessor extends ContextualProcessor<String, String, String, String> {
+ @Override
+ public void process(final Record<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ final int colonIndex = record.value().indexOf(':');
+ final String severity = colonIndex > 0 ? record.value().substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Route logs based on severity
+ switch (severity) {
+ case "ERROR":
+ context().forward(record.withKey(ERROR_LOGS_TOPIC));
+ break;
+ case "WARN":
+ context().forward(record.withKey(WARN_LOGS_TOPIC));
+ break;
+ case "INFO":
+ // INFO logs are ignored
+ break;
+ default:
+ // Forward to an "unknown" topic for logs with unrecognized severities
+ context().forward(record.withKey(UNKNOWN_LOGS_TOPIC));
+ }
+ }
+ }
}</code></pre>
- <div class="admonition tip">
- <p><b>Tip</b></p>
- <p class="last">Even though we do not demonstrate it in this example, a stream processor can access any available state stores by
- calling <code class="docutils literal"><span class="pre">ProcessorContext#getStateStore()</span></code>.
- State stores are only available if they have been connected to the processor, or if they are global stores. While global stores do not need to be connected explicitly, they only allow for read-only access.
- There are two ways to connect state stores to a processor:
- <ul class="simple">
- <li>By passing the name of a store that has already been added via <code class="docutils literal"><span class="pre">Topology#addStateStore()</span></code> to the corresponding <code class="docutils literal"><span class="pre">KStream#process()</span></code> method call.</li>
- <li>Implementing <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the <code class="docutils literal"><span class="pre">ProcessorSupplier</span></code>
- passed to <code class="docutils literal"><span class="pre">KStream#process()</span></code>. In this case there is no need to call <code class="docutils literal"><span class="pre">StreamsBuilder#addStateStore()</span></code>
- beforehand, the store will be automatically added for you. You can also implement <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the
- <code class="docutils literal"><span class="pre">Value*</span></code> or <code class="docutils literal"><span class="pre">*WithKey</span></code> supplier variants, or <code class="docutils literal"><span class="pre">TransformerSupplier</span></code> or any of its variants.
- </li>
- </ul>
- </div>
- <p>Then we can leverage the <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code> processor in the DSL via <code class="docutils literal"><span class="pre">KStream#process</span></code>.</p>
- <pre class="line-numbers"><code class="language-java">KStream<String, GenericRecord> pageViews = ...;
-
-// Send an email notification when the view count of a page reaches one thousand.
-pageViews.groupByKey()
- .count()
- .filter((PageId pageId, Long viewCount) -> viewCount == 1000)
- // PopularPageEmailAlert is your custom processor that implements the
- // `Processor` interface, see further down below.
- .process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
- </div>
+ <h5 id="traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</h5>
+ <ul>
+ <li><strong>Idea:</strong> A radar monitors cars passing along a road stretch. A system counts the cars for each
+ day, maintaining a cumulative total for the current day in a state store. At the end of the day, the count
+ is
+ emitted and the state is cleared for the next day.</li>
+ <li><strong>Real-World Context:</strong> A car counting system can be useful for determining measures for
+ widening
+ or controlling traffic depending on the number of cars passing through the monitored stretch.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.common.serialization;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessorContext;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.state.Stores;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.format.DateTimeFormatter;
+
+public class TrafficRadarMonitoringCarCountExample {
+ private static final String DAILY_COUNT_STORE = "price-state-store";
+ private static final String DAILY_COUNT_TOPIC = "price-state-topic";
+ private static final String RADAR_COUNT_TOPIC = "car-radar-topic";
+
+ public static void countWithProcessValues(final StreamsBuilder builder) {
+ // Define a state store for tracking daily car counts
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(DAILY_COUNT_STORE),
+ Serdes.String(),
+ Serdes.Long()
+ )
+ );
+ final KStream<Void, String> radarStream = builder.stream(RADAR_COUNT_TOPIC);
+ // Apply the FixedKeyProcessor with the state store
+ radarStream.processValues(DailyCarCountProcessor::new, DAILY_COUNT_STORE)
+ .to(DAILY_COUNT_TOPIC);
+ }
+
+ private static class DailyCarCountProcessor implements FixedKeyProcessor<Void, String, String> {
+ private FixedKeyProcessorContext<Void, String> context;
+ private KeyValueStore<String, Long> stateStore;
+ private static final DateTimeFormatter DATE_FORMATTER =
+ DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(ZoneId.systemDefault());
+
+ @Override
+ public void init(final FixedKeyProcessorContext<Void, String> context) {
+ this.context = context;
+ stateStore = context.getStateStore(DAILY_COUNT_STORE);
+ }
+
+ @Override
+ public void process(final FixedKeyRecord<Void, String> record) {
+ if (record.value() == null) {
+ return; // Skip null events
+ }
+
+ // Derive the current day from the event timestamp
+ final long timestamp = System.currentTimeMillis(); // Use system time for simplicity
+ final String currentDay = DATE_FORMATTER.format(Instant.ofEpochMilli(timestamp));
+ // Retrieve the current count for the day
+ Long dailyCount = stateStore.get(currentDay);
+ if (dailyCount == null) {
+ dailyCount = 0L;
+ }
+ // Increment the count
+ dailyCount++;
+ stateStore.put(currentDay, dailyCount);
+
+ // Emit the current day's count
+ context.forward(record.withValue(String.format("Day: %s, Car Count: %s", currentDay, dailyCount)));
+ }
+ }
+}</code></pre>
+ <h5 id="replacing-slang-in-text-messages">Replacing Slang in Text Messages</h5>
+ <ul>
+ <li><strong>Idea:</strong> A messaging stream contains user-generated content, and you want to replace slang
+ words
+ with their formal equivalents (e.g., "u" becomes "you", "brb" becomes "be
+ right back"). The operation only modifies the message value and keeps the key intact.</li>
+ <li><strong>Real-World Context:</strong> In customer support chat systems, normalizing text by replacing slang
+ with
+ formal equivalents ensures that automated sentiment analysis tools work accurately and provide reliable
+ insights.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualFixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
+
+public class ReplacingSlangTextInMessagesExample {
+ private static final Map<String, String> SLANG_DICTIONARY = Map.of(
+ "u", "you",
+ "brb", "be right back",
+ "omg", "oh my god",
+ "btw", "by the way"
+ );
+ private static final String INPUT_MESSAGES_TOPIC = "input-messages-topic";
+ private static final String OUTPUT_MESSAGES_TOPIC = "output-messages-topic";
+
+ public static void replaceWithProcessValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.processValues(SlangReplacementProcessor::new).to(OUTPUT_MESSAGES_TOPIC);
Review Comment:
nit: don't chain `.to(...);` in the same line, but move to next line -- easier to read.
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
+ </li>
+ <li><code>ProcessorSupplier</code>: A processor supplier that can create one or more <code>Processor</code>
+ instances; and</li>
+ <li><code>FixedKeyProcessorSupplier</code>: A processor supplier that can create one or more
+ <code>FixedKeyProcessor</code> instances.
+ </li>
+ </ul>
+ <h4>Examples</h4>
+ <p>Follow the examples below to learn how to apply <code>process</code> and <code>processValues</code> to your
+ <code>KStream</code>.
+ </p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Operation</th>
+ <th>State Type</th>
+ </tr>
</thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
</tbody>
</table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <h5 id="cumulative-discounts-for-a-loyalty-program"">Cumulative Discounts for a Loyalty Program</h5>
+ <ul>
+ <li><strong>Idea:</strong> A stream of purchase events contains user IDs and transaction amounts. Use a state
+ store
+ to accumulate the total spending of each user. When their total crosses a threshold, apply a discount on
+ their next transaction and update their accumulated total.</li>
+ <li><strong>Real-World Context:</strong> In a retail loyalty program, tracking cumulative customer spending
+ enables
+ dynamic rewards, such as issuing a discount when a customer's total purchases exceed a predefined limit.
+ </li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+
+public class CumulativeDiscountsForALoyaltyProgramExample {
+ private static final double DISCOUNT_THRESHOLD = 100.0;
+ private static final String CUSTOMER_SPENDING_STORE = "customer-spending-store";
+ private static final String DISCOUNT_NOTIFICATION_MESSAGE =
+ "Discount applied! You have received a reward for your purchases.";
+ private static final String DISCOUNT_NOTIFICATIONS_TOPIC = "discount-notifications-topic";
+ private static final String PURCHASE_EVENTS_TOPIC = "purchase-events-topic";
+
+ public static void applyDiscountWithProcess(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ Serdes.String(),
+ Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Processor with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.process(CumulativeDiscountProcessor::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ private static class CumulativeDiscountProcessor implements Processor<String, Double, String, String> {
+ private KeyValueStore<String, Double> spendingStore;
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public void process(final Record<String, Double> record) {
+ if (record.value() == null) {
+ return; // Skip null purchase amounts
+ }
- @Override
- void process(Record<PageId, Long> record) {
- // Here you would format and send the alert email.
- //
- // In this specific example, you would be able to include
- // information about the page's ID and its view count
- }
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(record.key());
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += record.value();
+ spendingStore.put(record.key(), currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(record.key(), currentSpending - DISCOUNT_THRESHOLD);
+ // Send a discount notification
+ context.forward(record.withValue(DISCOUNT_NOTIFICATION_MESSAGE));
+ }
+ }
+ }
+}</code></pre>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li><strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g.,
+ INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to a dedicated topic too.</li>
+ <li><strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity
+ ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks,
+ and
+ INFO logs are stored for basic reporting purposes.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- void close() {
- // Any code for clean up would go here, for example tearing down the email client and anything
- // else you created in the init() method
- // This processor instance will not be used again after this call.
- }
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new)
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
+
+ private static class LogSeverityProcessor extends ContextualProcessor<String, String, String, String> {
+ @Override
+ public void process(final Record<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ final int colonIndex = record.value().indexOf(':');
+ final String severity = colonIndex > 0 ? record.value().substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Route logs based on severity
+ switch (severity) {
+ case "ERROR":
+ context().forward(record.withKey(ERROR_LOGS_TOPIC));
+ break;
+ case "WARN":
+ context().forward(record.withKey(WARN_LOGS_TOPIC));
+ break;
+ case "INFO":
+ // INFO logs are ignored
+ break;
+ default:
+ // Forward to an "unknown" topic for logs with unrecognized severities
+ context().forward(record.withKey(UNKNOWN_LOGS_TOPIC));
+ }
+ }
+ }
}</code></pre>
- <div class="admonition tip">
- <p><b>Tip</b></p>
- <p class="last">Even though we do not demonstrate it in this example, a stream processor can access any available state stores by
- calling <code class="docutils literal"><span class="pre">ProcessorContext#getStateStore()</span></code>.
- State stores are only available if they have been connected to the processor, or if they are global stores. While global stores do not need to be connected explicitly, they only allow for read-only access.
- There are two ways to connect state stores to a processor:
- <ul class="simple">
- <li>By passing the name of a store that has already been added via <code class="docutils literal"><span class="pre">Topology#addStateStore()</span></code> to the corresponding <code class="docutils literal"><span class="pre">KStream#process()</span></code> method call.</li>
- <li>Implementing <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the <code class="docutils literal"><span class="pre">ProcessorSupplier</span></code>
- passed to <code class="docutils literal"><span class="pre">KStream#process()</span></code>. In this case there is no need to call <code class="docutils literal"><span class="pre">StreamsBuilder#addStateStore()</span></code>
- beforehand, the store will be automatically added for you. You can also implement <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the
- <code class="docutils literal"><span class="pre">Value*</span></code> or <code class="docutils literal"><span class="pre">*WithKey</span></code> supplier variants, or <code class="docutils literal"><span class="pre">TransformerSupplier</span></code> or any of its variants.
- </li>
- </ul>
- </div>
- <p>Then we can leverage the <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code> processor in the DSL via <code class="docutils literal"><span class="pre">KStream#process</span></code>.</p>
- <pre class="line-numbers"><code class="language-java">KStream<String, GenericRecord> pageViews = ...;
-
-// Send an email notification when the view count of a page reaches one thousand.
-pageViews.groupByKey()
- .count()
- .filter((PageId pageId, Long viewCount) -> viewCount == 1000)
- // PopularPageEmailAlert is your custom processor that implements the
- // `Processor` interface, see further down below.
- .process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
- </div>
+ <h5 id="traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</h5>
+ <ul>
+ <li><strong>Idea:</strong> A radar monitors cars passing along a road stretch. A system counts the cars for each
+ day, maintaining a cumulative total for the current day in a state store. At the end of the day, the count
+ is
+ emitted and the state is cleared for the next day.</li>
+ <li><strong>Real-World Context:</strong> A car counting system can be useful for determining measures for
+ widening
+ or controlling traffic depending on the number of cars passing through the monitored stretch.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.common.serialization;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessorContext;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.state.Stores;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.format.DateTimeFormatter;
+
+public class TrafficRadarMonitoringCarCountExample {
+ private static final String DAILY_COUNT_STORE = "price-state-store";
+ private static final String DAILY_COUNT_TOPIC = "price-state-topic";
+ private static final String RADAR_COUNT_TOPIC = "car-radar-topic";
+
+ public static void countWithProcessValues(final StreamsBuilder builder) {
+ // Define a state store for tracking daily car counts
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(DAILY_COUNT_STORE),
+ Serdes.String(),
+ Serdes.Long()
+ )
+ );
+ final KStream<Void, String> radarStream = builder.stream(RADAR_COUNT_TOPIC);
+ // Apply the FixedKeyProcessor with the state store
+ radarStream.processValues(DailyCarCountProcessor::new, DAILY_COUNT_STORE)
+ .to(DAILY_COUNT_TOPIC);
+ }
+
+ private static class DailyCarCountProcessor implements FixedKeyProcessor<Void, String, String> {
+ private FixedKeyProcessorContext<Void, String> context;
+ private KeyValueStore<String, Long> stateStore;
+ private static final DateTimeFormatter DATE_FORMATTER =
+ DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(ZoneId.systemDefault());
+
+ @Override
+ public void init(final FixedKeyProcessorContext<Void, String> context) {
+ this.context = context;
+ stateStore = context.getStateStore(DAILY_COUNT_STORE);
+ }
+
+ @Override
+ public void process(final FixedKeyRecord<Void, String> record) {
+ if (record.value() == null) {
+ return; // Skip null events
+ }
+
+ // Derive the current day from the event timestamp
+ final long timestamp = System.currentTimeMillis(); // Use system time for simplicity
+ final String currentDay = DATE_FORMATTER.format(Instant.ofEpochMilli(timestamp));
+ // Retrieve the current count for the day
+ Long dailyCount = stateStore.get(currentDay);
+ if (dailyCount == null) {
+ dailyCount = 0L;
+ }
+ // Increment the count
+ dailyCount++;
+ stateStore.put(currentDay, dailyCount);
+
+ // Emit the current day's count
+ context.forward(record.withValue(String.format("Day: %s, Car Count: %s", currentDay, dailyCount)));
+ }
+ }
+}</code></pre>
+ <h5 id="replacing-slang-in-text-messages">Replacing Slang in Text Messages</h5>
+ <ul>
+ <li><strong>Idea:</strong> A messaging stream contains user-generated content, and you want to replace slang
+ words
+ with their formal equivalents (e.g., "u" becomes "you", "brb" becomes "be
+ right back"). The operation only modifies the message value and keeps the key intact.</li>
+ <li><strong>Real-World Context:</strong> In customer support chat systems, normalizing text by replacing slang
+ with
+ formal equivalents ensures that automated sentiment analysis tools work accurately and provide reliable
+ insights.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualFixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
Review Comment:
Imports?
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
+ </li>
+ <li><code>ProcessorSupplier</code>: A processor supplier that can create one or more <code>Processor</code>
+ instances; and</li>
+ <li><code>FixedKeyProcessorSupplier</code>: A processor supplier that can create one or more
+ <code>FixedKeyProcessor</code> instances.
+ </li>
+ </ul>
+ <h4>Examples</h4>
+ <p>Follow the examples below to learn how to apply <code>process</code> and <code>processValues</code> to your
+ <code>KStream</code>.
+ </p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Operation</th>
+ <th>State Type</th>
+ </tr>
</thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
</tbody>
</table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <h5 id="cumulative-discounts-for-a-loyalty-program"">Cumulative Discounts for a Loyalty Program</h5>
+ <ul>
+ <li><strong>Idea:</strong> A stream of purchase events contains user IDs and transaction amounts. Use a state
+ store
+ to accumulate the total spending of each user. When their total crosses a threshold, apply a discount on
+ their next transaction and update their accumulated total.</li>
+ <li><strong>Real-World Context:</strong> In a retail loyalty program, tracking cumulative customer spending
+ enables
+ dynamic rewards, such as issuing a discount when a customer's total purchases exceed a predefined limit.
+ </li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+
+public class CumulativeDiscountsForALoyaltyProgramExample {
+ private static final double DISCOUNT_THRESHOLD = 100.0;
+ private static final String CUSTOMER_SPENDING_STORE = "customer-spending-store";
+ private static final String DISCOUNT_NOTIFICATION_MESSAGE =
+ "Discount applied! You have received a reward for your purchases.";
+ private static final String DISCOUNT_NOTIFICATIONS_TOPIC = "discount-notifications-topic";
+ private static final String PURCHASE_EVENTS_TOPIC = "purchase-events-topic";
+
+ public static void applyDiscountWithProcess(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ Serdes.String(),
+ Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Processor with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.process(CumulativeDiscountProcessor::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ private static class CumulativeDiscountProcessor implements Processor<String, Double, String, String> {
+ private KeyValueStore<String, Double> spendingStore;
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public void process(final Record<String, Double> record) {
+ if (record.value() == null) {
+ return; // Skip null purchase amounts
+ }
- @Override
- void process(Record<PageId, Long> record) {
- // Here you would format and send the alert email.
- //
- // In this specific example, you would be able to include
- // information about the page's ID and its view count
- }
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(record.key());
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += record.value();
+ spendingStore.put(record.key(), currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(record.key(), currentSpending - DISCOUNT_THRESHOLD);
+ // Send a discount notification
+ context.forward(record.withValue(DISCOUNT_NOTIFICATION_MESSAGE));
+ }
+ }
+ }
+}</code></pre>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li><strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g.,
+ INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to a dedicated topic too.</li>
+ <li><strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity
+ ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks,
+ and
+ INFO logs are stored for basic reporting purposes.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- void close() {
- // Any code for clean up would go here, for example tearing down the email client and anything
- // else you created in the init() method
- // This processor instance will not be used again after this call.
- }
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new)
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
+
+ private static class LogSeverityProcessor extends ContextualProcessor<String, String, String, String> {
+ @Override
+ public void process(final Record<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ final int colonIndex = record.value().indexOf(':');
+ final String severity = colonIndex > 0 ? record.value().substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Route logs based on severity
+ switch (severity) {
+ case "ERROR":
+ context().forward(record.withKey(ERROR_LOGS_TOPIC));
+ break;
+ case "WARN":
+ context().forward(record.withKey(WARN_LOGS_TOPIC));
+ break;
+ case "INFO":
+ // INFO logs are ignored
+ break;
+ default:
+ // Forward to an "unknown" topic for logs with unrecognized severities
+ context().forward(record.withKey(UNKNOWN_LOGS_TOPIC));
+ }
+ }
+ }
}</code></pre>
- <div class="admonition tip">
- <p><b>Tip</b></p>
- <p class="last">Even though we do not demonstrate it in this example, a stream processor can access any available state stores by
- calling <code class="docutils literal"><span class="pre">ProcessorContext#getStateStore()</span></code>.
- State stores are only available if they have been connected to the processor, or if they are global stores. While global stores do not need to be connected explicitly, they only allow for read-only access.
- There are two ways to connect state stores to a processor:
- <ul class="simple">
- <li>By passing the name of a store that has already been added via <code class="docutils literal"><span class="pre">Topology#addStateStore()</span></code> to the corresponding <code class="docutils literal"><span class="pre">KStream#process()</span></code> method call.</li>
- <li>Implementing <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the <code class="docutils literal"><span class="pre">ProcessorSupplier</span></code>
- passed to <code class="docutils literal"><span class="pre">KStream#process()</span></code>. In this case there is no need to call <code class="docutils literal"><span class="pre">StreamsBuilder#addStateStore()</span></code>
- beforehand, the store will be automatically added for you. You can also implement <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the
- <code class="docutils literal"><span class="pre">Value*</span></code> or <code class="docutils literal"><span class="pre">*WithKey</span></code> supplier variants, or <code class="docutils literal"><span class="pre">TransformerSupplier</span></code> or any of its variants.
- </li>
- </ul>
- </div>
- <p>Then we can leverage the <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code> processor in the DSL via <code class="docutils literal"><span class="pre">KStream#process</span></code>.</p>
- <pre class="line-numbers"><code class="language-java">KStream<String, GenericRecord> pageViews = ...;
-
-// Send an email notification when the view count of a page reaches one thousand.
-pageViews.groupByKey()
- .count()
- .filter((PageId pageId, Long viewCount) -> viewCount == 1000)
- // PopularPageEmailAlert is your custom processor that implements the
- // `Processor` interface, see further down below.
- .process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
- </div>
+ <h5 id="traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</h5>
+ <ul>
+ <li><strong>Idea:</strong> A radar monitors cars passing along a road stretch. A system counts the cars for each
+ day, maintaining a cumulative total for the current day in a state store. At the end of the day, the count
+ is
+ emitted and the state is cleared for the next day.</li>
+ <li><strong>Real-World Context:</strong> A car counting system can be useful for determining measures for
+ widening
+ or controlling traffic depending on the number of cars passing through the monitored stretch.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.common.serialization;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessorContext;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.state.Stores;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.format.DateTimeFormatter;
+
+public class TrafficRadarMonitoringCarCountExample {
+ private static final String DAILY_COUNT_STORE = "price-state-store";
+ private static final String DAILY_COUNT_TOPIC = "price-state-topic";
+ private static final String RADAR_COUNT_TOPIC = "car-radar-topic";
+
+ public static void countWithProcessValues(final StreamsBuilder builder) {
+ // Define a state store for tracking daily car counts
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(DAILY_COUNT_STORE),
+ Serdes.String(),
+ Serdes.Long()
+ )
+ );
+ final KStream<Void, String> radarStream = builder.stream(RADAR_COUNT_TOPIC);
+ // Apply the FixedKeyProcessor with the state store
+ radarStream.processValues(DailyCarCountProcessor::new, DAILY_COUNT_STORE)
+ .to(DAILY_COUNT_TOPIC);
+ }
+
+ private static class DailyCarCountProcessor implements FixedKeyProcessor<Void, String, String> {
+ private FixedKeyProcessorContext<Void, String> context;
+ private KeyValueStore<String, Long> stateStore;
+ private static final DateTimeFormatter DATE_FORMATTER =
+ DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(ZoneId.systemDefault());
+
+ @Override
+ public void init(final FixedKeyProcessorContext<Void, String> context) {
+ this.context = context;
+ stateStore = context.getStateStore(DAILY_COUNT_STORE);
+ }
+
+ @Override
+ public void process(final FixedKeyRecord<Void, String> record) {
+ if (record.value() == null) {
+ return; // Skip null events
+ }
+
+ // Derive the current day from the event timestamp
+ final long timestamp = System.currentTimeMillis(); // Use system time for simplicity
+ final String currentDay = DATE_FORMATTER.format(Instant.ofEpochMilli(timestamp));
+ // Retrieve the current count for the day
+ Long dailyCount = stateStore.get(currentDay);
+ if (dailyCount == null) {
+ dailyCount = 0L;
+ }
+ // Increment the count
+ dailyCount++;
+ stateStore.put(currentDay, dailyCount);
+
+ // Emit the current day's count
+ context.forward(record.withValue(String.format("Day: %s, Car Count: %s", currentDay, dailyCount)));
+ }
+ }
+}</code></pre>
+ <h5 id="replacing-slang-in-text-messages">Replacing Slang in Text Messages</h5>
+ <ul>
+ <li><strong>Idea:</strong> A messaging stream contains user-generated content, and you want to replace slang
+ words
+ with their formal equivalents (e.g., "u" becomes "you", "brb" becomes "be
+ right back"). The operation only modifies the message value and keeps the key intact.</li>
+ <li><strong>Real-World Context:</strong> In customer support chat systems, normalizing text by replacing slang
+ with
+ formal equivalents ensures that automated sentiment analysis tools work accurately and provide reliable
+ insights.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualFixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
+
+public class ReplacingSlangTextInMessagesExample {
+ private static final Map<String, String> SLANG_DICTIONARY = Map.of(
+ "u", "you",
+ "brb", "be right back",
+ "omg", "oh my god",
+ "btw", "by the way"
+ );
+ private static final String INPUT_MESSAGES_TOPIC = "input-messages-topic";
+ private static final String OUTPUT_MESSAGES_TOPIC = "output-messages-topic";
+
+ public static void replaceWithProcessValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.processValues(SlangReplacementProcessor::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ private static class SlangReplacementProcessor extends ContextualFixedKeyProcessor<String, String, String> {
+ @Override
+ public void process(final FixedKeyRecord<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+
+ // Replace slang words in the message
+ final String[] words = record.value().split("\\s+");
+ for (final String word : words) {
+ String replacedWord = SLANG_DICTIONARY.getOrDefault(word, word);
+ context().forward(record.withValue(replacedWord));
+ }
+ }
+ }
+}</code></pre>
+ <h4>Keynotes</h4>
+ <ul>
+ <li><strong>Type Safety and Flexibility:</strong> The process and processValues APIs utilize
+ <code>ProcessorContext</code> and <code>Record</code> or <code>FixedKeyRecord</code> objects for better type
+ safety and flexibility of custom processing logic.
+ </li>
+ <li><strong>Clear State and Logic Management:</strong> Implementations for <code>Processor</code> or
+ <code>FixedKeyProcessor</code> should manage state and logic clearly. Use <code>context().forward()</code>
+ for emitting records downstream.
+ </li>
+ <li><strong>Unified API:</strong> Consolidates multiple methods into a single, versatile API.</li>
+ <li><strong>Future-Proof:</strong> Ensures compatibility with the latest Kafka Streams releases.</li>
+ </ul>
+ </div>
+ <div class="section" id="transformers-removal-and-migration-to-processors">
+ <a class="headerlink" href="#transformers-removal-and-migration-to-processors" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id37">Transformers removal and migration to processors</a>
+ </h3>
+ </a>
+ <p>As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>,<br><code>transformValues</code>, <code>flatTransformValues</code>, and
+ <code>process</code> have been removed. These methods have been replaced with the<br>more versatile Processor API.
+ This guide provides detailed steps for migrating existing code to use the new Processor<br>API and explains the
+ benefits of the changes.</p>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ <li><code>KStream#process</code></li>
Review Comment:
Should we somehow highlight that we refer to the "old" `process(...)` methods -- there is still the new `process()` method -- otherwise, could be confusing?
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
+ </li>
+ <li><code>ProcessorSupplier</code>: A processor supplier that can create one or more <code>Processor</code>
+ instances; and</li>
+ <li><code>FixedKeyProcessorSupplier</code>: A processor supplier that can create one or more
+ <code>FixedKeyProcessor</code> instances.
+ </li>
+ </ul>
+ <h4>Examples</h4>
+ <p>Follow the examples below to learn how to apply <code>process</code> and <code>processValues</code> to your
+ <code>KStream</code>.
+ </p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Operation</th>
+ <th>State Type</th>
+ </tr>
</thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
</tbody>
</table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <h5 id="cumulative-discounts-for-a-loyalty-program"">Cumulative Discounts for a Loyalty Program</h5>
+ <ul>
+ <li><strong>Idea:</strong> A stream of purchase events contains user IDs and transaction amounts. Use a state
+ store
+ to accumulate the total spending of each user. When their total crosses a threshold, apply a discount on
+ their next transaction and update their accumulated total.</li>
+ <li><strong>Real-World Context:</strong> In a retail loyalty program, tracking cumulative customer spending
+ enables
+ dynamic rewards, such as issuing a discount when a customer's total purchases exceed a predefined limit.
+ </li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+
+public class CumulativeDiscountsForALoyaltyProgramExample {
+ private static final double DISCOUNT_THRESHOLD = 100.0;
+ private static final String CUSTOMER_SPENDING_STORE = "customer-spending-store";
+ private static final String DISCOUNT_NOTIFICATION_MESSAGE =
+ "Discount applied! You have received a reward for your purchases.";
+ private static final String DISCOUNT_NOTIFICATIONS_TOPIC = "discount-notifications-topic";
+ private static final String PURCHASE_EVENTS_TOPIC = "purchase-events-topic";
+
+ public static void applyDiscountWithProcess(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ Serdes.String(),
+ Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Processor with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.process(CumulativeDiscountProcessor::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ private static class CumulativeDiscountProcessor implements Processor<String, Double, String, String> {
+ private KeyValueStore<String, Double> spendingStore;
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public void process(final Record<String, Double> record) {
+ if (record.value() == null) {
+ return; // Skip null purchase amounts
+ }
- @Override
- void process(Record<PageId, Long> record) {
- // Here you would format and send the alert email.
- //
- // In this specific example, you would be able to include
- // information about the page's ID and its view count
- }
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(record.key());
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += record.value();
+ spendingStore.put(record.key(), currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(record.key(), currentSpending - DISCOUNT_THRESHOLD);
+ // Send a discount notification
+ context.forward(record.withValue(DISCOUNT_NOTIFICATION_MESSAGE));
+ }
+ }
+ }
+}</code></pre>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li><strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g.,
+ INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to a dedicated topic too.</li>
+ <li><strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity
+ ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks,
+ and
+ INFO logs are stored for basic reporting purposes.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- void close() {
- // Any code for clean up would go here, for example tearing down the email client and anything
- // else you created in the init() method
- // This processor instance will not be used again after this call.
- }
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new)
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
+
+ private static class LogSeverityProcessor extends ContextualProcessor<String, String, String, String> {
+ @Override
+ public void process(final Record<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ final int colonIndex = record.value().indexOf(':');
+ final String severity = colonIndex > 0 ? record.value().substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Route logs based on severity
+ switch (severity) {
+ case "ERROR":
+ context().forward(record.withKey(ERROR_LOGS_TOPIC));
+ break;
+ case "WARN":
+ context().forward(record.withKey(WARN_LOGS_TOPIC));
+ break;
+ case "INFO":
+ // INFO logs are ignored
+ break;
+ default:
+ // Forward to an "unknown" topic for logs with unrecognized severities
+ context().forward(record.withKey(UNKNOWN_LOGS_TOPIC));
+ }
+ }
+ }
}</code></pre>
- <div class="admonition tip">
- <p><b>Tip</b></p>
- <p class="last">Even though we do not demonstrate it in this example, a stream processor can access any available state stores by
- calling <code class="docutils literal"><span class="pre">ProcessorContext#getStateStore()</span></code>.
- State stores are only available if they have been connected to the processor, or if they are global stores. While global stores do not need to be connected explicitly, they only allow for read-only access.
- There are two ways to connect state stores to a processor:
- <ul class="simple">
- <li>By passing the name of a store that has already been added via <code class="docutils literal"><span class="pre">Topology#addStateStore()</span></code> to the corresponding <code class="docutils literal"><span class="pre">KStream#process()</span></code> method call.</li>
- <li>Implementing <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the <code class="docutils literal"><span class="pre">ProcessorSupplier</span></code>
- passed to <code class="docutils literal"><span class="pre">KStream#process()</span></code>. In this case there is no need to call <code class="docutils literal"><span class="pre">StreamsBuilder#addStateStore()</span></code>
- beforehand, the store will be automatically added for you. You can also implement <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the
- <code class="docutils literal"><span class="pre">Value*</span></code> or <code class="docutils literal"><span class="pre">*WithKey</span></code> supplier variants, or <code class="docutils literal"><span class="pre">TransformerSupplier</span></code> or any of its variants.
- </li>
- </ul>
- </div>
- <p>Then we can leverage the <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code> processor in the DSL via <code class="docutils literal"><span class="pre">KStream#process</span></code>.</p>
- <pre class="line-numbers"><code class="language-java">KStream<String, GenericRecord> pageViews = ...;
-
-// Send an email notification when the view count of a page reaches one thousand.
-pageViews.groupByKey()
- .count()
- .filter((PageId pageId, Long viewCount) -> viewCount == 1000)
- // PopularPageEmailAlert is your custom processor that implements the
- // `Processor` interface, see further down below.
- .process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
- </div>
+ <h5 id="traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</h5>
+ <ul>
+ <li><strong>Idea:</strong> A radar monitors cars passing along a road stretch. A system counts the cars for each
+ day, maintaining a cumulative total for the current day in a state store. At the end of the day, the count
+ is
+ emitted and the state is cleared for the next day.</li>
+ <li><strong>Real-World Context:</strong> A car counting system can be useful for determining measures for
+ widening
+ or controlling traffic depending on the number of cars passing through the monitored stretch.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.common.serialization;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessorContext;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.state.Stores;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.format.DateTimeFormatter;
+
+public class TrafficRadarMonitoringCarCountExample {
+ private static final String DAILY_COUNT_STORE = "price-state-store";
+ private static final String DAILY_COUNT_TOPIC = "price-state-topic";
+ private static final String RADAR_COUNT_TOPIC = "car-radar-topic";
+
+ public static void countWithProcessValues(final StreamsBuilder builder) {
+ // Define a state store for tracking daily car counts
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(DAILY_COUNT_STORE),
+ Serdes.String(),
+ Serdes.Long()
+ )
+ );
+ final KStream<Void, String> radarStream = builder.stream(RADAR_COUNT_TOPIC);
+ // Apply the FixedKeyProcessor with the state store
+ radarStream.processValues(DailyCarCountProcessor::new, DAILY_COUNT_STORE)
+ .to(DAILY_COUNT_TOPIC);
+ }
+
+ private static class DailyCarCountProcessor implements FixedKeyProcessor<Void, String, String> {
+ private FixedKeyProcessorContext<Void, String> context;
+ private KeyValueStore<String, Long> stateStore;
+ private static final DateTimeFormatter DATE_FORMATTER =
+ DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(ZoneId.systemDefault());
+
+ @Override
+ public void init(final FixedKeyProcessorContext<Void, String> context) {
+ this.context = context;
+ stateStore = context.getStateStore(DAILY_COUNT_STORE);
+ }
+
+ @Override
+ public void process(final FixedKeyRecord<Void, String> record) {
+ if (record.value() == null) {
+ return; // Skip null events
+ }
+
+ // Derive the current day from the event timestamp
+ final long timestamp = System.currentTimeMillis(); // Use system time for simplicity
+ final String currentDay = DATE_FORMATTER.format(Instant.ofEpochMilli(timestamp));
+ // Retrieve the current count for the day
+ Long dailyCount = stateStore.get(currentDay);
+ if (dailyCount == null) {
+ dailyCount = 0L;
+ }
+ // Increment the count
+ dailyCount++;
+ stateStore.put(currentDay, dailyCount);
+
+ // Emit the current day's count
+ context.forward(record.withValue(String.format("Day: %s, Car Count: %s", currentDay, dailyCount)));
+ }
+ }
+}</code></pre>
+ <h5 id="replacing-slang-in-text-messages">Replacing Slang in Text Messages</h5>
+ <ul>
+ <li><strong>Idea:</strong> A messaging stream contains user-generated content, and you want to replace slang
+ words
+ with their formal equivalents (e.g., "u" becomes "you", "brb" becomes "be
+ right back"). The operation only modifies the message value and keeps the key intact.</li>
+ <li><strong>Real-World Context:</strong> In customer support chat systems, normalizing text by replacing slang
+ with
+ formal equivalents ensures that automated sentiment analysis tools work accurately and provide reliable
+ insights.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualFixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
+
+public class ReplacingSlangTextInMessagesExample {
+ private static final Map<String, String> SLANG_DICTIONARY = Map.of(
+ "u", "you",
+ "brb", "be right back",
+ "omg", "oh my god",
+ "btw", "by the way"
+ );
+ private static final String INPUT_MESSAGES_TOPIC = "input-messages-topic";
+ private static final String OUTPUT_MESSAGES_TOPIC = "output-messages-topic";
+
+ public static void replaceWithProcessValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.processValues(SlangReplacementProcessor::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ private static class SlangReplacementProcessor extends ContextualFixedKeyProcessor<String, String, String> {
+ @Override
+ public void process(final FixedKeyRecord<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+
+ // Replace slang words in the message
+ final String[] words = record.value().split("\\s+");
+ for (final String word : words) {
+ String replacedWord = SLANG_DICTIONARY.getOrDefault(word, word);
+ context().forward(record.withValue(replacedWord));
+ }
+ }
+ }
+}</code></pre>
+ <h4>Keynotes</h4>
+ <ul>
+ <li><strong>Type Safety and Flexibility:</strong> The process and processValues APIs utilize
+ <code>ProcessorContext</code> and <code>Record</code> or <code>FixedKeyRecord</code> objects for better type
+ safety and flexibility of custom processing logic.
+ </li>
+ <li><strong>Clear State and Logic Management:</strong> Implementations for <code>Processor</code> or
+ <code>FixedKeyProcessor</code> should manage state and logic clearly. Use <code>context().forward()</code>
+ for emitting records downstream.
+ </li>
+ <li><strong>Unified API:</strong> Consolidates multiple methods into a single, versatile API.</li>
+ <li><strong>Future-Proof:</strong> Ensures compatibility with the latest Kafka Streams releases.</li>
+ </ul>
+ </div>
+ <div class="section" id="transformers-removal-and-migration-to-processors">
+ <a class="headerlink" href="#transformers-removal-and-migration-to-processors" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id37">Transformers removal and migration to processors</a>
+ </h3>
+ </a>
+ <p>As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>,<br><code>transformValues</code>, <code>flatTransformValues</code>, and
+ <code>process</code> have been removed. These methods have been replaced with the<br>more versatile Processor API.
+ This guide provides detailed steps for migrating existing code to use the new Processor<br>API and explains the
+ benefits of the changes.</p>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ <li><code>KStream#process</code></li>
+ </ul>
+ <p>The Processor API now serves as a unified replacement for all these methods. It simplifies the API surface
+ while<br>maintaining support for both stateless and stateful operations.</p>
+ <h4>Migration Examples</h4>
+ <p>To migrate from the deprecated <code>transform</code>, <code>transformValues</code>, <code>flatTransform</code>, and
+ <code>flatTransformValues</code> methods to the<br>Processor API (PAPI) in Kafka Streams, let's resume the
+ previouss examples. The new <code>process</code> and <code>processValues</code><br>methods enable a more flexible
+ and reusable approach by requiring implementations of the <code>Processor</code>
+ or<br><code>FixedKeyProcessor</code> interfaces.</p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Migrating from</th>
+ <th>Migrating to</th>
+ <th>State Type</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program-removal">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>transform</code></td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity-removal">Categorizing Logs by Severity</a></td>
+ <td><code>flatTransform</code></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count-removal">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>transformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages-removal">Replacing Slang in Text Messages</a></td>
+ <td><code>flatTransformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
+ </tbody>
+ </table>
+ <h5 id="cumulative-discounts-for-a-loyalty-program-removal">Cumulative Discounts for a Loyalty Program</h5>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+
+public class CumulativeDiscountsForALoyaltyProgramExample {
+ private static final double DISCOUNT_THRESHOLD = 100.0;
+ private static final String CUSTOMER_SPENDING_STORE = "customer-spending-store";
+ private static final String DISCOUNT_NOTIFICATION_MESSAGE =
+ "Discount applied! You have received a reward for your purchases.";
+ private static final String DISCOUNT_NOTIFICATIONS_TOPIC = "discount-notifications-topic";
+ private static final String PURCHASE_EVENTS_TOPIC = "purchase-events-topic";
+
+ public static void applyDiscountWithTransform(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ Serdes.String(),
+ Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Transformer with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.transform(CumulativeDiscountTransformer::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
+
+ public static void applyDiscountWithProcess(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ org.apache.kafka.common.serialization.Serdes.String(),
+ org.apache.kafka.common.serialization.Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Processor with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.process(CumulativeDiscountProcessor::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
+
+ private static class CumulativeDiscountTransformer implements Transformer<String, Double, KeyValue<String, String>> {
+ private KeyValueStore<String, Double> spendingStore;
+
+ @Override
+ public void init(final org.apache.kafka.streams.processor.ProcessorContext context) {
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public KeyValue<String, String> transform(final String key, final Double value) {
+ if (value == null) {
+ return null; // Skip null purchase amounts
+ }
+
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(key);
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += value;
+ spendingStore.put(key, currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(key, currentSpending - DISCOUNT_THRESHOLD);
+ // Return a notification message
+ return new KeyValue<>(key, DISCOUNT_NOTIFICATION_MESSAGE);
+ }
+ return null; // No discount, so no output for this record
+ }
+
+ @Override
+ public void close() {
+ }
+ }
+
+ private static class CumulativeDiscountProcessor implements Processor<String, Double, String, String> {
+ private KeyValueStore<String, Double> spendingStore;
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public void process(final Record<String, Double> record) {
+ if (record.value() == null) {
+ return; // Skip null purchase amounts
+ }
+
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(record.key());
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += record.value();
+ spendingStore.put(record.key(), currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(record.key(), currentSpending - DISCOUNT_THRESHOLD);
+ // Send a discount notification
+ context.forward(record.withValue(DISCOUNT_NOTIFICATION_MESSAGE));
+ }
+ }
+ }
+}</code></pre>
+ <h5 id="categorizing-logs-by-severity-removal">Categorizing Logs by Severity</h5>
+ <p>Below, methods <code>categorizeWithFlatTransform</code> and <code>categorizeWithProcess</code> show how you can
+ migrate from <code>flatTransform</code><br>to <code>process</code>.</p>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithFlatTransform(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.flatTransform(LogSeverityTransformer::new)
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
+
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new)
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
+
+ private static class LogSeverityTransformer implements Transformer<String, String, Iterable<KeyValue<String, String>>> {
+ @Override
+ public void init(org.apache.kafka.streams.processor.ProcessorContext context) {
+ }
+
+ @Override
+ public Iterable<KeyValue<String, String>> transform(String key, String value) {
+ if (value == null) {
+ return Collections.emptyList(); // Skip null values
+ }
+
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ int colonIndex = value.indexOf(':');
+ String severity = colonIndex > 0 ? value.substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Create appropriate KeyValue pair based on severity
+ return switch (severity) {
+ case "ERROR" -> List.of(new KeyValue<>("ERROR", value));
+ case "WARN" -> List.of(new KeyValue<>("WARN", value));
+ case "INFO" -> Collections.emptyList(); // INFO logs are ignored
+ default -> List.of(new KeyValue<>("UNKNOWN", value));
+ };
+ }
+
+ @Override
+ public void close() {
+ }
+ }
+
+ private static class LogSeverityProcessor extends ContextualProcessor<String, String, String, String> {
+ @Override
+ public void process(final Record<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ final int colonIndex = record.value().indexOf(':');
+ final String severity = colonIndex > 0 ? record.value().substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Route logs based on severity
+ switch (severity) {
+ case "ERROR":
+ context().forward(record.withKey(ERROR_LOGS_TOPIC));
+ break;
+ case "WARN":
+ context().forward(record.withKey(WARN_LOGS_TOPIC));
+ break;
+ case "INFO":
+ // INFO logs are ignored
+ break;
+ default:
+ // Forward to an "unknown" topic for logs with unrecognized severities
+ context().forward(record.withKey(UNKNOWN_LOGS_TOPIC));
+ }
+ }
+ }
+}</code></pre>
+ <h5 id="traffic-radar-monitoring-car-count-removal">Traffic Radar Monitoring Car Count</h5>
+ <p>Below, methods <code>countWithTransformValues</code> and <code>countWithProcessValues</code> show how you can migrate
+ from <code>transformValues</code><br>to <code>processValues</code>.</p>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.processor.api.FixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessorContext;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.state.Stores;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.format.DateTimeFormatter;
+
+public class TrafficRadarMonitoringCarCountExample {
+ private static final String DAILY_COUNT_STORE = "price-state-store";
+ private static final String DAILY_COUNT_TOPIC = "price-state-topic";
+ private static final String RADAR_COUNT_TOPIC = "car-radar-topic";
+
+ public static void countWithTransformValues(final StreamsBuilder builder) {
+ // Define a state store for tracking daily car counts
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(DAILY_COUNT_STORE),
+ org.apache.kafka.common.serialization.Serdes.String(),
+ org.apache.kafka.common.serialization.Serdes.Long()
+ )
+ );
+ final KStream<Void, String> radarStream = builder.stream(RADAR_COUNT_TOPIC);
+ // Apply the ValueTransformer with the state store
+ radarStream.transformValues(DailyCarCountTransformer::new, DAILY_COUNT_STORE)
+ .to(DAILY_COUNT_TOPIC);
+ }
+
+ public static void countWithProcessValues(final StreamsBuilder builder) {
+ // Define a state store for tracking daily car counts
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(DAILY_COUNT_STORE),
+ org.apache.kafka.common.serialization.Serdes.String(),
+ org.apache.kafka.common.serialization.Serdes.Long()
+ )
+ );
+ final KStream<Void, String> radarStream = builder.stream(RADAR_COUNT_TOPIC);
+ // Apply the FixedKeyProcessor with the state store
+ radarStream.processValues(DailyCarCountProcessor::new, DAILY_COUNT_STORE)
+ .to(DAILY_COUNT_TOPIC);
+ }
+
+ private static class DailyCarCountTransformer implements ValueTransformerWithKey<Void, String, String> {
+ private KeyValueStore<String, Long> stateStore;
+ private static final DateTimeFormatter DATE_FORMATTER =
+ DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(ZoneId.systemDefault());
+
+ @Override
+ public void init(final org.apache.kafka.streams.processor.ProcessorContext context) {
+ // Access the state store
+ stateStore = context.getStateStore(DAILY_COUNT_STORE);
+ }
+
+ @Override
+ public String transform(Void readOnlyKey, String value) {
+ if (value == null) {
+ return null; // Skip null events
+ }
+
+ // Derive the current day from the event timestamp
+ final long timestamp = System.currentTimeMillis(); // Use system time for simplicity
+ final String currentDay = DATE_FORMATTER.format(Instant.ofEpochMilli(timestamp));
+ // Retrieve the current count for the day
+ Long dailyCount = stateStore.get(currentDay);
+ if (dailyCount == null) {
+ dailyCount = 0L;
+ }
+ // Increment the count
+ dailyCount++;
+ stateStore.put(currentDay, dailyCount);
+
+ // Return the current day's count
+ return String.format("Day: %s, Car Count: %s", currentDay, dailyCount);
+ }
+
+ @Override
+ public void close() {
+ }
+ }
+
+ private static class DailyCarCountProcessor implements FixedKeyProcessor<Void, String, String> {
+ private FixedKeyProcessorContext<Void, String> context;
+ private KeyValueStore<String, Long> stateStore;
+ private static final DateTimeFormatter DATE_FORMATTER =
+ DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(ZoneId.systemDefault());
+
+ @Override
+ public void init(final FixedKeyProcessorContext<Void, String> context) {
+ this.context = context;
+ stateStore = context.getStateStore(DAILY_COUNT_STORE);
+ }
+
+ @Override
+ public void process(final FixedKeyRecord<Void, String> record) {
+ if (record.value() == null) {
+ return; // Skip null events
+ }
+
+ // Derive the current day from the event timestamp
+ final long timestamp = System.currentTimeMillis(); // Use system time for simplicity
+ final String currentDay = DATE_FORMATTER.format(Instant.ofEpochMilli(timestamp));
+ // Retrieve the current count for the day
+ Long dailyCount = stateStore.get(currentDay);
+ if (dailyCount == null) {
+ dailyCount = 0L;
+ }
+ // Increment the count
+ dailyCount++;
+ stateStore.put(currentDay, dailyCount);
+
+ // Emit the current day's count
+ context.forward(record.withValue(String.format("Day: %s, Car Count: %s", currentDay, dailyCount)));
+ }
+ }
+}</code></pre>
+ <h5 id="replacing-slang-in-text-messages-removal">Replacing Slang in Text Messages</h5>
+ <p>Below, methods <code>replaceWithFlatTransformValues</code> and <code>replaceWithProcessValues</code> show how you can
+ migrate from<br><code>flatTransformValues</code> to <code>processValues</code>.</p>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualFixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
+
+public class ReplacingSlangTextInMessagesExample {
+ private static final Map<String, String> SLANG_DICTIONARY = Map.of(
+ "u", "you",
+ "brb", "be right back",
+ "omg", "oh my god",
+ "btw", "by the way"
+ );
+ private static final String INPUT_MESSAGES_TOPIC = "input-messages-topic";
+ private static final String OUTPUT_MESSAGES_TOPIC = "output-messages-topic";
+
+ public static void replaceWithFlatTransformValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.flatTransformValues(SlangReplacementTransformer::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ public static void replaceWithProcessValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.processValues(SlangReplacementProcessor::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ private static class SlangReplacementTransformer implements ValueTransformer<String, Iterable<String>> {
+
+ @Override
+ public void init(final org.apache.kafka.streams.processor.ProcessorContext context) {
+ }
+
+ @Override
+ public Iterable<String> transform(final String value) {
+ if (value == null) {
+ return Collections.emptyList(); // Skip null values
+ }
+
+ // Replace slang words in the message
+ final String[] words = value.split("\\s+");
+ return Arrays.asList(
+ Arrays.stream(words)
+ .map(word -> SLANG_DICTIONARY.getOrDefault(word, word))
+ .toArray(String[]::new)
+ );
+ }
+
+ @Override
+ public void close() {
+ }
+ }
+
+ private static class SlangReplacementProcessor extends ContextualFixedKeyProcessor<String, String, String> {
+ @Override
+ public void process(final FixedKeyRecord<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+
+ // Replace slang words in the message
+ final String[] words = record.value().split("\\s+");
+ for (final String word : words) {
+ String replacedWord = SLANG_DICTIONARY.getOrDefault(word, word);
+ context().forward(record.withValue(replacedWord));
+ }
+ }
+ }
+}</code></pre>
+ <h4>Keynotes</h4>
+ <ul>
+ <li><strong>Type Safety and Flexibility:</strong> The process and processValues APIs utilize
+ <code>ProcessorContext</code> and <code>Record</code> or<br><code>FixedKeyRecord</code> objects for better type
+ safety and flexibility of custom processing logic.</li>
+ <li><strong>Clear State and Logic Management:</strong> Implementations for <code>Processor</code> or
+ <code>FixedKeyProcessor</code> should manage state and<br>logic clearly. Use <code>context().forward()</code>
+ for emitting records downstream.</li>
+ <li><strong>Unified API:</strong> Consolidates multiple methods into a single, versatile API.</li>
+ <li><strong>Future-Proof:</strong> Ensures compatibility with the latest Kafka Streams releases.</li>
+ </ul>
+ <h4>Removal of Old <code>process</code> Method</h4>
+ <p>It is worth mentioning that, in addition to the methods mentioned above, the <code>process</code> method (which is
+ not part of the<br>Processor API) has also been removed. The following example shows how to migrate to the new
Review Comment:
```
which is not part of the<br>Processor API
```
Not sure what you mean by this?
Maybe we want to day: "which did integrate the 'old' Processor API (ie,`Processor` in contrast to the new `api.Processor`) into the DSL", or something like this?
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
+ </li>
+ <li><code>ProcessorSupplier</code>: A processor supplier that can create one or more <code>Processor</code>
+ instances; and</li>
+ <li><code>FixedKeyProcessorSupplier</code>: A processor supplier that can create one or more
+ <code>FixedKeyProcessor</code> instances.
+ </li>
+ </ul>
+ <h4>Examples</h4>
+ <p>Follow the examples below to learn how to apply <code>process</code> and <code>processValues</code> to your
+ <code>KStream</code>.
+ </p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Operation</th>
+ <th>State Type</th>
+ </tr>
</thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
</tbody>
</table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <h5 id="cumulative-discounts-for-a-loyalty-program"">Cumulative Discounts for a Loyalty Program</h5>
+ <ul>
+ <li><strong>Idea:</strong> A stream of purchase events contains user IDs and transaction amounts. Use a state
+ store
+ to accumulate the total spending of each user. When their total crosses a threshold, apply a discount on
+ their next transaction and update their accumulated total.</li>
+ <li><strong>Real-World Context:</strong> In a retail loyalty program, tracking cumulative customer spending
+ enables
+ dynamic rewards, such as issuing a discount when a customer's total purchases exceed a predefined limit.
+ </li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+
+public class CumulativeDiscountsForALoyaltyProgramExample {
+ private static final double DISCOUNT_THRESHOLD = 100.0;
+ private static final String CUSTOMER_SPENDING_STORE = "customer-spending-store";
+ private static final String DISCOUNT_NOTIFICATION_MESSAGE =
+ "Discount applied! You have received a reward for your purchases.";
+ private static final String DISCOUNT_NOTIFICATIONS_TOPIC = "discount-notifications-topic";
+ private static final String PURCHASE_EVENTS_TOPIC = "purchase-events-topic";
+
+ public static void applyDiscountWithProcess(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ Serdes.String(),
+ Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Processor with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.process(CumulativeDiscountProcessor::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ private static class CumulativeDiscountProcessor implements Processor<String, Double, String, String> {
+ private KeyValueStore<String, Double> spendingStore;
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public void process(final Record<String, Double> record) {
+ if (record.value() == null) {
+ return; // Skip null purchase amounts
+ }
- @Override
- void process(Record<PageId, Long> record) {
- // Here you would format and send the alert email.
- //
- // In this specific example, you would be able to include
- // information about the page's ID and its view count
- }
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(record.key());
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += record.value();
+ spendingStore.put(record.key(), currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(record.key(), currentSpending - DISCOUNT_THRESHOLD);
+ // Send a discount notification
+ context.forward(record.withValue(DISCOUNT_NOTIFICATION_MESSAGE));
+ }
+ }
+ }
+}</code></pre>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li><strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g.,
+ INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to a dedicated topic too.</li>
+ <li><strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity
+ ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks,
+ and
+ INFO logs are stored for basic reporting purposes.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- void close() {
- // Any code for clean up would go here, for example tearing down the email client and anything
- // else you created in the init() method
- // This processor instance will not be used again after this call.
- }
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new)
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
+
+ private static class LogSeverityProcessor extends ContextualProcessor<String, String, String, String> {
+ @Override
+ public void process(final Record<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ final int colonIndex = record.value().indexOf(':');
+ final String severity = colonIndex > 0 ? record.value().substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Route logs based on severity
+ switch (severity) {
+ case "ERROR":
+ context().forward(record.withKey(ERROR_LOGS_TOPIC));
+ break;
+ case "WARN":
+ context().forward(record.withKey(WARN_LOGS_TOPIC));
+ break;
+ case "INFO":
+ // INFO logs are ignored
+ break;
+ default:
+ // Forward to an "unknown" topic for logs with unrecognized severities
+ context().forward(record.withKey(UNKNOWN_LOGS_TOPIC));
+ }
+ }
+ }
}</code></pre>
- <div class="admonition tip">
- <p><b>Tip</b></p>
- <p class="last">Even though we do not demonstrate it in this example, a stream processor can access any available state stores by
- calling <code class="docutils literal"><span class="pre">ProcessorContext#getStateStore()</span></code>.
- State stores are only available if they have been connected to the processor, or if they are global stores. While global stores do not need to be connected explicitly, they only allow for read-only access.
- There are two ways to connect state stores to a processor:
- <ul class="simple">
- <li>By passing the name of a store that has already been added via <code class="docutils literal"><span class="pre">Topology#addStateStore()</span></code> to the corresponding <code class="docutils literal"><span class="pre">KStream#process()</span></code> method call.</li>
- <li>Implementing <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the <code class="docutils literal"><span class="pre">ProcessorSupplier</span></code>
- passed to <code class="docutils literal"><span class="pre">KStream#process()</span></code>. In this case there is no need to call <code class="docutils literal"><span class="pre">StreamsBuilder#addStateStore()</span></code>
- beforehand, the store will be automatically added for you. You can also implement <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the
- <code class="docutils literal"><span class="pre">Value*</span></code> or <code class="docutils literal"><span class="pre">*WithKey</span></code> supplier variants, or <code class="docutils literal"><span class="pre">TransformerSupplier</span></code> or any of its variants.
- </li>
- </ul>
- </div>
- <p>Then we can leverage the <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code> processor in the DSL via <code class="docutils literal"><span class="pre">KStream#process</span></code>.</p>
- <pre class="line-numbers"><code class="language-java">KStream<String, GenericRecord> pageViews = ...;
-
-// Send an email notification when the view count of a page reaches one thousand.
-pageViews.groupByKey()
- .count()
- .filter((PageId pageId, Long viewCount) -> viewCount == 1000)
- // PopularPageEmailAlert is your custom processor that implements the
- // `Processor` interface, see further down below.
- .process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
- </div>
+ <h5 id="traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</h5>
+ <ul>
+ <li><strong>Idea:</strong> A radar monitors cars passing along a road stretch. A system counts the cars for each
+ day, maintaining a cumulative total for the current day in a state store. At the end of the day, the count
+ is
+ emitted and the state is cleared for the next day.</li>
+ <li><strong>Real-World Context:</strong> A car counting system can be useful for determining measures for
+ widening
+ or controlling traffic depending on the number of cars passing through the monitored stretch.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.common.serialization;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessorContext;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.state.Stores;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.format.DateTimeFormatter;
+
+public class TrafficRadarMonitoringCarCountExample {
+ private static final String DAILY_COUNT_STORE = "price-state-store";
+ private static final String DAILY_COUNT_TOPIC = "price-state-topic";
+ private static final String RADAR_COUNT_TOPIC = "car-radar-topic";
+
+ public static void countWithProcessValues(final StreamsBuilder builder) {
+ // Define a state store for tracking daily car counts
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(DAILY_COUNT_STORE),
+ Serdes.String(),
+ Serdes.Long()
+ )
+ );
+ final KStream<Void, String> radarStream = builder.stream(RADAR_COUNT_TOPIC);
+ // Apply the FixedKeyProcessor with the state store
+ radarStream.processValues(DailyCarCountProcessor::new, DAILY_COUNT_STORE)
+ .to(DAILY_COUNT_TOPIC);
+ }
+
+ private static class DailyCarCountProcessor implements FixedKeyProcessor<Void, String, String> {
+ private FixedKeyProcessorContext<Void, String> context;
+ private KeyValueStore<String, Long> stateStore;
+ private static final DateTimeFormatter DATE_FORMATTER =
+ DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(ZoneId.systemDefault());
+
+ @Override
+ public void init(final FixedKeyProcessorContext<Void, String> context) {
+ this.context = context;
+ stateStore = context.getStateStore(DAILY_COUNT_STORE);
+ }
+
+ @Override
+ public void process(final FixedKeyRecord<Void, String> record) {
+ if (record.value() == null) {
+ return; // Skip null events
+ }
+
+ // Derive the current day from the event timestamp
+ final long timestamp = System.currentTimeMillis(); // Use system time for simplicity
+ final String currentDay = DATE_FORMATTER.format(Instant.ofEpochMilli(timestamp));
+ // Retrieve the current count for the day
+ Long dailyCount = stateStore.get(currentDay);
+ if (dailyCount == null) {
+ dailyCount = 0L;
+ }
+ // Increment the count
+ dailyCount++;
+ stateStore.put(currentDay, dailyCount);
+
+ // Emit the current day's count
+ context.forward(record.withValue(String.format("Day: %s, Car Count: %s", currentDay, dailyCount)));
+ }
+ }
+}</code></pre>
+ <h5 id="replacing-slang-in-text-messages">Replacing Slang in Text Messages</h5>
+ <ul>
+ <li><strong>Idea:</strong> A messaging stream contains user-generated content, and you want to replace slang
+ words
+ with their formal equivalents (e.g., "u" becomes "you", "brb" becomes "be
+ right back"). The operation only modifies the message value and keeps the key intact.</li>
+ <li><strong>Real-World Context:</strong> In customer support chat systems, normalizing text by replacing slang
+ with
+ formal equivalents ensures that automated sentiment analysis tools work accurately and provide reliable
+ insights.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualFixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
+
+public class ReplacingSlangTextInMessagesExample {
+ private static final Map<String, String> SLANG_DICTIONARY = Map.of(
+ "u", "you",
+ "brb", "be right back",
+ "omg", "oh my god",
+ "btw", "by the way"
+ );
+ private static final String INPUT_MESSAGES_TOPIC = "input-messages-topic";
+ private static final String OUTPUT_MESSAGES_TOPIC = "output-messages-topic";
+
+ public static void replaceWithProcessValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.processValues(SlangReplacementProcessor::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ private static class SlangReplacementProcessor extends ContextualFixedKeyProcessor<String, String, String> {
+ @Override
+ public void process(final FixedKeyRecord<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+
+ // Replace slang words in the message
+ final String[] words = record.value().split("\\s+");
+ for (final String word : words) {
+ String replacedWord = SLANG_DICTIONARY.getOrDefault(word, word);
+ context().forward(record.withValue(replacedWord));
+ }
+ }
+ }
+}</code></pre>
+ <h4>Keynotes</h4>
+ <ul>
+ <li><strong>Type Safety and Flexibility:</strong> The process and processValues APIs utilize
+ <code>ProcessorContext</code> and <code>Record</code> or <code>FixedKeyRecord</code> objects for better type
+ safety and flexibility of custom processing logic.
+ </li>
+ <li><strong>Clear State and Logic Management:</strong> Implementations for <code>Processor</code> or
+ <code>FixedKeyProcessor</code> should manage state and logic clearly. Use <code>context().forward()</code>
+ for emitting records downstream.
+ </li>
+ <li><strong>Unified API:</strong> Consolidates multiple methods into a single, versatile API.</li>
+ <li><strong>Future-Proof:</strong> Ensures compatibility with the latest Kafka Streams releases.</li>
+ </ul>
+ </div>
+ <div class="section" id="transformers-removal-and-migration-to-processors">
+ <a class="headerlink" href="#transformers-removal-and-migration-to-processors" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id37">Transformers removal and migration to processors</a>
+ </h3>
+ </a>
+ <p>As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>,<br><code>transformValues</code>, <code>flatTransformValues</code>, and
+ <code>process</code> have been removed. These methods have been replaced with the<br>more versatile Processor API.
+ This guide provides detailed steps for migrating existing code to use the new Processor<br>API and explains the
+ benefits of the changes.</p>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ <li><code>KStream#process</code></li>
+ </ul>
+ <p>The Processor API now serves as a unified replacement for all these methods. It simplifies the API surface
+ while<br>maintaining support for both stateless and stateful operations.</p>
+ <h4>Migration Examples</h4>
+ <p>To migrate from the deprecated <code>transform</code>, <code>transformValues</code>, <code>flatTransform</code>, and
+ <code>flatTransformValues</code> methods to the<br>Processor API (PAPI) in Kafka Streams, let's resume the
+ previouss examples. The new <code>process</code> and <code>processValues</code><br>methods enable a more flexible
+ and reusable approach by requiring implementations of the <code>Processor</code>
+ or<br><code>FixedKeyProcessor</code> interfaces.</p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Migrating from</th>
+ <th>Migrating to</th>
+ <th>State Type</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program-removal">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>transform</code></td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity-removal">Categorizing Logs by Severity</a></td>
+ <td><code>flatTransform</code></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count-removal">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>transformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages-removal">Replacing Slang in Text Messages</a></td>
+ <td><code>flatTransformValues</code></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
+ </tbody>
+ </table>
+ <h5 id="cumulative-discounts-for-a-loyalty-program-removal">Cumulative Discounts for a Loyalty Program</h5>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+
+public class CumulativeDiscountsForALoyaltyProgramExample {
+ private static final double DISCOUNT_THRESHOLD = 100.0;
+ private static final String CUSTOMER_SPENDING_STORE = "customer-spending-store";
+ private static final String DISCOUNT_NOTIFICATION_MESSAGE =
+ "Discount applied! You have received a reward for your purchases.";
+ private static final String DISCOUNT_NOTIFICATIONS_TOPIC = "discount-notifications-topic";
+ private static final String PURCHASE_EVENTS_TOPIC = "purchase-events-topic";
+
+ public static void applyDiscountWithTransform(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ Serdes.String(),
+ Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Transformer with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.transform(CumulativeDiscountTransformer::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
+
+ public static void applyDiscountWithProcess(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ org.apache.kafka.common.serialization.Serdes.String(),
+ org.apache.kafka.common.serialization.Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Processor with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.process(CumulativeDiscountProcessor::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
+
+ private static class CumulativeDiscountTransformer implements Transformer<String, Double, KeyValue<String, String>> {
+ private KeyValueStore<String, Double> spendingStore;
+
+ @Override
+ public void init(final org.apache.kafka.streams.processor.ProcessorContext context) {
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public KeyValue<String, String> transform(final String key, final Double value) {
+ if (value == null) {
+ return null; // Skip null purchase amounts
+ }
+
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(key);
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += value;
+ spendingStore.put(key, currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(key, currentSpending - DISCOUNT_THRESHOLD);
+ // Return a notification message
+ return new KeyValue<>(key, DISCOUNT_NOTIFICATION_MESSAGE);
+ }
+ return null; // No discount, so no output for this record
+ }
+
+ @Override
+ public void close() {
+ }
+ }
+
+ private static class CumulativeDiscountProcessor implements Processor<String, Double, String, String> {
+ private KeyValueStore<String, Double> spendingStore;
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public void process(final Record<String, Double> record) {
+ if (record.value() == null) {
+ return; // Skip null purchase amounts
+ }
+
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(record.key());
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += record.value();
+ spendingStore.put(record.key(), currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(record.key(), currentSpending - DISCOUNT_THRESHOLD);
+ // Send a discount notification
+ context.forward(record.withValue(DISCOUNT_NOTIFICATION_MESSAGE));
+ }
+ }
+ }
+}</code></pre>
+ <h5 id="categorizing-logs-by-severity-removal">Categorizing Logs by Severity</h5>
+ <p>Below, methods <code>categorizeWithFlatTransform</code> and <code>categorizeWithProcess</code> show how you can
+ migrate from <code>flatTransform</code><br>to <code>process</code>.</p>
Review Comment:
Another `<br>` tag... Seems there is 18 overall that needs to get replace with a space
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
+ </li>
+ <li><code>ProcessorSupplier</code>: A processor supplier that can create one or more <code>Processor</code>
+ instances; and</li>
+ <li><code>FixedKeyProcessorSupplier</code>: A processor supplier that can create one or more
+ <code>FixedKeyProcessor</code> instances.
+ </li>
+ </ul>
+ <h4>Examples</h4>
+ <p>Follow the examples below to learn how to apply <code>process</code> and <code>processValues</code> to your
+ <code>KStream</code>.
+ </p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Operation</th>
+ <th>State Type</th>
+ </tr>
</thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
</tbody>
</table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <h5 id="cumulative-discounts-for-a-loyalty-program"">Cumulative Discounts for a Loyalty Program</h5>
+ <ul>
+ <li><strong>Idea:</strong> A stream of purchase events contains user IDs and transaction amounts. Use a state
+ store
+ to accumulate the total spending of each user. When their total crosses a threshold, apply a discount on
+ their next transaction and update their accumulated total.</li>
+ <li><strong>Real-World Context:</strong> In a retail loyalty program, tracking cumulative customer spending
+ enables
+ dynamic rewards, such as issuing a discount when a customer's total purchases exceed a predefined limit.
+ </li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+
+public class CumulativeDiscountsForALoyaltyProgramExample {
+ private static final double DISCOUNT_THRESHOLD = 100.0;
+ private static final String CUSTOMER_SPENDING_STORE = "customer-spending-store";
+ private static final String DISCOUNT_NOTIFICATION_MESSAGE =
+ "Discount applied! You have received a reward for your purchases.";
+ private static final String DISCOUNT_NOTIFICATIONS_TOPIC = "discount-notifications-topic";
+ private static final String PURCHASE_EVENTS_TOPIC = "purchase-events-topic";
+
+ public static void applyDiscountWithProcess(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ Serdes.String(),
+ Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Processor with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.process(CumulativeDiscountProcessor::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ private static class CumulativeDiscountProcessor implements Processor<String, Double, String, String> {
+ private KeyValueStore<String, Double> spendingStore;
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public void process(final Record<String, Double> record) {
+ if (record.value() == null) {
+ return; // Skip null purchase amounts
+ }
- @Override
- void process(Record<PageId, Long> record) {
- // Here you would format and send the alert email.
- //
- // In this specific example, you would be able to include
- // information about the page's ID and its view count
- }
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(record.key());
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += record.value();
+ spendingStore.put(record.key(), currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(record.key(), currentSpending - DISCOUNT_THRESHOLD);
+ // Send a discount notification
+ context.forward(record.withValue(DISCOUNT_NOTIFICATION_MESSAGE));
+ }
+ }
+ }
+}</code></pre>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li><strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g.,
+ INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to a dedicated topic too.</li>
+ <li><strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity
+ ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks,
+ and
+ INFO logs are stored for basic reporting purposes.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- void close() {
- // Any code for clean up would go here, for example tearing down the email client and anything
- // else you created in the init() method
- // This processor instance will not be used again after this call.
- }
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new)
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
+
+ private static class LogSeverityProcessor extends ContextualProcessor<String, String, String, String> {
+ @Override
+ public void process(final Record<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ final int colonIndex = record.value().indexOf(':');
+ final String severity = colonIndex > 0 ? record.value().substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Route logs based on severity
+ switch (severity) {
+ case "ERROR":
+ context().forward(record.withKey(ERROR_LOGS_TOPIC));
+ break;
+ case "WARN":
+ context().forward(record.withKey(WARN_LOGS_TOPIC));
+ break;
+ case "INFO":
+ // INFO logs are ignored
+ break;
+ default:
+ // Forward to an "unknown" topic for logs with unrecognized severities
+ context().forward(record.withKey(UNKNOWN_LOGS_TOPIC));
+ }
+ }
+ }
}</code></pre>
- <div class="admonition tip">
- <p><b>Tip</b></p>
- <p class="last">Even though we do not demonstrate it in this example, a stream processor can access any available state stores by
- calling <code class="docutils literal"><span class="pre">ProcessorContext#getStateStore()</span></code>.
- State stores are only available if they have been connected to the processor, or if they are global stores. While global stores do not need to be connected explicitly, they only allow for read-only access.
- There are two ways to connect state stores to a processor:
- <ul class="simple">
- <li>By passing the name of a store that has already been added via <code class="docutils literal"><span class="pre">Topology#addStateStore()</span></code> to the corresponding <code class="docutils literal"><span class="pre">KStream#process()</span></code> method call.</li>
- <li>Implementing <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the <code class="docutils literal"><span class="pre">ProcessorSupplier</span></code>
- passed to <code class="docutils literal"><span class="pre">KStream#process()</span></code>. In this case there is no need to call <code class="docutils literal"><span class="pre">StreamsBuilder#addStateStore()</span></code>
- beforehand, the store will be automatically added for you. You can also implement <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the
- <code class="docutils literal"><span class="pre">Value*</span></code> or <code class="docutils literal"><span class="pre">*WithKey</span></code> supplier variants, or <code class="docutils literal"><span class="pre">TransformerSupplier</span></code> or any of its variants.
- </li>
- </ul>
- </div>
- <p>Then we can leverage the <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code> processor in the DSL via <code class="docutils literal"><span class="pre">KStream#process</span></code>.</p>
- <pre class="line-numbers"><code class="language-java">KStream<String, GenericRecord> pageViews = ...;
-
-// Send an email notification when the view count of a page reaches one thousand.
-pageViews.groupByKey()
- .count()
- .filter((PageId pageId, Long viewCount) -> viewCount == 1000)
- // PopularPageEmailAlert is your custom processor that implements the
- // `Processor` interface, see further down below.
- .process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
- </div>
+ <h5 id="traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</h5>
+ <ul>
+ <li><strong>Idea:</strong> A radar monitors cars passing along a road stretch. A system counts the cars for each
+ day, maintaining a cumulative total for the current day in a state store. At the end of the day, the count
+ is
+ emitted and the state is cleared for the next day.</li>
+ <li><strong>Real-World Context:</strong> A car counting system can be useful for determining measures for
+ widening
+ or controlling traffic depending on the number of cars passing through the monitored stretch.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.common.serialization;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessorContext;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.state.Stores;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.format.DateTimeFormatter;
+
+public class TrafficRadarMonitoringCarCountExample {
+ private static final String DAILY_COUNT_STORE = "price-state-store";
+ private static final String DAILY_COUNT_TOPIC = "price-state-topic";
+ private static final String RADAR_COUNT_TOPIC = "car-radar-topic";
+
+ public static void countWithProcessValues(final StreamsBuilder builder) {
+ // Define a state store for tracking daily car counts
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(DAILY_COUNT_STORE),
+ Serdes.String(),
+ Serdes.Long()
+ )
+ );
+ final KStream<Void, String> radarStream = builder.stream(RADAR_COUNT_TOPIC);
+ // Apply the FixedKeyProcessor with the state store
+ radarStream.processValues(DailyCarCountProcessor::new, DAILY_COUNT_STORE)
+ .to(DAILY_COUNT_TOPIC);
+ }
+
+ private static class DailyCarCountProcessor implements FixedKeyProcessor<Void, String, String> {
+ private FixedKeyProcessorContext<Void, String> context;
+ private KeyValueStore<String, Long> stateStore;
+ private static final DateTimeFormatter DATE_FORMATTER =
+ DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(ZoneId.systemDefault());
+
+ @Override
+ public void init(final FixedKeyProcessorContext<Void, String> context) {
+ this.context = context;
+ stateStore = context.getStateStore(DAILY_COUNT_STORE);
+ }
+
+ @Override
+ public void process(final FixedKeyRecord<Void, String> record) {
+ if (record.value() == null) {
+ return; // Skip null events
+ }
+
+ // Derive the current day from the event timestamp
+ final long timestamp = System.currentTimeMillis(); // Use system time for simplicity
+ final String currentDay = DATE_FORMATTER.format(Instant.ofEpochMilli(timestamp));
+ // Retrieve the current count for the day
+ Long dailyCount = stateStore.get(currentDay);
+ if (dailyCount == null) {
+ dailyCount = 0L;
+ }
+ // Increment the count
+ dailyCount++;
+ stateStore.put(currentDay, dailyCount);
+
+ // Emit the current day's count
+ context.forward(record.withValue(String.format("Day: %s, Car Count: %s", currentDay, dailyCount)));
+ }
+ }
+}</code></pre>
+ <h5 id="replacing-slang-in-text-messages">Replacing Slang in Text Messages</h5>
+ <ul>
+ <li><strong>Idea:</strong> A messaging stream contains user-generated content, and you want to replace slang
+ words
+ with their formal equivalents (e.g., "u" becomes "you", "brb" becomes "be
+ right back"). The operation only modifies the message value and keeps the key intact.</li>
+ <li><strong>Real-World Context:</strong> In customer support chat systems, normalizing text by replacing slang
+ with
+ formal equivalents ensures that automated sentiment analysis tools work accurately and provide reliable
+ insights.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualFixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
+
+public class ReplacingSlangTextInMessagesExample {
+ private static final Map<String, String> SLANG_DICTIONARY = Map.of(
+ "u", "you",
+ "brb", "be right back",
+ "omg", "oh my god",
+ "btw", "by the way"
+ );
+ private static final String INPUT_MESSAGES_TOPIC = "input-messages-topic";
+ private static final String OUTPUT_MESSAGES_TOPIC = "output-messages-topic";
+
+ public static void replaceWithProcessValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.processValues(SlangReplacementProcessor::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ private static class SlangReplacementProcessor extends ContextualFixedKeyProcessor<String, String, String> {
+ @Override
+ public void process(final FixedKeyRecord<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+
+ // Replace slang words in the message
+ final String[] words = record.value().split("\\s+");
+ for (final String word : words) {
+ String replacedWord = SLANG_DICTIONARY.getOrDefault(word, word);
+ context().forward(record.withValue(replacedWord));
+ }
+ }
+ }
+}</code></pre>
+ <h4>Keynotes</h4>
+ <ul>
+ <li><strong>Type Safety and Flexibility:</strong> The process and processValues APIs utilize
+ <code>ProcessorContext</code> and <code>Record</code> or <code>FixedKeyRecord</code> objects for better type
+ safety and flexibility of custom processing logic.
+ </li>
+ <li><strong>Clear State and Logic Management:</strong> Implementations for <code>Processor</code> or
+ <code>FixedKeyProcessor</code> should manage state and logic clearly. Use <code>context().forward()</code>
+ for emitting records downstream.
+ </li>
+ <li><strong>Unified API:</strong> Consolidates multiple methods into a single, versatile API.</li>
+ <li><strong>Future-Proof:</strong> Ensures compatibility with the latest Kafka Streams releases.</li>
+ </ul>
+ </div>
+ <div class="section" id="transformers-removal-and-migration-to-processors">
+ <a class="headerlink" href="#transformers-removal-and-migration-to-processors" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id37">Transformers removal and migration to processors</a>
+ </h3>
+ </a>
+ <p>As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>,<br><code>transformValues</code>, <code>flatTransformValues</code>, and
+ <code>process</code> have been removed. These methods have been replaced with the<br>more versatile Processor API.
+ This guide provides detailed steps for migrating existing code to use the new Processor<br>API and explains the
+ benefits of the changes.</p>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ <li><code>KStream#process</code></li>
+ </ul>
+ <p>The Processor API now serves as a unified replacement for all these methods. It simplifies the API surface
+ while<br>maintaining support for both stateless and stateful operations.</p>
+ <h4>Migration Examples</h4>
+ <p>To migrate from the deprecated <code>transform</code>, <code>transformValues</code>, <code>flatTransform</code>, and
+ <code>flatTransformValues</code> methods to the<br>Processor API (PAPI) in Kafka Streams, let's resume the
+ previouss examples. The new <code>process</code> and <code>processValues</code><br>methods enable a more flexible
+ and reusable approach by requiring implementations of the <code>Processor</code>
+ or<br><code>FixedKeyProcessor</code> interfaces.</p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Migrating from</th>
+ <th>Migrating to</th>
+ <th>State Type</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program-removal">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>transform</code></td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
Review Comment:
Should we start with stateless?
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
+ </li>
+ <li><code>FixedKeyProcessor</code>: A processor of key-value pair records where keys are immutable;</li>
+ <li><code>ContextualFixedKeyProcessor</code>: An abstract implementation of <code>FixedKeyProcessor</code> that
+ manages the <code>FixedKeyProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>;
+ </li>
+ <li><code>ProcessorSupplier</code>: A processor supplier that can create one or more <code>Processor</code>
+ instances; and</li>
+ <li><code>FixedKeyProcessorSupplier</code>: A processor supplier that can create one or more
+ <code>FixedKeyProcessor</code> instances.
+ </li>
+ </ul>
+ <h4>Examples</h4>
+ <p>Follow the examples below to learn how to apply <code>process</code> and <code>processValues</code> to your
+ <code>KStream</code>.
+ </p>
+ <table>
+ <thead>
+ <tr>
+ <th>Example</th>
+ <th>Operation</th>
+ <th>State Type</th>
+ </tr>
</thead>
- <tbody valign="top">
- <tr class="row-even"><td><p class="first"><strong>Process</strong></p>
- <ul class="last simple">
- <li>KStream -> void</li>
- </ul>
- </td>
- <td><p class="first"><strong>Terminal operation.</strong> Applies a <code class="docutils literal"><span class="pre">Processor</span></code> to each record.
- <code class="docutils literal"><span class="pre">process()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">details</a>)</p>
- <p>This is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Processor</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#process-org.apache.kafka.streams.processor.ProcessorSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
- <tr class="row-odd"><td><p class="first"><strong>Transform</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">Transformer</span></code> to each record.
- <code class="docutils literal"><span class="pre">transform()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into zero, one, or more output records (similar to the stateless <code class="docutils literal"><span class="pre">flatMap</span></code>).
- The <code class="docutils literal"><span class="pre">Transformer</span></code> must return <code class="docutils literal"><span class="pre">null</span></code> for zero output.
- You can modify the record’s key and value, including their types.</p>
- <p><strong>Marks the stream for data re-partitioning:</strong>
- Applying a grouping or a join after <code class="docutils literal"><span class="pre">transform</span></code> will result in re-partitioning of the records.
- If possible use <code class="docutils literal"><span class="pre">transformValues</span></code> instead, which will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transform</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">Transformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transform-org.apache.kafka.streams.kstream.TransformerSupplier-java.lang.String...-">javadocs</a>.
- </p>
- </td>
- </tr>
- <tr class="row-even"><td><p class="first"><strong>Transform (values only)</strong></p>
- <ul class="last simple">
- <li>KStream -> KStream</li>
- <li>KTable -> KTable</li>
- </ul>
- </td>
- <td><p class="first">Applies a <code class="docutils literal"><span class="pre">ValueTransformer</span></code> to each record, while retaining the key of the original record.
- <code class="docutils literal"><span class="pre">transformValues()</span></code> allows you to leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- (<a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">details</a>)</p>
- <p>Each input record is transformed into exactly one output record (zero output records or multiple output records are not possible).
- The <code class="docutils literal"><span class="pre">ValueTransformer</span></code> may return <code class="docutils literal"><span class="pre">null</span></code> as the new value for a record.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is preferable to <code class="docutils literal"><span class="pre">transform</span></code> because it will not cause data re-partitioning.</p>
- <p><code class="docutils literal"><span class="pre">transformValues</span></code> is essentially equivalent to adding the <code class="docutils literal"><span class="pre">ValueTransformer</span></code> via <code class="docutils literal"><span class="pre">Topology#addProcessor()</span></code> to your
- <a class="reference internal" href="../core-concepts.html#streams_topology"><span class="std std-ref">processor topology</span></a>.</p>
- <p class="last">An example is available in the
- <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-">javadocs</a>.</p>
- </td>
- </tr>
+ <tbody>
+ <tr>
+ <td><a href="#cumulative-discounts-for-a-loyalty-program">Cumulative Discounts for a Loyalty Program</a>
+ </td>
+ <td><code>process</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#categorizing-logs-by-severity">Categorizing Logs by Severity</a></td>
+ <td><code>process</code></td>
+ <td>Stateless</td>
+ </tr>
+ <tr>
+ <td><a href="#traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateful</td>
+ </tr>
+ <tr>
+ <td><a href="#replacing-slang-in-text-messages">Replacing Slang in Text Messages</a></td>
+ <td><code>processValues</code></td>
+ <td>Stateless</td>
+ </tr>
</tbody>
</table>
- <p>The following example shows how to leverage, via the <code class="docutils literal"><span class="pre">KStream#process()</span></code> method, a custom <code class="docutils literal"><span class="pre">Processor</span></code> that sends an
- email notification whenever a page view count reaches a predefined threshold.</p>
- <p>First, we need to implement a custom stream processor, <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code>, that implements the <code class="docutils literal"><span class="pre">Processor</span></code>
- interface:</p>
- <pre class="line-numbers"><code class="language-java">// A processor that sends an alert message about a popular page to a configurable email address
-public class PopularPageEmailAlert implements Processor<PageId, Long, Void, Void> {
-
- private final String emailAddress;
- private ProcessorContext<Void, Void> context;
-
- public PopularPageEmailAlert(String emailAddress) {
- this.emailAddress = emailAddress;
- }
+ <h5 id="cumulative-discounts-for-a-loyalty-program"">Cumulative Discounts for a Loyalty Program</h5>
+ <ul>
+ <li><strong>Idea:</strong> A stream of purchase events contains user IDs and transaction amounts. Use a state
+ store
+ to accumulate the total spending of each user. When their total crosses a threshold, apply a discount on
+ their next transaction and update their accumulated total.</li>
+ <li><strong>Real-World Context:</strong> In a retail loyalty program, tracking cumulative customer spending
+ enables
+ dynamic rewards, such as issuing a discount when a customer's total purchases exceed a predefined limit.
+ </li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- public void init(ProcessorContext<Void, Void> context) {
- this.context = context;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+
+public class CumulativeDiscountsForALoyaltyProgramExample {
+ private static final double DISCOUNT_THRESHOLD = 100.0;
+ private static final String CUSTOMER_SPENDING_STORE = "customer-spending-store";
+ private static final String DISCOUNT_NOTIFICATION_MESSAGE =
+ "Discount applied! You have received a reward for your purchases.";
+ private static final String DISCOUNT_NOTIFICATIONS_TOPIC = "discount-notifications-topic";
+ private static final String PURCHASE_EVENTS_TOPIC = "purchase-events-topic";
+
+ public static void applyDiscountWithProcess(final StreamsBuilder builder) {
+ // Define the state store for tracking cumulative spending
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(CUSTOMER_SPENDING_STORE),
+ Serdes.String(),
+ Serdes.Double()
+ )
+ );
+ final KStream<String, Double> purchaseStream = builder.stream(PURCHASE_EVENTS_TOPIC);
+ // Apply the Processor with the state store
+ final KStream<String, String> notificationStream =
+ purchaseStream.process(CumulativeDiscountProcessor::new, CUSTOMER_SPENDING_STORE);
+ // Send the notifications to the output topic
+ notificationStream.to(DISCOUNT_NOTIFICATIONS_TOPIC);
+ }
- // Here you would perform any additional initializations such as setting up an email client.
- }
+ private static class CumulativeDiscountProcessor implements Processor<String, Double, String, String> {
+ private KeyValueStore<String, Double> spendingStore;
+ private ProcessorContext<String, String> context;
+
+ @Override
+ public void init(final ProcessorContext<String, String> context) {
+ this.context = context;
+ // Retrieve the state store for cumulative spending
+ spendingStore = context.getStateStore(CUSTOMER_SPENDING_STORE);
+ }
+
+ @Override
+ public void process(final Record<String, Double> record) {
+ if (record.value() == null) {
+ return; // Skip null purchase amounts
+ }
- @Override
- void process(Record<PageId, Long> record) {
- // Here you would format and send the alert email.
- //
- // In this specific example, you would be able to include
- // information about the page's ID and its view count
- }
+ // Get the current spending total for the customer
+ Double currentSpending = spendingStore.get(record.key());
+ if (currentSpending == null) {
+ currentSpending = 0.0;
+ }
+ // Update the cumulative spending
+ currentSpending += record.value();
+ spendingStore.put(record.key(), currentSpending);
+
+ // Check if the customer qualifies for a discount
+ if (currentSpending >= DISCOUNT_THRESHOLD) {
+ // Reset the spending after applying the discount
+ spendingStore.put(record.key(), currentSpending - DISCOUNT_THRESHOLD);
+ // Send a discount notification
+ context.forward(record.withValue(DISCOUNT_NOTIFICATION_MESSAGE));
+ }
+ }
+ }
+}</code></pre>
+ <h5 id="categorizing-logs-by-severity">Categorizing Logs by Severity</h5>
+ <ul>
+ <li><strong>Idea:</strong> You have a stream of log messages. Each message contains a severity level (e.g.,
+ INFO,
+ WARN, ERROR) in the value. The processor filters messages, routing ERROR messages to a dedicated topic and
+ discarding INFO messages. The rest (WARN) are forwarded to a dedicated topic too.</li>
+ <li><strong>Real-World Context:</strong> In a production monitoring system, categorizing logs by severity
+ ensures
+ ERROR logs are sent to a critical incident management system, WARN logs are analyzed for potential risks,
+ and
+ INFO logs are stored for basic reporting purposes.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
- @Override
- void close() {
- // Any code for clean up would go here, for example tearing down the email client and anything
- // else you created in the init() method
- // This processor instance will not be used again after this call.
- }
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Collections;
+import java.util.List;
+
+public class CategorizingLogsBySeverityExample {
+ private static final String ERROR_LOGS_TOPIC = "error-logs-topic";
+ private static final String INPUT_LOGS_TOPIC = "input-logs-topic";
+ private static final String UNKNOWN_LOGS_TOPIC = "unknown-logs-topic";
+ private static final String WARN_LOGS_TOPIC = "warn-logs-topic";
+
+ public static void categorizeWithProcess(final StreamsBuilder builder) {
+ final KStream<String, String> logStream = builder.stream(INPUT_LOGS_TOPIC);
+ logStream.process(LogSeverityProcessor::new)
+ .to((key, value, recordContext) -> {
+ // Determine the target topic dynamically
+ if ("ERROR".equals(key)) return ERROR_LOGS_TOPIC;
+ if ("WARN".equals(key)) return WARN_LOGS_TOPIC;
+ return UNKNOWN_LOGS_TOPIC;
+ });
+ }
+
+ private static class LogSeverityProcessor extends ContextualProcessor<String, String, String, String> {
+ @Override
+ public void process(final Record<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+ // Assume the severity is the first word in the log message
+ // For example: "ERROR: Disk not found" -> "ERROR"
+ final int colonIndex = record.value().indexOf(':');
+ final String severity = colonIndex > 0 ? record.value().substring(0, colonIndex).trim() : "UNKNOWN";
+
+ // Route logs based on severity
+ switch (severity) {
+ case "ERROR":
+ context().forward(record.withKey(ERROR_LOGS_TOPIC));
+ break;
+ case "WARN":
+ context().forward(record.withKey(WARN_LOGS_TOPIC));
+ break;
+ case "INFO":
+ // INFO logs are ignored
+ break;
+ default:
+ // Forward to an "unknown" topic for logs with unrecognized severities
+ context().forward(record.withKey(UNKNOWN_LOGS_TOPIC));
+ }
+ }
+ }
}</code></pre>
- <div class="admonition tip">
- <p><b>Tip</b></p>
- <p class="last">Even though we do not demonstrate it in this example, a stream processor can access any available state stores by
- calling <code class="docutils literal"><span class="pre">ProcessorContext#getStateStore()</span></code>.
- State stores are only available if they have been connected to the processor, or if they are global stores. While global stores do not need to be connected explicitly, they only allow for read-only access.
- There are two ways to connect state stores to a processor:
- <ul class="simple">
- <li>By passing the name of a store that has already been added via <code class="docutils literal"><span class="pre">Topology#addStateStore()</span></code> to the corresponding <code class="docutils literal"><span class="pre">KStream#process()</span></code> method call.</li>
- <li>Implementing <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the <code class="docutils literal"><span class="pre">ProcessorSupplier</span></code>
- passed to <code class="docutils literal"><span class="pre">KStream#process()</span></code>. In this case there is no need to call <code class="docutils literal"><span class="pre">StreamsBuilder#addStateStore()</span></code>
- beforehand, the store will be automatically added for you. You can also implement <code class="docutils literal"><span class="pre">ConnectedStoreProvider#stores()</span></code> on the
- <code class="docutils literal"><span class="pre">Value*</span></code> or <code class="docutils literal"><span class="pre">*WithKey</span></code> supplier variants, or <code class="docutils literal"><span class="pre">TransformerSupplier</span></code> or any of its variants.
- </li>
- </ul>
- </div>
- <p>Then we can leverage the <code class="docutils literal"><span class="pre">PopularPageEmailAlert</span></code> processor in the DSL via <code class="docutils literal"><span class="pre">KStream#process</span></code>.</p>
- <pre class="line-numbers"><code class="language-java">KStream<String, GenericRecord> pageViews = ...;
-
-// Send an email notification when the view count of a page reaches one thousand.
-pageViews.groupByKey()
- .count()
- .filter((PageId pageId, Long viewCount) -> viewCount == 1000)
- // PopularPageEmailAlert is your custom processor that implements the
- // `Processor` interface, see further down below.
- .process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
- </div>
+ <h5 id="traffic-radar-monitoring-car-count">Traffic Radar Monitoring Car Count</h5>
+ <ul>
+ <li><strong>Idea:</strong> A radar monitors cars passing along a road stretch. A system counts the cars for each
+ day, maintaining a cumulative total for the current day in a state store. At the end of the day, the count
+ is
+ emitted and the state is cleared for the next day.</li>
+ <li><strong>Real-World Context:</strong> A car counting system can be useful for determining measures for
+ widening
+ or controlling traffic depending on the number of cars passing through the monitored stretch.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.common.serialization;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyProcessorContext;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.state.Stores;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.format.DateTimeFormatter;
+
+public class TrafficRadarMonitoringCarCountExample {
+ private static final String DAILY_COUNT_STORE = "price-state-store";
+ private static final String DAILY_COUNT_TOPIC = "price-state-topic";
+ private static final String RADAR_COUNT_TOPIC = "car-radar-topic";
+
+ public static void countWithProcessValues(final StreamsBuilder builder) {
+ // Define a state store for tracking daily car counts
+ builder.addStateStore(
+ Stores.keyValueStoreBuilder(
+ Stores.inMemoryKeyValueStore(DAILY_COUNT_STORE),
+ Serdes.String(),
+ Serdes.Long()
+ )
+ );
+ final KStream<Void, String> radarStream = builder.stream(RADAR_COUNT_TOPIC);
+ // Apply the FixedKeyProcessor with the state store
+ radarStream.processValues(DailyCarCountProcessor::new, DAILY_COUNT_STORE)
+ .to(DAILY_COUNT_TOPIC);
+ }
+
+ private static class DailyCarCountProcessor implements FixedKeyProcessor<Void, String, String> {
+ private FixedKeyProcessorContext<Void, String> context;
+ private KeyValueStore<String, Long> stateStore;
+ private static final DateTimeFormatter DATE_FORMATTER =
+ DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(ZoneId.systemDefault());
+
+ @Override
+ public void init(final FixedKeyProcessorContext<Void, String> context) {
+ this.context = context;
+ stateStore = context.getStateStore(DAILY_COUNT_STORE);
+ }
+
+ @Override
+ public void process(final FixedKeyRecord<Void, String> record) {
+ if (record.value() == null) {
+ return; // Skip null events
+ }
+
+ // Derive the current day from the event timestamp
+ final long timestamp = System.currentTimeMillis(); // Use system time for simplicity
+ final String currentDay = DATE_FORMATTER.format(Instant.ofEpochMilli(timestamp));
+ // Retrieve the current count for the day
+ Long dailyCount = stateStore.get(currentDay);
+ if (dailyCount == null) {
+ dailyCount = 0L;
+ }
+ // Increment the count
+ dailyCount++;
+ stateStore.put(currentDay, dailyCount);
+
+ // Emit the current day's count
+ context.forward(record.withValue(String.format("Day: %s, Car Count: %s", currentDay, dailyCount)));
+ }
+ }
+}</code></pre>
+ <h5 id="replacing-slang-in-text-messages">Replacing Slang in Text Messages</h5>
+ <ul>
+ <li><strong>Idea:</strong> A messaging stream contains user-generated content, and you want to replace slang
+ words
+ with their formal equivalents (e.g., "u" becomes "you", "brb" becomes "be
+ right back"). The operation only modifies the message value and keeps the key intact.</li>
+ <li><strong>Real-World Context:</strong> In customer support chat systems, normalizing text by replacing slang
+ with
+ formal equivalents ensures that automated sentiment analysis tools work accurately and provide reliable
+ insights.</li>
+ </ul>
+ <pre class="line-numbers"><code class="language-java">package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.processor.api.ContextualFixedKeyProcessor;
+import org.apache.kafka.streams.processor.api.FixedKeyRecord;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
+
+public class ReplacingSlangTextInMessagesExample {
+ private static final Map<String, String> SLANG_DICTIONARY = Map.of(
+ "u", "you",
+ "brb", "be right back",
+ "omg", "oh my god",
+ "btw", "by the way"
+ );
+ private static final String INPUT_MESSAGES_TOPIC = "input-messages-topic";
+ private static final String OUTPUT_MESSAGES_TOPIC = "output-messages-topic";
+
+ public static void replaceWithProcessValues(final StreamsBuilder builder) {
+ KStream<String, String> messageStream = builder.stream(INPUT_MESSAGES_TOPIC);
+ messageStream.processValues(SlangReplacementProcessor::new).to(OUTPUT_MESSAGES_TOPIC);
+ }
+
+ private static class SlangReplacementProcessor extends ContextualFixedKeyProcessor<String, String, String> {
+ @Override
+ public void process(final FixedKeyRecord<String, String> record) {
+ if (record.value() == null) {
+ return; // Skip null values
+ }
+
+ // Replace slang words in the message
+ final String[] words = record.value().split("\\s+");
+ for (final String word : words) {
+ String replacedWord = SLANG_DICTIONARY.getOrDefault(word, word);
+ context().forward(record.withValue(replacedWord));
+ }
+ }
+ }
+}</code></pre>
+ <h4>Keynotes</h4>
+ <ul>
+ <li><strong>Type Safety and Flexibility:</strong> The process and processValues APIs utilize
+ <code>ProcessorContext</code> and <code>Record</code> or <code>FixedKeyRecord</code> objects for better type
+ safety and flexibility of custom processing logic.
+ </li>
+ <li><strong>Clear State and Logic Management:</strong> Implementations for <code>Processor</code> or
+ <code>FixedKeyProcessor</code> should manage state and logic clearly. Use <code>context().forward()</code>
+ for emitting records downstream.
+ </li>
+ <li><strong>Unified API:</strong> Consolidates multiple methods into a single, versatile API.</li>
+ <li><strong>Future-Proof:</strong> Ensures compatibility with the latest Kafka Streams releases.</li>
+ </ul>
+ </div>
+ <div class="section" id="transformers-removal-and-migration-to-processors">
+ <a class="headerlink" href="#transformers-removal-and-migration-to-processors" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id37">Transformers removal and migration to processors</a>
+ </h3>
+ </a>
+ <p>As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>,<br><code>transformValues</code>, <code>flatTransformValues</code>, and
Review Comment:
There is more `<br>` tags below -- seems they should all be replace why a space?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on code in PR #18314:
URL: https://github.com/apache/kafka/pull/18314#discussion_r1933744549
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3097,152 +3098,994 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
</div>
- <div class="section" id="applying-processors-and-transformers-processor-api-integration">
- <span id="streams-developer-guide-dsl-process"></span><h3><a class="toc-backref" href="#id24">Applying processors and transformers (Processor API integration)</a><a class="headerlink" href="#applying-processors-and-transformers-processor-api-integration" title="Permalink to this headline"></a></h3>
- <p>Beyond the aforementioned <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateless</span></a> and
- <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"><span class="std std-ref">stateful</span></a> transformations, you may also
- leverage the <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a> from the DSL.
- There are a number of scenarios where this may be helpful:</p>
- <ul class="simple">
- <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet available in the DSL.</li>
- <li><strong>Combining ease-of-use with full flexibility where it’s needed:</strong> Even though you generally prefer to use
- the expressiveness of the DSL, there are certain steps in your processing that require more flexibility and
- tinkering than the DSL provides. For example, only the Processor API provides access to a
- record’s metadata such as its topic, partition, and offset information.
- However, you don’t want to switch completely to the Processor API just because of that.</li>
- <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies that provide an
- imperative API, and migrating some of your legacy code to the Processor API was faster and/or easier than to
- migrate completely to the DSL right away.</li>
+ <div class="section" id="applying-processors-processor-api-integration">
+ <a class="headerlink" href="#applying-processors-processor-api-integration" title="Permalink to this headline">
+ <h3>
+ <a class="toc-backref" href="#id24">Applying processors (Processor API integration)</a>
+ </h3>
+ </a>
+ <p>Beyond the aforementioned <a class="reference internal"
+ href="#streams-developer-guide-dsl-transformations-stateless">
+ <span class="std std-ref">stateless</span></a> and
+ <a class="reference internal" href="#streams-developer-guide-dsl-transformations-stateless"> <span
+ class="std std-ref">stateful</span></a> transformations, you may also leverage the Processor API from the
+ DSL. There are a number of scenarios where this may be helpful:
+ </p>
+ <ul>
+ <li><strong>Customization:</strong> You need to implement special, customized logic that is not or not yet
+ available
+ in the DSL.</li>
+ <li><strong>Combining ease-of-use with full flexibility where it's needed:</strong> Even though you generally
+ prefer
+ to use the expressiveness of the DSL, there are certain steps in your processing that require more
+ flexibility and tinkering than the DSL provides. For example, only the Processor API provides access to a
+ record's metadata such as its topic, partition, and offset information. However, you don't want to switch
+ completely to the Processor API just because of that; and</li>
+ <li><strong>Migrating from other tools:</strong> You are migrating from other stream processing technologies
+ that
+ provide an imperative API, and migrating some of your legacy code to the Processor API was faster and/or
+ easier than to migrate completely to the DSL right away.</li>
</ul>
- <table border="1" class="non-scrolling-table width-100-percent docutils">
- <colgroup>
- <col width="19%" />
- <col width="81%" />
- </colgroup>
- <thead valign="bottom">
- <tr class="row-odd"><th class="head">Transformation</th>
- <th class="head">Description</th>
- </tr>
+ <h4>Operations and concepts</h4>
+ <ul>
+ <li><code>KStream#process</code>: Process all records in a stream, one record at a time, by applying a
+ <code>Processor</code> (provided by a given <code>ProcessorSupplier</code>);
+ </li>
+ <li><code>KStream#processValues</code>: Process all records in a stream, one record at a time, by applying a
+ <code>FixedKeyProcessor</code> (provided by a given <code>FixedKeyProcessorSupplier</code>);
+ </li>
+ <li><code>Processor</code>: A processor of key-value pair records;</li>
+ <li><code>ContextualProcessor</code>: An abstract implementation of <code>Processor</code> that manages the
+ <code>ProcessorContext</code> instance and provides default no-op implementation of
+ <code>Processor#close</code>.
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #18314:
URL: https://github.com/apache/kafka/pull/18314#discussion_r1896961093
##########
docs/streams/developer-guide/dsl-api.html:
##########
@@ -3243,6 +3244,177 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
// `Processor` interface, see further down below.
.process(() -> new PopularPageEmailAlert("alerts@yourcompany.com"));</code></pre>
</div>
+ <div class="section" id="migrating-from-transform-to-process">
+ <h2>
+ <a class="headerlink" href="#migrating-from-transform-to-process" title="Permalink to this headline">
+ Migrating from transform to process
+ </a>
+ </h2>
+ <div>
+ <p>
+ As of Kafka 4.0, several deprecated methods in the Kafka Streams API, such as <code>transform</code>,
+ <code>flatTransform</code>, <code>transformValues</code>, and <code>flatTransformValues</code>, have
+ been removed. These methods have been replaced with the more versatile <code>process</code> API. This
+ guide provides detailed steps for migrating existing code to use the new <code>process</code> API and
+ explains the benefits of the changes.
+ </p>
+ <h3>Overview of Changes</h3>
+ <p>The following deprecated methods are no longer available in Kafka Streams:</p>
+ <ul>
+ <li><code>KStream#transform</code></li>
+ <li><code>KStream#flatTransform</code></li>
+ <li><code>KStream#transformValues</code></li>
+ <li><code>KStream#flatTransformValues</code></li>
+ </ul>
+ <p>The <code>process</code> API now serves as a unified replacement for all these methods. It simplifies the
+ API surface while maintaining support for both stateless and stateful operations.</p>
+
+ <h3>Migration Steps</h3>
+
+ <h4>1. Migrate from <code>transform</code> to <code>process</code></h4>
+
+ <pre class="line-numbers"><code class="language-java">// Before
+KStream<String, String> transformedStream = stream.transform(
+ () -> new CustomTransformer(),
+ "state-store"
+);
+
+// After
+KStream<String, String> processedStream = stream.process(
+ () -> new CustomProcessor(),
+ "state-store"
+);</code></pre>
+
+ <h4>2. Migrate from <code>flatTransform</code> to <code>process</code></h4>
+ <pre class="line-numbers"><code class="language-java">// Before
+KStream<String, String> flatTransformedStream = stream.flatTransform(
+ () -> new CustomFlatTransformer(),
+ "state-store"
+);
+
+// After
+KStream<String, String> processedStream = stream.process(
+ () -> new Processor<String, String>() {
Review Comment:
The new `api.Processor` has 4 generic, key/value in/out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #18314:
URL: https://github.com/apache/kafka/pull/18314#issuecomment-2568465445
Did not look into HTML yet...
```
The migration process consists of:
Replace Transformer with Processor or ValueTransformer with FixedKeyProcessor;
Replace record key and value with Record or FixedKeyRecord;
Rewrite the transform method of Transformer and ValueTransformer as process or processValues;
Use the new Record or FixedKeyRecord as argument of the renamed method;
Rewrite the return type of the renamed method to void and forward the record through the context; and finally
Change the KStream call of the transform method to process or processValues.
```
This seems to be _very_ detailed? When saying `Replace Transformer with Processor or ValueTransformer with FixedKeyProcessor`, it seems to be more or less clear that one also needs to do `Replace record key and value with Record or FixedKeyRecord;` and `Rewrite the transform method of Transformer and ValueTransformer as process or processValues;`? This one (`Use the new Record or FixedKeyRecord as argument of the renamed method;`) seems to be redundant anyway?
I am worries that people get intimidated, as it sound more complex than it actually is? Might it be better to keep it simpler?
Btw: You did not work on the code change for this, but we also did remove `KStream#process(/*old Processor*/)` which is also replaced by the same `KStream#process(/*new api.Processor*/`). It would be great to add this to this migration guide, too.
Example 1:
`logStream.flatTransform(() -> new LogSeverityTransformer())` vs `logStream.process(LogSeverityProcessor::new)` -- why are you switching the pattern from lambda to method reference? This can be confusing for reads. Stick with either one. Seem you are using method reference in the other examples.
Also, the `to(...)` operator is missing in the `process()` example.
Example 2:
`SlangReplacementTransformer implements ValueTransformer<String, Iterable<String>>` -- it seem the result type should be `String` not `Iterable<String>` and the `returns` statement should concatenate the words (plus adding back blanks) similar to the `Processor` example?
Example 3:
```
// Send a discount notification
context.forward(new Record<>(record.key(), DISCOUNT_NOTIFICATION_MESSAGE, record.timestamp()));
```
Why no `context.forward(record.withValue(DISCOUNT_NOTIFICATION_MESSAGE)` ?
Should we also mention `ValueTransformerWithKey` that was removed? I noticed that the `ValueTransformer` example now needs to add the key-type (`Void` in your example), which was not necessary for `ValueTransformer` as it does not have access to the key anyway, however, `ValueTransformerWithKey` has access to it, and thus three generic types, so basically the new `FixedKeyProcessor` is closer to `ValueTransformerWithKey` compared to `ValueTransformer`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16339: [Docs] Add migrating from transform to process [kafka]
Posted by "fonsdant (via GitHub)" <gi...@apache.org>.
fonsdant commented on PR #18314:
URL: https://github.com/apache/kafka/pull/18314#issuecomment-2605940755
Hi, @mjsax! I have updated (instead of removing) the migration/applying section and added the example for old process. Now, I am reviewing the Slang example and analyzing the use of ContextualProcessor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org