You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@dubbo.apache.org by "YuZhangLarry (via GitHub)" <gi...@apache.org> on 2026/06/30 10:11:31 UTC
[PR] feat: bootstrap RAG knowledge base and add think-stage intent eval suite [dubbo-admin]
YuZhangLarry opened a new pull request, #1496:
URL: https://github.com/apache/dubbo-admin/pull/1496
- **RAG seeding** (`ai/component/rag/`): bundled seed docs (`seeds/*.md` — Dubbo
overview, configuration, registry, traffic governance, Dubbo Admin) are
embedded via `//go:embed` and indexed at startup, so a fresh deployment can
answer documentation-grade questions out of the box. Indexing is best-effort
and idempotent (dedup by content hash) and batched to the embedding backend's
per-request input limit; a missing embedder key degrades gracefully instead of
failing startup, and reranking falls back to raw vector results when no rerank
key is configured.
- **ReAct think-stage tuning** (`ai/component/agent/`, `ai/prompts/agentThink.txt`):
the think stage now prefers answering directly (`suggested_tools: []`) and no
longer confuses objective documentation lookups (config keys, defaults,
required addresses) with memory search mid-conversation. Adds a `ThinkOnce`
entry point so the stage can be evaluated in isolation.
- **Intent-classification eval suite** (`ai/test/e2e/`):
`TestThinkClassificationIsolated` runs each documentation question in a fresh
session (no multi-turn bias) and repeats it to surface nondeterminism as a
distribution, asserting a stable `DOCUMENTATION_QUERY` classification routed to
`query_knowledge_base`; `TestMultiTurnConversation` exercises the full
think→act→observe loop end to end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] feat: bootstrap RAG knowledge base and add think-stage intent eval suite [dubbo-admin]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #1496:
URL: https://github.com/apache/dubbo-admin/pull/1496#issuecomment-4842238573
## [](https://sonarcloud.io/dashboard?id=apache_dubbo-admin&pullRequest=1496) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_dubbo-admin&pullRequest=1496&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_dubbo-admin&pullRequest=1496&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_dubbo-admin&pullRequest=1496&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-admin&pullRequest=1496&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-admin&pullRequest=1496&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_dubbo-admin&pullRequest=1496)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] feat: bootstrap RAG knowledge base and add think-stage intent eval suite [dubbo-admin]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #1496:
URL: https://github.com/apache/dubbo-admin/pull/1496#discussion_r3563521273
##########
ai/component/server/component.go:
##########
@@ -106,6 +106,12 @@ func (s *ServerComponent) Init(rt *runtime.Runtime) error {
}
func (s *ServerComponent) Start() error {
+ // Check if server is already running (idempotent Start)
+ if s.srv != nil {
+ s.rt.GetLogger().Info("Server already running", "addr", fmt.Sprintf("%s:%d", s.host, s.port))
+ return nil
+ }
Review Comment:
Start() now returns early when s.srv != nil, but Stop() never resets s.srv to nil after shutdown. This makes the server component impossible to restart after calling Stop(), which is surprising given the new 'idempotent Start' semantics.
##########
ai/component/server/engine/handlers.go:
##########
@@ -123,6 +163,26 @@ func (h *AgentHandler) StreamChat(c *gin.Context) {
// MessageDelta finishes stream and handles usage
func (h *AgentHandler) MessageDelta(sseHandler *sse.SSEHandler, output schema.Schema) {
stopReason := "end_turn"
+
+ // If the output is an Observation with a final answer, stream it as text delta first
+ switch v := output.(type) {
+ case *schema.Observation:
+ // Send Summary as text if present
+ if v.Summary != "" {
+ if err := sseHandler.HandleText(v.Summary+"\n", 0); err != nil {
+ rt.GetLogger().Error("Failed to stream summary in MessageDelta", "error", err)
+ }
+ }
+ // Send FinalAnswer as text if present
+ if v.FinalAnswer != "" {
+ if err := sseHandler.HandleText(v.FinalAnswer+"\n", 0); err != nil {
+ rt.GetLogger().Error("Failed to stream final answer in MessageDelta", "error", err)
+ }
+ }
+ default:
+ rt.GetLogger().Info("MessageDelta: unexpected output type", "type", fmt.Sprintf("%T", output))
+ }
+
if err := sseHandler.MessageDeltaWithUsage(stopReason, output); err != nil {
sseHandler.HandleError("finish_stream_error", fmt.Sprintf("failed to finish stream: %v", err))
}
Review Comment:
MessageDelta streams Observation.Summary/FinalAnswer via HandleText and then also emits message_delta. However, the observe stage already streams Summary/FinalAnswer into UserRespChan (see ai/component/agent/react/react.go:onOutput2Flow) before StreamFinal is sent (ai/component/agent/agent.go). This risks duplicating the final answer in SSE clients.
##########
ai/component/rag/config.go:
##########
@@ -32,13 +32,20 @@ type QueryProcessorConfig = query.QueryProcessorConfig
// RAGSpec defines RAG component configuration with recursive structure
// Each subcomponent uses the standard Config pattern (type + spec)
type RAGSpec struct {
- Embedder *config.Config `yaml:"embedder"`
- Loader *config.Config `yaml:"loader"`
- Splitter *config.Config `yaml:"splitter"`
- Indexer *config.Config `yaml:"indexer"`
- Retriever *config.Config `yaml:"retriever"`
- Reranker *config.Config `yaml:"reranker,omitempty"`
- QueryProcessor *config.Config `yaml:"query_processor,omitempty"`
+ Embedder *config.Config `yaml:"embedder"`
+ Loader *config.Config `yaml:"loader"`
+ Splitter *config.Config `yaml:"splitter"`
+ Indexer *config.Config `yaml:"indexer"`
+ Retriever *config.Config `yaml:"retriever"`
+ Reranker *config.Config `yaml:"reranker,omitempty"`
+ QueryProcessor *config.Config `yaml:"query_processor,omitempty"`
+ Seed *SeedSpec `yaml:"seed,omitempty"`
+}
+
+// SeedSpec controls auto-seeding the local vector store with bundled knowledge
+// documents at startup, so retrieval is never empty on a fresh deployment.
+type SeedSpec struct {
+ Enabled bool `yaml:"enabled"`
Review Comment:
SeedSpec.Enabled is a bool, so if a user config includes `seed: {}` (or `seed:` without `enabled`) YAML decoding will set Enabled=false and silently disable seeding, even though the JSON schema declares default=true and the comments say seeding is on by default. To make 'enabled by default' real, consider using `*bool` (nil => default true) and updating seedEnabled() accordingly.
##########
ai/component/agent/react/react.go:
##########
@@ -393,26 +469,111 @@ func observe(g *genkit.Genkit, observePrompt ai.Prompt) agent.StreamFlow {
return nil, fmt.Errorf("history is empty")
}
- resp, err := observePrompt.Execute(ctx,
+ // Create context with timeout for observe stage
+ observeCtx, cancel := context.WithTimeout(ctx, observeTimeout)
+ defer cancel()
+
+ resp, err := observePrompt.Execute(observeCtx,
ai.WithMessages(history.WindowMemory(sessionID)...),
)
+ // Check for timeout or other errors
if err != nil {
+ // If timeout occurred, return fallback observation
+ if errors.Is(err, context.DeadlineExceeded) || errors.Is(err, context.Canceled) {
+ runtime.GetLogger().Warn("Observe stage timeout, returning fallback response", "timeout", observeTimeout)
+ fallbackObs := generateFallbackObservation(in)
+ msg := fallback.NewHandler(nil).MarshalObservation(&fallbackObs)
+ history.AddHistory(sessionID, msg)
+ return fallbackObs, nil
+ }
return nil, fmt.Errorf("failed to execute observe prompt: %w", err)
}
- // Parse output
- var observation schema.Observation
- observation.UsageInfo = &ai.GenerationUsage{}
- err = resp.Output(&observation)
+ // Parse output with fallback
+ fbHandler := fallback.NewHandler(nil)
+ observation, err := fbHandler.ParseObservation(resp)
if err != nil {
- return nil, fmt.Errorf("failed to parse observe prompt response: %w", err)
+ // If parsing fails, generate fallback
+ runtime.GetLogger().Warn("Failed to parse observation, returning fallback", "error", err)
+ fallbackObs := generateFallbackObservation(in)
+ msg := fallback.NewHandler(nil).MarshalObservation(&fallbackObs)
+ history.AddHistory(sessionID, msg)
+ return fallbackObs, nil
}
runtime.GetLogger().Info("Observe out:", "out", observation)
- history.AddHistory(sessionID, resp.Message)
+ // Use fallback marshal if JSON marshaling fails
+ msg := fbHandler.MarshalObservation(observation)
+ history.AddHistory(sessionID, msg)
schema.AccumulateUsage(observation.UsageInfo, resp.Usage, in.Usage())
return observation, err
})
}
+
+// generateFallbackObservation creates a fallback observation when observe stage times out
+func generateFallbackObservation(input schema.Schema) schema.Observation {
+ fallback := schema.Observation{
+ Heartbeat: false,
+ FinalAnswer: "",
+ Summary: "Generate response based on available context",
+ Evidence: "Timeout - using available context",
+ }
+
+ // Try to extract information from input to generate a meaningful response
+ switch v := input.(type) {
+ case schema.ThinkOutput:
+ // If we have think output, use the thought to generate a response
+ if v.Thought != "" {
+ fallback.FinalAnswer = generateResponseFromThought(v.Thought, v.Intent)
+ }
+ case schema.Observation:
+ // If we have previous observation, reuse it if it has final answer
+ if v.FinalAnswer != "" {
+ fallback.FinalAnswer = v.FinalAnswer
+ fallback.Evidence = v.Evidence
+ } else {
+ // Continue from previous observation
+ fallback.Heartbeat = true
+ fallback.Focus = "Continue processing with available information"
+ }
+ case schema.ToolOutputs:
+ // If we have tool outputs, generate response from them
+ if len(v.Outputs) > 0 {
+ fallback.FinalAnswer = generateResponseFromToolOutputs(v.Outputs)
+ }
+ default:
+ // Default fallback for any other input type
+ fallback.FinalAnswer = "I apologize, but I need more time to process your request. Based on the available context, I cannot provide a complete answer at this moment."
+ }
Review Comment:
generateFallbackObservation only matches value types (schema.ThinkOutput / schema.ToolOutputs / schema.Observation), but the pipeline frequently passes pointers (e.g. *schema.Observation from ParseObservation, *schema.ThinkOutput from ParseThinkOutput). When observe times out, this makes the fallback almost always hit the default branch and ignore available context.
##########
ai/component/server/engine/handlers.go:
##########
@@ -92,7 +92,14 @@ func (h *AgentHandler) StreamChat(c *gin.Context) {
channels.UserRespChan = nil
continue
}
+ rt.GetLogger().Info("Handler received feedback",
+ "session_id", sessionID,
+ "text", feedback.Text(),
+ "done", feedback.IsDone(),
+ "final", feedback.IsFinal(),
+ "final_nil", feedback.Final() == nil)
Review Comment:
The per-chunk "Handler received feedback" log includes the full streamed text (feedback.Text()), which can leak user content / model output into logs and may be large/noisy at Info level. Consider logging only metadata (index/done/final/text length) and/or lowering to Debug.
##########
ai/test/e2e/multi_turn_conversation_test.go:
##########
@@ -0,0 +1,583 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package e2e
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "fmt"
+ "io"
+ "net/http"
+ "os"
+ "path/filepath"
+ "runtime"
+ "strings"
+ "testing"
+ "time"
+
+ appruntime "dubbo-admin-ai/runtime"
+)
+
+// TestMultiTurnConversation tests 10 rounds of conversation
+// This test verifies the agent's ability to maintain context and provide coherent responses
+func TestMultiTurnConversation(t *testing.T) {
+ if testing.Short() {
+ t.Skip("skipping e2e test in short mode")
+ }
+
+ // Bootstrap runtime
+ ctx := context.Background()
+ configPath, _ := createTestConfig(t)
+ rt, err := appruntime.Bootstrap(configPath, registerFactories)
+ if err != nil {
+ t.Fatalf("Failed to bootstrap runtime: %v", err)
+ }
+ defer rt.StopAll()
+
+ // Get server component
+ _, err = rt.GetComponent("server")
+ if err != nil {
+ t.Fatalf("Failed to get server component: %v", err)
+ }
+
+ // Note: Server is already started by Bootstrap, no need to call Start() again
+ // Runtime.Bootstrap automatically calls Start() on all components
+
+ // Wait for server to be ready (server starts asynchronously)
+ // Use health check to ensure server is actually ready
+ baseURL := fmt.Sprintf("http://127.0.0.1:8880")
+ if !waitForServerReady(t, ctx, baseURL, 30*time.Second) {
+ t.Fatalf("Server did not become ready in time")
+ }
+
+ t.Run("create_and_chat_10_rounds", func(t *testing.T) {
+ // Create a new session
+ sessionID := createSession(t, ctx, baseURL)
+ t.Logf("Created session: %s", sessionID)
+
+ // Define 10 rounds of conversation
+ // Each round tests different capabilities
+ conversationRounds := []ConversationRound{
+ {
+ Name: "Round 1: Basic Introduction",
+ Message: "Hello, what is Dubbo?",
+ Expected: []string{"Dubbo", "RPC", "framework"},
+ },
+ {
+ Name: "Round 2: Follow-up on Architecture",
+ Message: "What are the main components of Dubbo?",
+ Expected: []string{"Provider", "Consumer", "Registry", "Monitor"},
+ },
+ {
+ Name: "Round 3: Service Governance",
+ Message: "Tell me about Dubbo's service governance features",
+ Expected: []string{"load balancing", "circuit breaking", "service degradation"},
+ },
+ {
+ Name: "Round 4: Load Balancing Details",
+ Message: "What load balancing strategies does Dubbo support?",
+ Expected: []string{"random", "round-robin", "least active", "consistent hashing"},
+ },
+ {
+ Name: "Round 5: Protocol Support",
+ Message: "Which protocols does Dubbo support?",
+ Expected: []string{"Dubbo", "REST", "HTTP", "gRPC"},
+ },
+ {
+ Name: "Round 6: Fault Tolerance",
+ Message: "What are the cluster fault tolerance strategies?",
+ Expected: []string{"Failover", "Failfast", "Failsafe", "Failback"},
+ },
+ {
+ Name: "Round 7: Context Retention Check",
+ Message: "Based on what we discussed, which load balancing is the default?",
+ Expected: []string{"random", "default"},
+ },
+ {
+ Name: "Round 8: Advanced Features",
+ Message: "How does Dubbo handle service registration and discovery?",
+ Expected: []string{"registry", "Nacos", "Zookeeper"},
+ },
+ {
+ Name: "Round 9: Performance Question",
+ Message: "What makes Dubbo perform well in high-concurrency scenarios?",
+ Expected: []string{"NIO", "async", "long connection"},
+ },
+ {
+ Name: "Round 10: Summary Request",
+ Message: "Can you summarize the key points about Dubbo we discussed?",
+ Expected: []string{"architecture", "protocols", "governance", "fault tolerance"},
+ },
+ // Rounds 11-14 ask for precise, documentation-grade details that live in
+ // the seeded knowledge base (component/rag/seeds/*.md). Per the Think
+ // stage's tool-use policy these should classify as DOCUMENTATION_QUERY and
+ // trigger query_knowledge_base, exercising retrieval against seed data.
+ {
+ Name: "Round 11: Precise Config Keys (RAG)",
+ Message: "What are the exact dubbo.provider config keys and their default values for timeout, retries and loadbalance?",
+ Expected: []string{"timeout", "retries", "loadbalance"},
+ },
+ {
+ Name: "Round 12: Serialization Options (RAG)",
+ Message: "Which serialization protocols does Dubbo support and which one is the default?",
+ Expected: []string{"hessian2", "protobuf", "kryo"},
+ },
+ {
+ Name: "Round 13: Register Mode Default (RAG)",
+ Message: "In Dubbo 3, what is the default register-mode and what values can it take?",
+ Expected: []string{"instance", "interface", "all"},
+ },
+ {
+ Name: "Round 14: Dubbo Admin Connection Config (RAG)",
+ Message: "Which addresses must Dubbo Admin be configured with to manage a cluster, and what are the config keys?",
+ Expected: []string{"registry", "config-center", "metadata-report"},
+ },
+ }
+
+ // Track test results
+ results := &TestResults{
+ SessionID: sessionID,
+ Rounds: make([]RoundResult, 0, len(conversationRounds)),
+ }
+
+ // Execute each round
+ for i, round := range conversationRounds {
+ t.Logf("\n=== %s ===", round.Name)
+
+ startTime := time.Now()
+
+ // Send message
+ response := sendMessage(t, ctx, baseURL, sessionID, round.Message)
+
+ duration := time.Since(startTime)
+
+ // Record result
+ roundResult := RoundResult{
+ RoundNumber: i + 1,
+ Name: round.Name,
+ UserMessage: round.Message,
+ Response: response,
+ Duration: duration,
+ Timestamp: startTime,
+ Expected: round.Expected,
+ }
+
+ // Validate response
+ roundResult.Success = validateResponse(response, round.Expected)
+ roundResult.MatchedKeywords = findMatchedKeywords(response, round.Expected)
+
+ results.Rounds = append(results.Rounds, roundResult)
+
+ // Log results
+ if roundResult.Success {
+ t.Logf("✓ Response received in %v (matched keywords: %v)",
+ duration, roundResult.MatchedKeywords)
+ } else {
+ t.Logf("⚠ Response received but might be incomplete (matched: %v)",
+ roundResult.MatchedKeywords)
+ }
+
+ t.Logf("Response preview: %s", truncateString(response, 200))
+
+ // Small delay between messages
+ time.Sleep(500 * time.Millisecond)
+ }
+
+ // Print summary
+ printTestSummary(t, results)
+
+ // Export results to file
+ exportResultsToFile(t, results)
+ })
+}
+
+// ConversationRound defines a single round of conversation
+type ConversationRound struct {
+ Name string
+ Message string
+ Expected []string
+}
+
+// RoundResult stores the result of a single conversation round
+type RoundResult struct {
+ RoundNumber int
+ Name string
+ UserMessage string
+ Response string
+ Duration time.Duration
+ Timestamp time.Time
+ Success bool
+ Expected []string
+ MatchedKeywords []string
+}
+
+// TestResults stores all test results
+type TestResults struct {
+ SessionID string
+ Rounds []RoundResult
+}
+
+// createSession creates a new chat session
+func createSession(t *testing.T, ctx context.Context, baseURL string) string {
+ req, err := http.NewRequestWithContext(ctx, "POST", baseURL+"/api/v1/ai/sessions", nil)
+ if err != nil {
+ t.Fatalf("Failed to create request: %v", err)
+ }
+
+ resp, err := http.DefaultClient.Do(req)
+ if err != nil {
+ t.Fatalf("Create session failed: %v", err)
+ }
+ defer resp.Body.Close()
+
+ if resp.StatusCode != http.StatusOK {
+ body, _ := io.ReadAll(resp.Body)
+ t.Fatalf("Create session status = %d, body: %s", resp.StatusCode, string(body))
+ }
+
+ var result struct {
+ Message string `json:"message"`
+ Data struct {
+ SessionID string `json:"session_id"`
+ } `json:"data"`
+ }
+ if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
+ t.Fatalf("Failed to decode response: %v", err)
+ }
+
+ return result.Data.SessionID
+}
+
+// sendMessage sends a chat message and returns the response
+func sendMessage(t *testing.T, ctx context.Context, baseURL, sessionID, message string) string {
+ chatReq := map[string]string{
+ "message": message,
+ "sessionID": sessionID,
+ }
+ body, err := json.Marshal(chatReq)
+ if err != nil {
+ t.Fatalf("Failed to marshal request: %v", err)
+ }
+
+ req, err := http.NewRequestWithContext(ctx, "POST", baseURL+"/api/v1/ai/chat/stream", bytes.NewReader(body))
+ if err != nil {
+ t.Fatalf("Failed to create request: %v", err)
+ }
+ req.Header.Set("Content-Type", "application/json")
+ req.Header.Set("Accept", "text/event-stream")
+
+ // Use 10 minute timeout to match server timeout (15m) with buffer
+ client := &http.Client{Timeout: 600 * time.Second}
+ resp, err := client.Do(req)
+ if err != nil {
+ t.Fatalf("Chat request failed: %v", err)
+ }
+ defer resp.Body.Close()
+
+ if resp.StatusCode != http.StatusOK {
+ body, _ := io.ReadAll(resp.Body)
+ t.Fatalf("Chat status = %d, body: %s", resp.StatusCode, string(body))
+ }
+
+ // Read SSE stream and collect response
+ return readSSEStream(t, resp.Body)
+}
+
+// readSSEStream reads the SSE stream and returns the accumulated response
+func readSSEStream(t *testing.T, body io.ReadCloser) string {
+ buf := new(strings.Builder)
+ chunk := make([]byte, 4096)
+ noDataCount := 0
+ maxNoDataCounts := 10 // Allow up to 10 consecutive reads with no new data (5 seconds total)
+
+ for {
+ n, err := body.Read(chunk)
+ if n > 0 {
+ chunkStr := string(chunk[:n])
+ buf.Write(chunk[:n])
+ noDataCount = 0
+
+ // Check if stream is complete - look for message_stop event
+ if strings.Contains(chunkStr, "event: message_stop") {
+ t.Logf("Found message_stop event, stream complete")
+ break
+ }
+ }
+ if err != nil {
+ if err == io.EOF {
+ t.Logf("EOF received, stream complete")
+ break
+ }
+ t.Logf("Read error (non-fatal): %v", err)
+ break
+ }
+
+ // Check for timeout - if no new data for 500ms for multiple consecutive reads
+ if n == 0 {
+ noDataCount++
+ if noDataCount > maxNoDataCounts {
+ t.Logf("No new data for %d consecutive reads, assuming stream complete", noDataCount)
+ break
+ }
+ time.Sleep(500 * time.Millisecond)
+ }
+
+ // Timeout protection - increased limit
+ if buf.Len() > 500000 {
+ t.Logf("Response too large, stopping read")
+ break
+ }
+ }
+
+ // Extract text content from SSE events
+ return extractTextFromSSE(buf.String())
+}
+
+// extractTextFromSSE extracts text content from SSE format
+func extractTextFromSSE(sseData string) string {
+ lines := strings.Split(sseData, "\n")
+ var textParts []string
+
+ for _, line := range lines {
+ if strings.HasPrefix(line, "data:") {
+ data := strings.TrimPrefix(line, "data:")
+ data = strings.TrimSpace(data)
+
+ // Skip [DONE] markers
+ if data == "[DONE]" || data == "" {
+ continue
+ }
+
+ // Try to parse JSON data
+ var parsed map[string]interface{}
+ if err := json.Unmarshal([]byte(data), &parsed); err == nil {
+ // Handle content_block_delta event with nested delta.text structure
+ // Format: {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "..."}}
+ if delta, ok := parsed["delta"].(map[string]interface{}); ok {
+ if text, ok := delta["text"].(string); ok && text != "" {
+ textParts = append(textParts, text)
+ }
+ }
+ // Handle direct text field (for other event types)
+ if text, ok := parsed["text"].(string); ok && text != "" {
+ textParts = append(textParts, text)
+ }
+ // Handle content_block.text (for content_block_start events)
+ if contentBlock, ok := parsed["content_block"].(map[string]interface{}); ok {
+ if text, ok := contentBlock["text"].(string); ok && text != "" {
+ textParts = append(textParts, text)
+ }
+ }
+ }
+ }
+ }
+
+ result := strings.Join(textParts, "")
+ // Debug: log if we got very little content (likely only status messages)
+ if len(result) < 200 && strings.Contains(result, "分析问题中") {
+ // This is likely just status messages, add a marker
+ result += "[SSE_CAPTURE_INCOMPLETE: Only status messages captured]"
+ }
+ return result
+}
+
+// validateResponse checks if response contains expected keywords
+func validateResponse(response string, expected []string) bool {
+ responseLower := strings.ToLower(response)
+ matchCount := 0
+
+ for _, keyword := range expected {
+ if strings.Contains(responseLower, strings.ToLower(keyword)) {
+ matchCount++
+ }
+ }
+
+ // Consider successful if at least half of expected keywords are found
+ return matchCount >= len(expected)/2
+}
+
+// findMatchedKeywords returns which expected keywords were found in response
+func findMatchedKeywords(response string, expected []string) []string {
+ responseLower := strings.ToLower(response)
+ var matched []string
+
+ for _, keyword := range expected {
+ if strings.Contains(responseLower, strings.ToLower(keyword)) {
+ matched = append(matched, keyword)
+ }
+ }
+
+ return matched
+}
+
+// printTestSummary prints test summary
+func printTestSummary(t *testing.T, results *TestResults) {
+ t.Log("\n" + strings.Repeat("=", 60))
+ t.Log("MULTI-TURN CONVERSATION TEST SUMMARY")
+ t.Log(strings.Repeat("=", 60))
+
+ successCount := 0
+ totalDuration := time.Duration(0)
+
+ for _, round := range results.Rounds {
+ if round.Success {
+ successCount++
+ }
+ totalDuration += round.Duration
+ }
+
+ successRate := float64(successCount) / float64(len(results.Rounds)) * 100
+ avgDuration := totalDuration / time.Duration(len(results.Rounds))
+
+ t.Logf("Session ID: %s", results.SessionID)
+ t.Logf("Total Rounds: %d", len(results.Rounds))
+ t.Logf("Success Rate: %.1f%% (%d/%d)", successRate, successCount, len(results.Rounds))
+ t.Logf("Total Duration: %v", totalDuration)
+ t.Logf("Average Response Time: %v", avgDuration)
+
+ t.Log("\nRound-by-Round Results:")
+ for _, round := range results.Rounds {
+ status := "✓"
+ if !round.Success {
+ status = "⚠"
+ }
+ t.Logf(" %s Round %d: %s (%v) - Matched: %v",
+ status, round.RoundNumber, round.Name, round.Duration, round.MatchedKeywords)
+ }
+
+ t.Log(strings.Repeat("=", 60))
+}
+
+// exportResultsToFile exports test results to a markdown file
+func exportResultsToFile(t *testing.T, results *TestResults) {
+ _, file, _, _ := runtime.Caller(0)
+ aiDir := filepath.Dir(filepath.Dir(filepath.Dir(file)))
+
Review Comment:
exportResultsToFile writes markdown artifacts into the repo workspace (ai/test/results). This makes `go test ./...` non-hermetic and can create noisy/unexpected files in CI or developer runs. Consider gating export behind an env var (or writing to t.TempDir()).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] feat: bootstrap RAG knowledge base and add think-stage intent eval suite [dubbo-admin]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #1496:
URL: https://github.com/apache/dubbo-admin/pull/1496#issuecomment-5220743298
## [](https://sonarcloud.io/dashboard?id=apache_dubbo-admin&pullRequest=1496) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_dubbo-admin&pullRequest=1496&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_dubbo-admin&pullRequest=1496&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_dubbo-admin&pullRequest=1496&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-admin&pullRequest=1496&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-admin&pullRequest=1496&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_dubbo-admin&pullRequest=1496)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] feat: bootstrap RAG knowledge base and add think-stage intent eval suite [dubbo-admin]
Posted by "robocanic (via GitHub)" <gi...@apache.org>.
robocanic merged PR #1496:
URL: https://github.com/apache/dubbo-admin/pull/1496
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org