kkdai

將原來 Python Gemini ADK 的框架改成 Go ADK ，資源少超多的。 這支 Bot 還能直接聽語音、幫你做網頁摘要，甚至連 GitHub 的 Issues 都能自動彙整。 kkdai/linebot-helper-go

[Agent Platform in Action (3)] Solving Stateless Container HITL Amnesia: Cross-Instance Business Card Recognition and Pause/Resu !Phase 2: Stateless Cross-Instance HITL & GCS Artifact Architecture Recap In the previous two articles of this series: - "[[Agent Platform in Action (1)] Building a LINE AI Business Card King with Go and Google ADK 2.0 Graph Workflow](/2026/10/16/adk-graph-workflow-go/)": We used google.golang.org/adk/v2/workflow to implement a 12-node business card assistant featuring a Deterministic Router, AddFanOut / AddFanIn three-way parallel background checks, and NewRequestInputEvent Human-in-the-Loop (HITL) pause and loop correction. - "[[Agent Platform in Action (2)] From Monolithic Cloud Run to Google Cloud Agent Runtime: Refactoring LINE AI Business Card King with agents-cli and ADK Go](/2026/10/17/adk-graph-workflow-go-2/)": We introduced the agents-cli official development lifecycle, explained the Agent Runtime's Scale-to-Zero mechanism, and completed Phase 1 (PR #10), transforming the project into a standard adk_go dual-channel architecture supporting LINE Webhook, ADK WebUI Playground, and A2A protocols. Entering Phase 2 (Issue #3 & Issue #4), we face a classic and common pitfall in distributed systems when pushing to production (Cloud Run / Agent Runtime auto-scaling clusters): When containers can Scale to Zero or be load-balanced to different instances at any time, where should multimodal business card photos (Binary Images) be stored? And how can the HITL pause state, waiting for a user to click a button, be recovered across instances? In the third installment of this series, we will deep dive into how we used ADK Go v2.6.0's artifact/gcsartifact (GCS Artifact Service) and session/vertexai (Agent Platform Sessions) in PR #11 to completely cure the "HITL amnesia" of stateless containers! - * * 1. Why did the original HITL suffer from "Amnesia" and "State Bloat" in cloud multi-instance environments? During local development or with a single container instance, stuffing everything into memory is fast. However, once deployed to Google Cloud Run or Vertex AI Agent Runtime, which support Horizontal Auto-scaling or Scale-to-Zero, the first version's design immediately hits two major walls: Pain Point 1: The "State Bloat" anti-pattern of stuffing multi-MB business card []byte into Session.State In the first version, when a LINE user sent a high-resolution business card photo, we directly put the downloaded imageBytes []byte into ADK's Session.State["image_bytes"]: // ❌ Version 1 approach: Directly stuffing multi-MB raw image []byte into Session State InitialState: map[string]any{ StateKeyInputType: RouteIngest, StateKeyImageBytes: imageBytes, // A 3MB JPEG inflates to a ~4MB Base64 string after JSON serialization! } When using the local session.InMemoryService(), this is just a Go slice pointer. But when switching to cloud-persisted Agent Platform Sessions (session/vertexai) or Cloud SQL (session/database): 1. Serialization and network transfer for every node state change (StateDelta): As the ADK Graph Workflow moves from RouterNode to VisionOCRNode, fans out to three parallel nodes, and finally pauses at HITLConfirmNode, keeping multi-MB image binary data in Session.State significantly slows down every Cloud Session API read/write and might even trigger Cloud Session payload size limits! 2. Difficulty in auditing and versioning history: If a user sends multiple business cards across several turns, mixing raw binary files with structured JSON state makes the conversation state extremely bloated. Pain Point 2: Container instance switching leads to loss of HITL pendingSession memory mapping Let's look at a real-world user scenario: 1. Turn 1 (hits Container Instance A): A user takes a photo of a business card at a conference and sends it to the LINE Bot. Instance A completes Vision OCR and parallel background checks, then pauses at HITLConfirmNode via workflow.NewRequestInputEvent, pushing a Flex Message preview card to the user. 2. 10-minute break: The user goes to exchange thoughts with the contact and grabs a coffee. During this time, Instance A scales to 0 due to inactivity, or Container Instance B is scaled out due to other traffic. 3. Turn 2 (hits Container Instance B): The user clicks "✏️ Edit Fields" on LINE and enters "_Change title to CTO_", or clicks "✅ Confirm Save". In the first version, our Engine used an in-memory Go map[string]pendingHITL to track which userID was waiting for which SessionID and InterruptID: // ❌ Version 1 approach: Relying solely on a local in-memory map to determine if there's a pending HITL type Engine struct { mu sync.RWMutex pendingSession map[string]pendingHITL // userID -> {SessionID, InterruptID} } As a result, when the Turn 2 Webhook request is routed by the load balancer to the brand new Instance B, Instance B's pendingSession map is empty! HasPendingHITL(userID) returns false, and the system treats "_Change title to CTO_" as a brand new "Business Card Search" request, instantly breaking the HITL loop! - * * 2. The Standard ADK 2.0 Solution: Separation of Concerns between Session State and Artifact Storage To solve this, Google ADK 2.0's architectural design explicitly splits Agent runtime data into two independent services: | Comparison Dimension | Session Service (session.Service) | Artifact Service (artifact.Service) | | --- | --- | --- | | Core Purpose | Stores conversation event logs (Events), workflow breakpoints (Interrupts), and lightweight structured state (JSON State) | Stores large binary files (images, PDFs, audio, CSV reports) and provides automatic version control (Versioning) | | Cloud Implementation Package | google.golang.org/adk/v2/session/vertexai (Agent Platform Sessions) or session/database | google.golang.org/adk/v2/artifact/gcsartifact (underlying storage in Google Cloud Storage Bucket) | | Local Development Implementation | session.InMemoryService() | artifact.InMemoryService() | | Role in Business Card King | Stores draft_card JSON, active_interrupt_id, and lightweight pointers (filename + version number) to the image | Stores raw business card photo business_card.jpg (automatically increments Version 1, 2, 3... per upload) | With this dual-track design, the entire cross-instance multimodal business card recognition and HITL recovery process becomes crystal clear: sequenceDiagram autonumber actor User as LINE / WebUI User participant InstA as Container Instance A (Turn 1: Upload Card) participant GCS as GCS Artifact Service<br/>(artifact/gcsartifact) participant Sess as Agent Platform Sessions<br/>(session/vertexai) participant InstB as Container Instance B (Turn 2: Edit Fields) participant InstC as Container Instance C (Turn 3: Confirm Save) User->>InstA: 1. Upload business card photo (JPEG []byte) InstA->>GCS: 2. Save("business_card.jpg", inlineData)<br/>Returns Version = 1 InstA->>Sess: 3. Create Session, State only records:<br/>card_image_artifact="business_card.jpg"<br/>card_image_version=1 InstA->>GCS: 4. VisionOCRNode calls LoadVersion("business_card.jpg", 1) Note over InstA,Sess: Fan-Out Parallel duplicate check / Company search / Interactive retrieval -> Generate preview card InstA->>Sess: 5. HITLConfirmNode triggers NewRequestInputEvent<br/>Persists active_interrupt_id and draft_card InstA-->>User: 6. Return LINE Flex preview card (⏸️ Workflow Paused) Note over InstA,InstB: ☁️ Container scales to 0 or load balancer switches to a brand new stateless Instance B User->>InstB: 7. Input: "Change title to CTO" InstB->>Sess: 8. resolvePendingHITL(userID):<br/>Query latest Session's active_interrupt_id Sess-->>InstB: 9. Retrieve {SessionID, InterruptID} and draft_card InstB->>Sess: 10. ResumeHITL -> ApplyEditNode updates title -> Re-pauses and updates active_interrupt_id InstB-->>User: 11. Return updated Flex preview card (Title: CTO) Note over InstB,InstC: ☁️ Switch again to brand new stateless Instance C User->>InstC: 12. Click "✅ Confirm Save" (action=confirm_save) InstC->>Sess: 13. resolvePendingHITL(userID) retrieves latest InterruptID InstC->>Sess: 14. ResumeHITL -> SaveCardNode writes to card library + clears active_interrupt_id InstC-->>User: 15. Return save success and AI icebreaker draft! - * * 3. Code Implementation Breakdown (Phase 2 — PR #11) Now let's look at the code to see how we completed this upgrade with less than a few hundred lines of clean Go code. 1. Creating a Multi-backend Artifact Factory: sessions/artifacts.go We added sessions/artifacts.go under the sessions package, using ADK Go v2.6.0's built-in google.golang.org/adk/v2/artifact/gcsartifact package to automatically switch between Cloud GCS Bucket and local memory based on environment variables: package sessions import ( "context" "log/slog" "os" "strings" "google.golang.org/adk/v2/artifact" "google.golang.org/adk/v2/artifact/gcsartifact" ) // NewArtifactService returns an ADK artifact.Service backed by Google Cloud Storage // when ARTIFACT_BUCKET_NAME or LOGS_BUCKET_NAME is configured, and falls back to // artifact.InMemoryService() for local development and unit tests. func NewArtifactService(ctx context.Context) (artifact.Service, error) { bucketName := resolveArtifactBucketName() if bucketName == "" { slog.Info("Using in-memory artifact service (set ARTIFACT_BUCKET_NAME or LOGS_BUCKET_NAME for GCS)") return artifact.InMemoryService(), nil } svc, err := gcsartifact.NewService(ctx, bucketName) if err != nil { return nil, err } slog.Info("Using GCS artifact service", "bucket", bucketName) return svc, nil } A thoughtful detail: In the agents-cli Agent Runtime deployment environment, the system usually configures LOGS_BUCKET_NAME (or our custom ARTIFACT_BUCKET_NAME). Some might accidentally fill it in with a format like gs://my-bucket/prefix. Our resolveArtifactBucketName() automatically strips the gs:// prefix and path to ensure gcsartifact.NewService connects reliably, falling back to artifact.InMemoryService() if no bucket is set. Simultaneously, we inject ArtifactService and SessionService into launcher.Config in main.go and cmd/server/main.go: // main.go — Injecting both SessionService and ArtifactService into ADK Universal Launcher config := &launcher.Config{ SessionService: sessionService, ArtifactService: artifactService, AgentLoader: NewSingleAgentLoader(rootAgent), A2AOptions: a2aOpts, } - * * 2. Storing Business Card Images in Artifacts, Keeping only "Lightweight Pointers" in Session State Next, we refactor internal/workflow/graph.go and internal/workflow/nodes_ingest.go. First, we define two new lightweight State Keys and a default filename: const ( StateKeyImageBytes = "image_bytes" // Kept only for backward compatibility fallback StateKeyCardImageArtifact = "card_image_artifact" // Artifact filename StateKeyCardImageVersion = "card_image_version" // Artifact version number (int) DefaultCardImageArtifactName = "business_card.jpg" ) (1) Entering from LINE Webhook (Engine.StartIngest) When a LINE user sends an image, StartIngest first calls e.artifactService.Save to write the image binary to the Artifact service and only records the filename and version number in InitialState. StateKeyImageBytes is no longer written to Session.State: initialState := map[string]any{ StateKeyInputType: RouteIngest, StateKeyRecentEvents: recentEvents, } if e.artifactService != nil && len(imageBytes) > 0 { part := genai.NewPartFromBytes(imageBytes, detectImageMIMEType(imageBytes)) saveResp, saveErr := e.artifactService.Save(ctx, &artifact.SaveRequest{ AppName: appName, UserID: userID, SessionID: sess.ID(), FileName: DefaultCardImageArtifactName, Part: part, }) if saveErr == nil && saveResp != nil { initialState[StateKeyCardImageArtifact] = DefaultCardImageArtifactName initialState[StateKeyCardImageVersion] = saveResp.Version } } (2) Uploading images directly from ADK WebUI / Agent Engine Playground / A2A (RouterNode) What if the user isn't using LINE but uploads a business card photo directly via ADK WebUI (make playground) or Agent Engine Console? In RouterNode, we use ADK 2.0's ctx.Artifacts().Save(...) to transfer InlineData from ctx.UserContent() into the Artifact Service: // internal/workflow/nodes_ingest.go — RouterNode if getStringState(ctx, StateKeyCardImageArtifact) == "" && len(getBytesState(ctx, StateKeyImageBytes)) == 0 && ctx.UserContent() != nil { for _, part := range ctx.UserContent().Parts { if part != nil && part.InlineData != nil && len(part.InlineData.Data) > 0 { if ctx.Artifacts() != nil { mimeType := part.InlineData.MIMEType if mimeType == "" { mimeType = detectImageMIMEType(part.InlineData.Data) } artifactPart := genai.NewPartFromBytes(part.InlineData.Data, mimeType) if saveResp, saveErr := ctx.Artifacts().Save(ctx, DefaultCardImageArtifactName, artifactPart); saveErr == nil { setState(ctx, ev, StateKeyCardImageArtifact, DefaultCardImageArtifactName) setState(ctx, ev, StateKeyCardImageVersion, saveResp.Version) } else { setState(ctx, ev, StateKeyImageBytes, part.InlineData.Data) } } if getStringState(ctx, StateKeyInputType) == "" { setState(ctx, ev, StateKeyInputType, RouteIngest) } break } } } (3) VisionOCRNode Reads specific versions of business card images on demand (loadCardImageBytes) When the workflow enters VisionOCRNode to call Gemini Vision for multimodal recognition, it uses ctx.Artifacts().LoadVersion (or Load) to load the image into memory for the model. It's discarded after use, keeping Session.State clean: func loadCardImageBytes(ctx agent.Context) []byte { if artifactName := getStringState(ctx, StateKeyCardImageArtifact); artifactName != "" && ctx.Artifacts() != nil { version := getIntState(ctx, StateKeyCardImageVersion) var ( resp *artifact.LoadResponse err error ) if version > 0 { resp, err = ctx.Artifacts().LoadVersion(ctx, artifactName, version) } else { resp, err = ctx.Artifacts().Load(ctx, artifactName) } if err == nil && resp != nil && resp.Part != nil && resp.Part.InlineData != nil { return resp.Part.InlineData.Data } } return getBytesState(ctx, StateKeyImageBytes) // Fallback } - * * 3. Implementing Stateless Cross-Instance HITL State Discovery: resolvePendingHITL With image storage solved, here's the main event: When Container Instance B receives a user's "Change title to CTO" or "✅ Confirm Save", how does it automatically retrieve the HITL workflow paused by Instance A when its local memory is empty? The secret is: when HITLConfirmNode calls workflow.NewRequestInputEvent to pause, we've already written the current InterruptID into Session.State["active_interrupt_id"] (and SaveCardNode clears it to "" once saving is complete). Therefore, we implemented resolvePendingHITL(ctx, userID) in internal/workflow/graph.go: // resolvePendingHITL checks the local cache first and falls back to querying // the shared session.Service for any session belonging to userID with a non-empty // StateKeyActiveInterruptID. This enables stateless cross-instance HITL resume // on Cloud Run / Vertex AI Agent Runtime. func (e *Engine) resolvePendingHITL(ctx context.Context, userID string) (pendingHITL, bool) { // 1. Fast path: If a record exists in the same instance's memory, verify with sessionService if it's still paused e.mu.RLock() cached, ok := e.pendingSession[userID] e.mu.RUnlock() if ok && cached.SessionID != "" && cached.InterruptID != "" { if getResp, err := e.sessionService.Get(ctx, &session.GetRequest{ AppName: appName, UserID: userID, SessionID: cached.SessionID, }); err == nil && getResp != nil && getResp.Session != nil { if activeID, err := getResp.Session.State().Get(StateKeyActiveInterruptID); err == nil { if s, ok := activeID.(string); ok && strings.TrimSpace(s) != "" { cached.InterruptID = strings.TrimSpace(s) return cached, true } } } } // 2. Stateless Slow Path: When hitting a brand new container instance, query the shared sessionService directly! listResp, err := e.sessionService.List(ctx, &session.ListRequest{ AppName: appName, UserID: userID, }) if err != nil || listResp == nil { return pendingHITL{}, false } var ( matched pendingHITL latestTime time.Time foundActive bool ) for _, sess := range listResp.Sessions { if sess == nil || sess.State() == nil { continue } val, err := sess.State().Get(StateKeyActiveInterruptID) if err != nil || val == nil { continue } interruptID, ok := val.(string) interruptID = strings.TrimSpace(interruptID) if !ok || interruptID == "" { continue } updatedAt := sess.LastUpdateTime() if !foundActive || updatedAt.After(latestTime) { foundActive = true latestTime = updatedAt matched = pendingHITL{ SessionID: sess.ID(), InterruptID: interruptID, } } } if foundActive { e.mu.Lock() e.pendingSession[userID] = matched e.mu.Unlock() return matched, true } return pendingHITL{}, false } Behold the power of this design: - Regardless of which stateless instance the user's request is dispatched to by the Cloud Run or Agent Runtime load balancer, HasPendingHITL(userID) and ResumeHITL(ctx, userID, userInput) can retrieve the user's latest paused {SessionID, InterruptID} directly via the shared session.Service. - Then, Instance B simply wraps the user's input into an ADK 2.0 standard adk_request_input (workflow.WorkflowInputFunctionCallName) FunctionResponse and calls e.runner.Run to seamlessly wake the workflow from HITLConfirmNode! - * * 4. Writing Unit Tests to Verify "Across Three Container Instances"! To prove this architecture works in a multi-instance environment with no shared local memory, we wrote the TestCrossInstanceHITLResumeAndArtifactStorage test in internal/workflow/graph_test.go, creating 3 independent Engine instances (instanceA, instanceB, instanceC) to simulate three stateless containers: func TestCrossInstanceHITLResumeAndArtifactStorage(t *testing.T) { ctx := context.Background() cardStore := store.NewMemoryStore() mockLLM := NewMockLLM() // Simulating cloud-shared Agent Platform Session Service and GCS Artifact Service sharedSessionSvc := session.InMemoryService() sharedArtifactSvc := artifact.InMemoryService() // Creating 3 completely independent Engine instances (each with an independent and empty pendingSession map) instanceA, _ := NewEngine(cardStore, mockLLM, WithSessionService(sharedSessionSvc), WithArtifactService(sharedArtifactSvc), ) instanceB, _ := NewEngine(cardStore, mockLLM, WithSessionService(sharedSessionSvc), WithArtifactService(sharedArtifactSvc), ) instanceC, _ := NewEngine(cardStore, mockLLM, WithSessionService(sharedSessionSvc), WithArtifactService(sharedArtifactSvc), ) userID := "U_cross_instance_user" fakeImage := []byte{0xFF, 0xD8, 0xFF, 0xE0, 0x00, 0x10, 'J', 'F', 'I', 'F'} // Turn 1 (hits Instance A): Upload card photo -> Write to Artifact and pause at HITLConfirmNode res1, _ := instanceA.StartIngest(ctx, userID, fakeImage, []string{"Met at Cloud Summit"}) // Verify Session State has no bloated raw image_bytes, but lightweight card_image_artifact="business_card.jpg" // Turn 2 (hits brand new Instance B): User enters edit command 'Change title to CTO' // Instance B automatically detects the active_interrupt_id left by Instance A from sharedSessionSvc! res2, _ := instanceB.ResumeHITL(ctx, userID, "Change title to CTO") // Verify title updated to 'CTO' and re-entered HITL pause waiting for confirmation // Turn 3 (hits brand new Instance C): User clicks '✅ Confirm Save' res3, _ := instanceC.ResumeHITL(ctx, userID, "action=confirm_save") // Verify card successfully written to CardStore, AI icebreaker generated, and HasPendingHITL is zeroed for all instances! } Running go test -race -v ./...: === RUN TestCrossInstanceHITLResumeAndArtifactStorage --- PASS: TestCrossInstanceHITLResumeAndArtifactStorage (0.00s) === RUN TestRootAgentDirectImageArtifactIngestAndHITL --- PASS: TestRootAgentDirectImageArtifactIngestAndHITL (0.00s) PASS ok github.com/kkdai/line-card-assistant/internal/workflow 0.642s Not only was the three-stage relay from instanceA $\rightarrow$ instanceB $\rightarrow$ instanceC seamless, but TestRootAgentDirectImageArtifactIngestAndHITL in app/agent_test.go also verified that when a genai.Part image is passed directly from ADK WebUI / A2A, RouterNode automatically saves it to ArtifactService and pauses at HITLConfirmNode! - * * 4. Summary and Next Steps Through this practical guide (PR #11, completing Issue #3 and Issue #4 in the Roadmap), we've solved the two most critical persistence challenges for moving an AI Agent from a "toy prototype" to a "cloud stateless multi-instance production environment": 1. Separation of Large Files and State (Artifact vs. Session State): Using ADK Go v2.6.0's artifact/gcsartifact and ctx.Artifacts(), business card photos no longer bloat Session.State. Instead, they have independent GCS versioned storage, with Session State only keeping a few dozen bytes of pointers (card_image_artifact + card_image_version). 2. Stateless Cross-Instance HITL Recovery: Combining session/vertexai (Agent Platform Sessions) and resolvePendingHITL, even if a container scales to 0 or switches instances during the minutes spent waiting for human confirmation, the system can still resume the "edit field loop" or "confirm save" perfectly. In the next article, we will continue with cloud database persistence and a two-layer decoupled architecture, and use agents-cli eval to build an automated evaluation flywheel. Stay tuned! - * * Resources and Further Reading - Previous articles in the series: - [[Agent Platform in Action (1)] Building a LINE AI Business Card King with Go and Google ADK 2.0 Graph Workflow: From Fan-Out Parallel Retrieval to HITL Pause Loop Correction](/2026/10/16/adk-graph-workflow-go/) - [[Agent Platform in Action (2)] From Monolithic Cloud Run to Google Cloud Agent Runtime: Refactoring LINE AI Business Card King with agents-cli and ADK Go](/2026/10/17/adk-graph-workflow-go-2/) - Open source code and PRs for this project: - GitHub Repository: kkdai/line-card-assistant - Phase 2 Implementation Pull Request: PR #11: feat(phase2): integrate GCS Artifact Service and cross-instance HITL session recovery - Completed Issues: Issue #3, Issue #4 - Google ADK and Agent Platform Official Documentation: - ADK Documentation (llms.txt) - Getting Started - agents-cli

[Agent Platform in Practice (4)] Towards Cloud Two-layer Decoupling and Full Persistence: Completing the LINE AI Business Card M !Phase 3: Decoupled LINE Webhook Gateway + Agent Runtime & Cloud SQL Architecture Recap In the first three articles of the "Agent Platform in Action" series, we step-by-step pushed the LINE AI Business Card Assistant (line-card-assistant) from a conceptual prototype to a cloud-native architecture: - [[Agent Platform in Action (1)] Building a LINE AI Business Card Assistant with Go and Google ADK 2.0 Graph Workflow](/2026/10/16/adk-graph-workflow-go/): We used google.golang.org/adk/v2/workflow to build a 12-node business card processing graph (Graph Workflow), covering Vision OCR, AddFanOut / AddFanIn parallel background investigations, and the HITL (Human-in-the-Loop) pause and modification loop of NewRequestInputEvent. - [[Agent Platform in Action (2)] From Monolithic Cloud Run to Google Cloud Agent Runtime: Refactoring LINE AI Business Card Assistant with agents-cli and ADK Go](/2026/10/17/adk-graph-workflow-go-2/): Completed Phase 1 (PR #10), introducing the agents-cli (adk_go) standard dual-channel architecture, allowing the project to support LINE Webhook, ADK WebUI Playground, and the A2A protocol simultaneously. - [[Agent Platform in Action (3)] Solving HITL Amnesia in Stateless Containers: Implementing Cross-Instance Business Card Recognition and Pause/Resume with ADK 2.0 GCS Artifact Service and Agent Platform Sessions](/2026/10/20/adk-graph-workflow-go-3/): Completed the first half of Phase 2 (PR #11), using artifact/gcsartifact to move multi-MB business card photos out of Session.State, and combining session/vertexai to implement resolvePendingHITL, allowing stateless containers to seamlessly continue HITL modification and saving even if scaled to 0 or instances are switched. In today's fourth episode of the series, we will complete the final piece of Phase 2 (Issue #5) and the Phase 3 core architecture upgrade (Issue #6) (corresponding to PR #12): 1. Cloud Business Card Database Persistence (PostgresStore / Cloud SQL): Officially upgrading the business card library, which was originally stored in local memory or container temporary files, to a relational and JSON hybrid storage supporting Cloud SQL PostgreSQL, and enabling HybridSearch and token-scoring fault-tolerant BroadSearch. 2. Two-Tier Decoupled Call Architecture (Cloud Run LINE Webhook Gateway $\rightarrow$ Agent Runtime): Completely decoupling the lightweight edge layer (Edge Layer) responsible for "LINE verification and image downloading" from the execution layer (Execution Layer) responsible for "ADK 2.0 Graph Workflow multimodal inference and search"! - * * 1. Why Do We Need the "Cloud Persistence Iron Triangle" and "Two-Tier Decoupled Architecture"? When we deploy line-card-assistant to a Google Cloud production environment, a careful examination of the responsibilities of every piece of data and component in the system reveals two key problems that must be solved: 1. The Last Remaining "Local State": CardStore Business Card Database In the previous post (PR #11), we already placed the "original business card photos" in Google Cloud Storage (GCSArtifactService) and the "workflow execution state and HITL breakpoints" in Vertex AI / Cloud SQL (SessionService). However, when a user clicks "✅ Confirm Save" in LINE and reaches SaveCardNode, if the final confirmed business card data (models.BusinessCard) is still written to local memory MemoryStore or a local SQLite file inside the container: - Cleared on container restart or Scale-to-Zero: Once the Agent Runtime instance scales to 0, the hard-earned network business card library will evaporate instantly. - Multiple instances cannot share search results: When a user saves a business card on Instance A, and a search request is routed to Instance B a second later, HybridSearchNode won't find the card just saved in Instance B's memory! Therefore, we must complete the final corner of the "Cloud-Native Persistence Triad" — Cloud SQL PostgreSQL (PostgresStore): | Persistence Dimension | Content Responsible for Storing | Cloud Backend Service | ADK / Go Implementation Package | | --- | --- | --- | --- | | 1. Binary Large Files (Artifacts) | Original business card photos (business_card.jpg v1, v2…) | Google Cloud Storage (GCS) | sessions.NewArtifactService (artifact/gcsartifact) | | 2. Conversation and Graph Breakpoints (Sessions) | Graph Checkpoints, active_interrupt_id, draft_card | Vertex AI Agent Engine / Cloud SQL | sessions.NewService (session/vertexai, session/database) | | 3. Structured Business Data (Domain DB) | Verified BusinessCard official records, JSON industry tags, hybrid retrieval | Cloud SQL for PostgreSQL | store.NewStore (internal/store/postgres.go) | - * * 2. Why Decouple LINE Webhook Gateway and Agent Runtime into a Two-Tier Architecture? In monolithic (All-in-One) mode, cmd/server receives LINE Webhooks, runs Vision OCR, and executes three parallel Grounding search nodes in the same process. However, in an enterprise-grade Google Cloud Agent Platform architecture, splitting the system into "Cloud Run LINE Webhook Gateway (Edge Layer)" and "Google Cloud Agent Runtime (Agent Execution Layer)" offers three major advantages: 1. Isolation of Public Webhook Verification and Internal IAM Permissions: - LINE Messaging API requires a public HTTPS Endpoint (POST /webhook) to push events, uses LINE_CHANNEL_SECRET to verify the X-Line-Signature, and must immediately return 200 OK within 1-2 seconds. - Meanwhile, the actual Agent Runtime only needs internal authorization via Google Cloud IAM / Application Default Credentials (ADC), without needing exposure to unauthenticated public traffic or holding the LINE Channel Secret. 2. Multiple Channels Sharing the Same Session-backed Remote Agent: - After decoupling, whether a user sends a business card photo from LINE Bot, queries contacts from internal Gemini Enterprise (A2A), or a developer tests from ADK WebUI, they all call the same app.NewRootAgent hosted on the Agent Runtime! 3. Extreme Deployment Flexibility (One-click switch between All-in-One vs. Decoupled): - During local development or small projects, if AGENT_RUNTIME_URL is not set, cmd/server automatically runs in All-in-One monolithic mode. - In production, simply setting AGENT_RUNTIME_URL=https://... in Cloud Run automatically switches cmd/server to a lightweight stateless LINE Webhook Gateway! Let's look at the cross-layer interaction under the two-tier decoupled architecture with a complete system sequence diagram: sequenceDiagram autonumber actor User as LINE User participant GW as Cloud Run<br/>LINE Webhook Gateway<br/>(cmd/server + gateway.Client) participant AR as Google Cloud Agent Runtime<br/>(adkrest.Server / app.NewRootAgent) participant GCS as Cloud Storage<br/>(GCSArtifactService) participant Sess as Vertex AI / Cloud SQL<br/>(SessionService) participant PG as Cloud SQL PostgreSQL<br/>(PostgresStore) User->>GW: 1. Send business card photo (POST /webhook) GW-->>User: 2. Verify X-Line-Signature and immediately return HTTP 200 OK GW->>GW: 3. Download image from LINE Blob API []byte GW->>AR: 4. POST /api/apps/app/users/{user}/sessions/{id}<br/>+ POST /api/run (With Google Cloud ADC ID Token and image InlineData) AR->>GCS: 5. RouterNode automatically saves image to GCS Artifact ("business_card.jpg" v1) AR->>AR: 6. VisionOCRNode -> AddFanOut (Deduplication / Google Search company background / Conversation context) -> BuildPreviewNode AR->>Sess: 7. HITLConfirmNode triggers NewRequestInputEvent<br/>Persist active_interrupt_id="hitl_confirm_1" AR-->>GW: 8. Return Events and Session State (including flex_preview_json) GW-->>User: 9. Push LINE Flex Message business card preview card (⏸️ Waiting for user confirmation) Note over GW,AR: ☁️ Gateway is completely stateless; the next request can land on any Gateway instance User->>GW: 10. Input modification: "Change title to CTO" or click "✅ Confirm Save" GW->>AR: 11. HasPendingHITL -> GET /api/apps/app/users/{user}/sessions<br/>Automatically retrieve remote paused {SessionID, active_interrupt_id} GW->>AR: 12. POST /api/run (Send adk_request_input FunctionResponse) AR->>PG: 13. If confirming save: SaveCardNode calls PostgresStore.Save() Upsert to write to Cloud SQL AR->>Sess: 14. Clear active_interrupt_id="" and record saved_card_id AR-->>GW: 15. Return save success and AI icebreaker follow-up draft GW-->>User: 16. Push save confirmation and icebreaker to user! - * * 2. Implementation Breakdown (1): Building the Cloud SQL PostgreSQL Business Card Library (internal/store/postgres.go) — Issue #5 In our project, since sessions/sessions.go already introduced gorm.io/gorm and gorm.io/driver/postgres to support SESSION_TYPE=cloud_sql, we can directly use GORM to build a production-grade PostgresStore for CardStore without extra dependency bloat! 1. Extending the store.CardStore Interface: Native BroadSearch Support for All Backends While checking the original internal/workflow/nodes_search.go, we found a type assertion for *store.MemoryStore in BroadSearchNode: // ❌ Old way: Only MemoryStore supports BroadSearch; other Stores fall back to ListByUser if memStore, ok := cardStore.(*store.MemoryStore); ok { cards, err = memStore.BroadSearch(ctx, userID, keywords) } else { cards, err = cardStore.ListByUser(ctx, userID) } To ensure PostgresStore and MemoryStore share a consistent hybrid retrieval and fuzzy fallback experience, we officially included BroadSearch in the CardStore standard interface in internal/store/store.go: // CardStore defines the persistence and search interface for business cards. type CardStore interface { Save(ctx context.Context, card *models.BusinessCard) error GetByID(ctx context.Context, id string) (*models.BusinessCard, error) CheckDuplicate(ctx context.Context, userID, name, company, email string) (*models.BusinessCard, bool, error) HybridSearch(ctx context.Context, filter models.SearchFilter) ([]*models.BusinessCard, error) BroadSearch(ctx context.Context, userID string, keywords []string) ([]*models.BusinessCard, error) ListByUser(ctx context.Context, userID string) ([]*models.BusinessCard, error) } This way, BroadSearchNode only needs to cleanly call one line: cardStore.BroadSearch(ctx, userID, keywords)! 2. Designing GORM Schema and Upsert Mechanism (BusinessCardRecord) In internal/store/postgres.go, we defined BusinessCardRecord corresponding to the PostgreSQL business_cards table, added indexes to common query fields (user_id, name, company, email, met_date, updated_at), and stored Tags []string as a JSON string in tags_json: // BusinessCardRecord is the GORM database model for persisting a models.BusinessCard // in PostgreSQL / Cloud SQL. type BusinessCardRecord struct { ID string `gorm:"primaryKey;size:128"` UserID string `gorm:"index;size:128;not null"` Name string `gorm:"index;size:255"` Title string `gorm:"size:255"` Company string `gorm:"index;size:255"` Email string `gorm:"index;size:255"` Phone string `gorm:"size:64"` Address string `gorm:"type:text"` TaxID string `gorm:"size:32"` Website string `gorm:"size:512"` RawOCR string `gorm:"type:text"` CompanySummary string `gorm:"type:text"` CompanyInfo string `gorm:"type:text"` TagsJSON string `gorm:"column:tags_json;type:text"` MetContext string `gorm:"type:text"` RawImageURL string `gorm:"type:text"` MetDate time.Time `gorm:"index"` CreatedAt time.Time `gorm:"index"` UpdatedAt time.Time `gorm:"index"` } When a user takes a photo of an existing contact's business card (e.g., the person was promoted from "Senior Engineer" to "VP of Engineering", and CleanAndDedupNode identifies the duplicate and reuses the original card.ID), PostgresStore.Save performs an atomic Upsert using PostgreSQL's native ON CONFLICT (id) DO UPDATE (clause.OnConflict{UpdateAll: true}): if err := s.db.WithContext(ctx).Clauses(clause.OnConflict{ Columns: []clause.Column, UpdateAll: true, }).Create(rec).Error; err != nil { return fmt.Errorf("upsert business card %q: %w", cp.ID, err) } 3. The store.NewStore(ctx) Factory: Auto-sensing Cloud SQL Unix Sockets and Environment Variables When connecting to Cloud SQL from Cloud Run or Agent Runtime, the official recommendation is to use the Unix Domain Socket (/cloudsql/<PROJECT>:<REGION>:<INSTANCE>) established by the Cloud SQL Auth Proxy; locally or on GKE, DATABASE_URL or DB_HOST are common. We support all three modes in BuildPostgresDSNFromEnv() and store.NewStore(ctx), and automatically fall back seamlessly to MemoryStore preloaded with 4 Taiwan sample business cards when no database environment variables are set: // BuildPostgresDSNFromEnv resolves a PostgreSQL / Cloud SQL connection string from environment variables. // Priority: // 1. CARD_STORE_DSN / DATABASE_URL / POSTGRES_DSN // 2. Cloud SQL Unix socket when INSTANCE_CONNECTION_NAME + DB_PASS are set // 3. Standard TCP host when DB_HOST + DB_PASS are set func BuildPostgresDSNFromEnv() string { for _, key := range []string{"CARD_STORE_DSN", "DATABASE_URL", "POSTGRES_DSN"} { if v := strings.TrimSpace(os.Getenv(key)); v != "" { return v } } dbUser := envOrDefault("DB_USER", "postgres") dbName := envOrDefault("DB_NAME", "postgres") dbPass := strings.TrimSpace(os.Getenv("DB_PASS")) instanceConn := strings.TrimSpace(os.Getenv("INSTANCE_CONNECTION_NAME")) if instanceConn != "" && dbPass != "" { return fmt.Sprintf( "host=/cloudsql/%s user=%s password=%s dbname=%s sslmode=disable", instanceConn, dbUser, dbPass, dbName, ) } // ... } This way, as long as your environment has INSTANCE_CONNECTION_NAME and DB_PASS set for SESSION_TYPE=cloud_sql, SessionService and CardStore can share the same Cloud SQL PostgreSQL instance, without spending an extra penny on database costs! - * * 3. Implementation Breakdown (2): Building a Stateless LINE Webhook Gateway and Remote AgentRuntimeClient (internal/gateway/client.go) — Issue #6 Next is the core engineering of Phase 3: How to make the cmd/server LINE Bot Webhook Handler forward all business card recognition, HITL recovery, search, and icebreaker requests to a remote Agent Runtime via HTTP instead of running workflow.Engine locally? Remember how we had the foresight to separate the Webhook Handler and Workflow Engine with a linebot.WorkflowRunner interface in the first version of internal/linebot/handler.go? // internal/linebot/handler.go type WorkflowRunner interface { HasPendingHITL(userID string) bool RecordUserMessage(userID, text string) StartIngest(ctx context.Context, userID string, imageBytes []byte, recentMessages []string) (*WorkflowResponse, error) ResumeHITL(ctx context.Context, userID string, userReply string) (*WorkflowResponse, error) RunSearch(ctx context.Context, userID string, query string) (*WorkflowResponse, error) RunEmailDraft(ctx context.Context, userID string, cardID string) (*WorkflowResponse, error) } Because of this clean interface, we don't need to change a single line of code in linebot.Handler! We just need to implement a gateway.Client that satisfies the linebot.WorkflowRunner interface in internal/gateway/client.go! 1. gateway.NewClient Supporting Automatic Google Cloud ADC Token Issuance When the LINE Webhook Gateway on Cloud Run needs to call an IAM-protected remote Agent Runtime (or another ADK REST Server on an internal Cloud Run), gateway.NewClient automatically obtains the current Service Account's Google Cloud ADC credentials via google.golang.org/api/idtoken (or golang.org/x/oauth2/google) and attaches them to the Authorization: Bearer <token> header of every HTTP request: func NewClient(ctx context.Context, baseURL string, useADC bool, opts ...ClientOption) (*Client, error) { trimmed := strings.TrimRight(strings.TrimSpace(baseURL), "/") if trimmed == "" { return nil, fmt.Errorf("agent runtime baseURL cannot be empty") } if !strings.HasSuffix(trimmed, "/api") { trimmed += "/api" } hc := &http.Client{Timeout: 60 * time.Second} if useADC { if idClient, err := idtoken.NewClient(ctx, strings.TrimSuffix(trimmed, "/api")); err == nil { hc = idClient } else if oauthClient, err := google.DefaultClient(ctx, "https://www.googleapis.com/auth/cloud-platform"); err == nil { hc = oauthClient } } // ... } 2. Cross-Layer Image Forwarding: Gateway Downloads Image $\rightarrow$ Agent Runtime Automatically Writes to GCS Artifact When a user uploads a business card photo on LINE: 1. Gateway downloads imageBytes from LINE Blob API. 2. gateway.Client.StartIngest calls the remote ADK REST API POST /api/apps/{appName}/users/{userID}/sessions/{sessionID} to create a session. 3. Then it calls POST /api/run, wrapping imageBytes in the InlineData of newMessage.Parts to send to the remote Agent Runtime: func (c *Client) StartIngest(ctx context.Context, userID string, imageBytes []byte, recentMessages []string) (*linebot.WorkflowResponse, error) { sessionID := "ingest-" + uuid.NewString() initialState := map[string]any{ workflow.StateKeyInputType: workflow.RouteIngest, workflow.StateKeyUserID: userID, workflow.StateKeyRecentMessages: recentMessages, } if err := c.createSession(ctx, userID, sessionID, initialState); err != nil { return nil, err } parts := []*genai.Part{ {Text: workflow.RouteIngest}, } if len(imageBytes) > 0 { parts = append(parts, &genai.Part{ InlineData: &genai.Blob{ MIMEType: "image/jpeg", Data: imageBytes, }, }) } msg := &genai.Content{ Role: genai.RoleUser, Parts: parts, } res, err := c.runTurn(ctx, userID, sessionID, msg, initialState) // ... } At this point, the logic we wrote for RouterNode in Phase 2 works like magic: when the remote Agent Runtime's RouterNode receives this request with InlineData, it automatically calls ctx.Artifacts().Save(ctx, "business_card.jpg", ...) to store the image in the GCS Artifact Bucket, leaving only lightweight card_image_artifact and card_image_version in Session.State! This means the LINE Webhook Gateway doesn't even need direct write permissions to the GCS Bucket for the remote Agent Runtime to complete versioned GCS Artifact archiving! 3. Cross-Layer Stateless HITL Breakpoint Discovery and Recovery (resolvePendingHITL + ResumeHITL) The most exciting part: when a user sees the LINE Flex preview card and inputs "_Change title to CTO_" 5 minutes later, and this Webhook request lands on another completely new Cloud Run Gateway instance, how does gateway.Client know this user has a pending HITL Session on the remote Agent Runtime? We use the GET /api/apps/{appName}/users/{userID}/sessions endpoint via the ADK REST API in gateway.Client.resolvePendingHITL to query the remote Agent Runtime for the user's session status and check state["active_interrupt_id"]: func (c *Client) resolvePendingHITL(ctx context.Context, userID string) (pendingHITL, bool) { // 1. If local cache has SessionID, call remote GET /sessions/{id} to get the latest active_interrupt_id // (Important! Because after each Edit loop, hitl_seq increments to hitl_confirm_2, hitl_confirm_3...) c.mu.RLock() cached, hasCached := c.pendingSession[userID] c.mu.RUnlock() if hasCached && cached.SessionID != "" { if fullSess, err := c.getSession(ctx, userID, cached.SessionID); err == nil && fullSess != nil { if activeID := readStringFromMap(fullSess.State, workflow.StateKeyActiveInterruptID); activeID != "" { found := pendingHITL{SessionID: cached.SessionID, InterruptID: activeID} c.mu.Lock() c.pendingSession[userID] = found c.mu.Unlock() return found, true } } } // 2. If this is a brand-new stateless Gateway instance, call remote GET /sessions to find sessions with active_interrupt_id sessions, err := c.listSessions(ctx, userID) if err != nil { return pendingHITL{}, false } for _, sess := range sessions { activeID := readStringFromMap(sess.State, workflow.StateKeyActiveInterruptID) if activeID == "" && sess.ID != "" { if fullSess, err := c.getSession(ctx, userID, sess.ID); err == nil && fullSess != nil { activeID = readStringFromMap(fullSess.State, workflow.StateKeyActiveInterruptID) } } if activeID != "" { found := pendingHITL{SessionID: sess.ID, InterruptID: activeID} c.mu.Lock() c.pendingSession[userID] = found c.mu.Unlock() return found, true } } return pendingHITL{}, false } Here is a real bug and key detail we caught while writing integration tests: - When the user uploads a business card in Turn 1, the first breakpoint ID generated by HITLConfirmNode is hitl_confirm_1. - When the user inputs "_Change title to CTO_" in Turn 2 (possibly handled by Gateway Instance 2), ApplyEditNode modifies the title and loops back to HITLConfirmNode to pause again, incrementing the breakpoint ID to hitl_confirm_2! - If Turn 3 (user clicks "✅ Confirm Save") happens to go back to Gateway Instance 1, and Instance 1 blindly trusts its old local cache (hitl_confirm_1), the sent FunctionResponse won't match the latest breakpoint hitl_confirm_2! - Therefore, in resolvePendingHITL, we ensure that even if the local cache hits a SessionID, it synchronizes the latest active_interrupt_id from the remote via getSession. This way, no matter how many times the request jumps between Gateway instances or how many fields are modified, it always aligns perfectly with the latest InterruptID! 4. cmd/server/main.go: Seamless Switching Between Monolithic and Two-Tier Gateway Modes Finally, we added mode detection in cmd/server/main.go: var runner linebot.WorkflowRunner if cfg.AgentRuntimeURL != "" { // 🌐 Mode A: Decoupled Two-Tier LINE Webhook Gateway Mode (Forwarding to remote Agent Runtime) gwClient, err := gateway.NewClient( ctx, cfg.AgentRuntimeURL, cfg.AgentRuntimeUseADC, gateway.WithAppName(cfg.AgentRuntimeAppName), ) if err != nil { slog.Error("failed to initialize remote Agent Runtime gateway client", "url", cfg.AgentRuntimeURL, "err", err) os.Exit(1) } slog.Info("running in decoupled LINE Webhook Gateway mode", "agent_runtime_url", cfg.AgentRuntimeURL, "agent_runtime_app", cfg.AgentRuntimeAppName, "use_adc", cfg.AgentRuntimeUseADC, ) runner = gwClient } else { // 🚀 Mode B: All-in-One Local Engine Mode (Default for local development and single container deployment) cardStore, err := store.NewStore(ctx) // ... Initialize local workflow.NewEngine ... runner = engine.LinebotRunner() } botHandler, err := linebot.NewHandler(cfg, runner) - * * 4. End-to-End Two-Tier Architecture Integration Testing (internal/gateway/client_test.go) To ensure gateway.Client is 100% compatible with the official ADK Go v2.6.0 server/adkrest REST server, we started a real adkrest.NewServer (mounting our app.NewRootAgentWithDeps) using httptest.NewServer in internal/gateway/client_test.go and established two completely independent Gateway Clients (client1 and client2) for a five-stage end-to-end test: 1. Turn 1 (client1.StartIngest): Upload business card photo via HTTP $\rightarrow$ Verify remote ArtifactService successfully writes business_card.jpg $\rightarrow$ Verify return of WaitingHITL=true (hitl_confirm_1) and LINE Flex Preview JSON containing "Wang Daming". 2. Cross-Instance State Probing (client2.HasPendingHITL): The brand-new stateless client2 probes the paused session created by client1 via GET /api/apps/app/users/{user}/sessions. 3. Turn 2 (client2.ResumeHITL — Natural Language Modification): client2 sends "_Change title to CTO_" $\rightarrow$ Verify remote execution of ApplyEditNode, updating DraftCard.Title == "CTO", and pausing again at hitl_confirm_2. 4. Turn 3 (client1.ResumeHITL — Confirm Save): Switch back to client1 and send "action=confirm_save" $\rightarrow$ client1 automatically synchronizes the latest breakpoint hitl_confirm_2 and wakes up SaveCardNode $\rightarrow$ Verify business card successfully written to remote CardStore and returns AI icebreaker. 5. Turn 4 & Turn 5 (client2.RunSearch & client2.RunEmailDraft): Call remote hybrid search and individual icebreaker drafting via HTTP, verifying return of Flex Carousel JSON and icebreaker content. Running the full test suite: $ go test -race -v ./... === RUN TestGatewayClient_EndToEndDecoupledIngestHITLAndSearch --- PASS: TestGatewayClient_EndToEndDecoupledIngestHITLAndSearch (0.01s) === RUN TestBusinessCardRecordRoundTrip --- PASS: TestBusinessCardRecordRoundTrip (0.00s) === RUN TestBuildPostgresDSNFromEnv --- PASS: TestBuildPostgresDSNFromEnv (0.00s) === RUN TestNewStoreFactoryFallback --- PASS: TestNewStoreFactoryFallback (0.00s) === RUN TestRankCardsByKeywords --- PASS: TestRankCardsByKeywords (0.00s) PASS All unit tests and cross-layer integration tests passed with green lights under -race detector! - * * 5. Summary and Roadmap Progress Check After completing PR #12, looking back at the five stages of Roadmap Issue #9, Phase 1, Phase 2, and Phase 3 are now 100% complete! - <input type="checkbox" disabled checked>Phase 1 (agents-cli & adk_go Scaffolding): Completed Issues #1, #2 (PR #10) - <input type="checkbox" disabled checked>Phase 2 (Agent Runtime Adaptation and Cloud Persistence Iron Triangle): Completed Issues #3, #4 (PR #11) and Issue #5 (PR #12) - <input type="checkbox" disabled checked>Phase 3 (LINE Webhook Gateway and Agent Runtime Two-Tier Decoupled Architecture): Completed Issue #6 (PR #12) - <input type="checkbox" disabled>Phase 4 (agents-cli eval Automated Evaluation and Benchmarking): Issue #7 - <input type="checkbox" disabled>Phase 5 (Agent Runtime Official Deployment, Terraform IaC, and Cloud Trace Observability): Issue #8 In the upcoming articles of this series, we will enter Phase 4 and Phase 5, using agents-cli eval to establish a Quality Flywheel for this 12-node Graph Workflow, and officially deploy the entire system using agents-cli deploy and OpenTelemetry. Stay tuned! - * * Resources and Further Reading - "Agent Platform in Action" Series Articles: - [[Agent Platform in Action (1)] Building a LINE AI Business Card Assistant with Go and Google ADK 2.0 Graph Workflow: From Fan-Out Parallel Search to HITL Pause Loop Correction](/2026/10/16/adk-graph-workflow-go/) - [[Agent Platform in Action (2)] From Monolithic Cloud Run to Google Cloud Agent Runtime: Refactoring LINE AI Business Card Assistant with agents-cli and ADK Go](/2026/10/17/adk-graph-workflow-go-2/) - [[Agent Platform in Action (3)] Solving HITL Amnesia in Stateless Containers: Implementing Cross-Instance Business Card Recognition and Pause/Resume with ADK 2.0 GCS Artifact Service and Agent Platform Sessions](/2026/10/20/adk-graph-workflow-go-3/) - Project Open Source Code and PRs: - GitHub Repository: kkdai/line-card-assistant - Phase 2 & 3 Implementation Pull Request: PR #12: feat(phase2-3): Cloud SQL PostgresStore (#5) & Decoupled LINE Webhook Gateway (#6) - Completed Issues: Issue #5, Issue #6, Roadmap #9 - Google ADK and Agent Platform Official Documentation: - ADK Documentation (llms.txt) - Getting Started - agents-cli

[Agent Platform in Practice (5)] Building an AI Agent Quality Flywheel: Automated Benchmarking for LINE AI Business Card King wi !Phase 4: ADK 2.0 Go Agent Evaluation & Quality Flywheel (agents-cli eval) Recap In the first four articles of the "Agent Platform in Action" series, we have step-by-step advanced the LINE AI Business Card Master (line-card-assistant) from a standalone prototype to a cloud-native, dual-layer decoupled Google Cloud Agent Platform architecture: - [[Agent Platform in Action (1)] Building LINE AI Business Card Master with Go and Google ADK 2.0 Graph Workflow](/2026/10/16/adk-graph-workflow-go/): Used google.golang.org/adk/v2/workflow to build a 12-node business card processing graph (Graph Workflow), covering Vision OCR, AddFanOut / AddFanIn parallel background investigations, and the HITL (Human-in-the-Loop) pause and modification loop via NewRequestInputEvent. - [[Agent Platform in Action (2)] From Monolithic Cloud Run to Google Cloud Agent Runtime: Refactoring LINE AI Business Card Master with agents-cli and ADK Go](/2026/10/17/adk-graph-workflow-go-2/): Completed Phase 1 (PR #10), introducing the agents-cli (adk_go) standard dual-channel architecture, supporting LINE Webhook, ADK WebUI Playground, and the A2A protocol. - [[Agent Platform in Action (3)] Solving HITL Amnesia in Stateless Containers: Implementing Cross-Instance Recognition and Pause/Resume with ADK 2.0 GCS Artifact Service and Agent Platform Sessions](/2026/10/20/adk-graph-workflow-go-3/): Completed the first half of Phase 2 (PR #11), using artifact/gcsartifact and session/vertexai to enable cross-instance HITL pause and resume for stateless containers. - [[Agent Platform in Action (4)] Towards Cloud Dual-Layer Decoupling and Full Persistence: Completing the Architecture with Cloud Run LINE Webhook Gateway, Agent Runtime, and Cloud SQL PostgreSQL](/2026/10/21/adk-graph-workflow-go-4/): Completed the second half of Phase 2 (Issue #5) and Phase 3 (Issue #6, PR #12), migrating the business card database to Cloud SQL PostgreSQL (PostgresStore) and implementing the stateless Cloud Run LINE Webhook Gateway (gateway.Client). With the architecture and features in place and ready to officially deploy the Agent to the cloud production environment (Phase 5), every AI engineering team faces a core soul-searching question: > "When we modify a line of Prompt, adjust a Graph Node's routing condition, or upgrade the underlying model from gemini-2.5-flash to gemini-3.8-flash, how do we ensure that business card OCR fields aren't missed, duplicate card promotion detection doesn't fail, HITL modification loops don't get stuck, and hybrid search doesn't degrade?" In today's fifth episode of the series, we are going to implement Phase 4 (Issue #7, corresponding to PR #13): using agents-cli eval and ADK 2.0 Go trajectory testing to build a quantifiable, comparable, and CI/CD-automatable AI Agent Quality Flywheel for the LINE AI Business Card Master! - * * I. Why Traditional Unit Tests Aren't Enough? Understanding the agents-cli eval "Quality Flywheel" In traditional software engineering, we are used to writing deterministic unit tests like if got != want. However, in an AI Agent system driven by ADK 2.0 Graph Workflow + Multimodal LLM, what we need to verify is far more than just the return value of a single function; it is the entire "Execution Trajectory and State Evolution (State Delta)": 1. Multi-branch and Parallel Convergence (Fan-Out / Fan-In Trajectory): When a business card image is received, does the RouterNode correctly enter the VisionOCRNode? Subsequently, are the three parallel branches CleanAndDedupNode, CompanyEnrichNode, and ContextMatchNode triggered simultaneously and converged at JoinPreviewNode? 2. Intermediate State Changes (Session state_delta): Even if the final return to the user is a LINE Flex Message JSON, did the intermediate nodes correctly normalize the Taiwan mobile number (+886-912-345-678) and detect an existing contact's promotion (IsDuplicate=true, VP of Engineering $\rightarrow$ Chief Technology Officer (CTO))? 3. Human-in-the-Loop (HITL) Pause and Multi-turn Dialogue Loop: After the HITLConfirmNode issues an adk_request_input pause event, if the user replies "Change title to Technical VP, change occasion to COMPUTEX 2026 AI Forum" and confirms the save, does the workflow actually traverse ApplyEditNode $\rightarrow$ BuildPreviewNode $\rightarrow$ HITLConfirmNode $\rightarrow$ SaveCardNode? 4. Retrieval Degradation and Grounding: When a strict HybridSearchNode finds no results (0 hits), does it automatically degrade to trigger BroadSearchNode? Does the generated icebreaker follow-up email draft truly reference the meeting occasion and company background from the business card without hallucination? To systematically solve this problem, Google Cloud agents-cli proposes a five-step Quality Flywheel workflow: flowchart LR A["1. Generate<br/>(agents-cli eval generate)<br/>Start ADK server and collect real trajectories"] --> B["2. Grade<br/>(agents-cli eval grade)<br/>Execute Trajectory and quality metric scoring"] B --> C["3. Analyze<br/>Review scores for each metric and<br/>Explanation for failed cases"] C --> D["4. Fix<br/>Fix Graph Node, Prompt<br/>or event tracking structure"] D --> E["5. Compare<br/>(agents-cli eval compare)<br/>Baseline vs. Candidate quantitative comparison"] E -->|Continuous Iteration| A - * * II. Revealing How agents-cli eval generate Tests a Go-based ADK 2.0 Agent? Many Go developers new to agents-cli wonder: "The agents-cli is a CLI tool written in Python; how can it evaluate my adk/v2/workflow Agent written in Go?" The answer lies within the main.go and appinfo/appinfo.go we introduced in Phase 1! When you execute: DRY_RUN=true agents-cli eval generate --dataset eval/datasets/basic-dataset.json agents-cli automatically completes the following four steps in the background: 1. Automatically compile and start your Go Agent server: agents-cli reads the agents-cli-manifest.yaml in the project root (identified as template: adk_go) and automatically starts in the background: go run . web --port <random_port> api -path_prefix / a2a appinfo 2. Introspect Agent structure via GET /apps/app/app-info reflection: appinfo/appinfo.go uses Go's reflect and unsafe to traverse the Edges and Nodes of the entire workflowagent, assembling the embedded CompanyEnrichAgent (llmagent), Instructions, and tool declarations into a standard AppInfo JSON returned to agents-cli. 3. Create Sessions one by one and call POST /run_sse to collect event streams: For each test case (eval_case_id) in eval/datasets/basic-dataset.json, agents-cli first calls POST /apps/app/users/eval_user/sessions to create an independent Session, then sends the test case's prompt to POST /run_sse, and records every ADK Event (including author, content, actions.state_delta, actions.requested_tool_confirmations, etc.) streamed back from the server. 4. Output standardized trace file artifacts/traces/traces_<timestamp>.json: After testing is complete, it automatically shuts down the Go server and saves the full trajectory as a JSON, passing it to the next stage, agents-cli eval grade, for scoring. - * * III. Practical Design: A Golden Evaluation Set Covering 4 Core Scenarios (eval/datasets/basic-dataset.json) To fully cover the four core acceptance scenarios defined in Issue #7, we designed 6 Golden Eval Cases in eval/datasets/basic-dataset.json, marking the expected node trajectory (expected_nodes) and state assertions in each case's custom_Properties: | Eval Case ID | Corresponding Core Scenario | Input Prompt (Simulated Channel Signal) | Verification Focus (custom_Properties) | | --- | --- | --- | --- | | case_1_ingest_fanout_fanin_ocr | Scenario 1: Business Card Recognition and Parallel Enhancement | action=ingest&context=2026 Cloud Summit Taipei | Verify RouterNode $\rightarrow$ VisionOCRNode $\rightarrow$ Parallel Fan-Out (CleanAndDedupNode, CompanyEnrichNode, ContextMatchNode) $\rightarrow$ BuildPreviewNode $\rightarrow$ HITLConfirmNode pause, and phone normalization to +886-912-345-678. | | case_2_duplicate_promotion_detection | Scenario 2: Duplicate Card and Promotion Detection | action=ingest&card=promotion_david_chen_appier&context=2026 AI DevFest Taipei | Test database already contains David Chen from Appier (VP of Engineering); when a new card (Chief Technology Officer (CTO)) is received, CleanAndDedupNode must mark IsDuplicate=true and generate a promotion alert message. | | case_3_hitl_edit_and_confirm_loop | Scenario 3: HITL Loop Correction and Saving | action=ingest&context=2026 Cloud Summit Taipei&edit=Change title to Technical VP, change occasion to COMPUTEX 2026 AI Forum&hitl_action=confirm | Verify from OCR preview $\rightarrow$ ApplyEditNode semantic modification of title and occasion $\rightarrow$ BuildPreviewNode re-rendering preview $\rightarrow$ HITLConfirmNode confirmation $\rightarrow$ SaveCardNode writing to database and generating follow-up email. | | case_4a_hybrid_search_cloud_summit_ai | Scenario 4a: Natural Language Hybrid Retrieval | Help me find people doing AI I met at Cloud Summit | Verify QueryIntentNode converts colloquial speech to SearchFilter $\rightarrow$ HybridSearchNode hits 2 business cards $\rightarrow$ FormatCarouselNode outputs Flex Carousel JSON. | | case_4b_broad_search_fallback | Scenario 4b: 0 Hits Automatic Degradation to Fuzzy Retrieval | Help me find Zhang San who does quantum finance blockchain I met at Tokyo Game Show | Verify that when HybridSearchNode finds no results (0 hits), conditional routing automatically degrades to BroadSearchNode for Token fuzzy retrieval to avoid blank replies. | | case_4c_personalized_icebreaker_email | Scenario 4c: Personalized Icebreaker Follow-up Email | action=draft_email&card_id=card-seed-1 | Verify EmailDraftNode writes a hallucination-free business icebreaker email based on the specified card's MetContext (2026 Cloud Summit Taipei) and CompanyInfo (Google Cloud Taiwan). | - * * IV. Real-world Pitfalls and Debugging: Why was the Trajectory Score 0.0000 on the First agents-cli eval grade Run? After writing eval/eval_config.yaml and eval/metrics.py, we confidently ran the first Baseline evaluation: DRY_RUN=true agents-cli eval run --config eval/eval_config.yaml The first Baseline scorecard (results_20261022_223104.json) printed in the terminal shocked us: ╭─ Evaluation Summary ───────────────────────────╮ │ Metric Score Valid │ │ ────────────────────────────────────────────── │ │ workflow_trajectory_accuracy 0.0000 6/6 │ │ ocr_and_enrichment_quality 1.0000 6/6 │ │ duplicate_and_hitl_precision 0.7333 6/6 │ │ search_and_email_grounding 1.0000 6/6 │ ╰────────────────────────────────────────────────╯ At the same time, a very critical warning message appeared in the agents-cli eval grade execution log: > Dropped 26 event(s) with no content; the eval service rejects them 🔍 Root Cause: It turns out drop_empty_content_events filtered out intermediate node events! Why were 26 events dropped? We immediately checked the implementation of ADK Go workflow.FunctionNode and the event preprocessing logic of agents-cli: 1. In our original internal/workflow/nodes_ingest.go and internal/workflow/nodes_search.go, Intermediate Nodes like RouterNode, VisionOCRNode, CleanAndDedupNode, CompanyEnrichNode, ContextMatchNode, and QueryIntentNode were only responsible for writing results into ev.Actions.StateDelta (e.g., ev.Actions.StateDelta["draft_card"] = ...), while keeping ev.Content as nil; only the final output node (such as BuildPreviewNode, SaveCardNode, FormatCarouselNode) would set ev.Content. 2. However, before agents-cli eval grade sends the trajectory to the grader, the underlying drop_empty_content_events()—to comply with the Schema constraints of the cloud Vertex AI Eval Service—will directly remove all events where event.content == null from agent_data.turns[].events! 3. Once intermediate node events are removed because content == nil, the actions.state_delta and node execution records attached to those events also disappear! This caused the grader to only see the output of the last node, resulting in a workflow_trajectory_accuracy of 0.0000! 🛠️ Elegant Solution: Adding Structured Trajectory Summaries setNodeTrace to Intermediate Nodes Once the root cause was found, the solution was clear and elegant! We designed a lightweight helper function setNodeTrace in internal/workflow/nodes_ingest.go: // setNodeTrace attaches a concise model text part to intermediate workflow node // events so that `agents-cli eval` (which drops events with empty Content) // retains the full node execution trajectory and state_delta history. func setNodeTrace(ev *session.Event, nodeName, summary string) { if ev == nil || ev.Content != nil { return } ev.Content = genai.NewContentFromText(fmt.Sprintf("[%s] %s", nodeName, summary), genai.RoleModel) } Then, we call setNodeTrace before each intermediate node finishes, for example: - RouterNode: setNodeTrace(ev, "RouterNode", "route="+route) - VisionOCRNode: setNodeTrace(ev, "VisionOCRNode", fmt.Sprintf("ocr extracted name=%s company=%s title=%s", card.Name, card.Company, card.Title)) - CleanAndDedupNode: setNodeTrace(ev, "CleanAndDedupNode", fmt.Sprintf("phone=%s is_duplicate=%v", card.Phone, isDup)) - CompanyEnrichNode: setNodeTrace(ev, "CompanyEnrichNode", fmt.Sprintf("company_info=%s tags=%v", card.CompanyInfo, card.Tags)) - ContextMatchNode: setNodeTrace(ev, "ContextMatchNode", "met_context="+card.MetContext) - QueryIntentNode / HybridSearchNode / BroadSearchNode: Record the converted SearchFilter and the number of business cards hit. At the same time, to prevent the [NodeName] ... trajectory text of intermediate nodes from overwriting the ReplyText that Engine.executeTurn ultimately returns to the LINE caller, we simultaneously strengthened the state convergence logic in internal/workflow/graph.go—allowing TurnResult to read authoritative StateKeyReplyText, StateKeyFlexPreviewJSON, StateKeyFlexCarouselJSON, and StateKeyDraftEmail directly from the final converged Session.State! - * * V. Turning the Quality Flywheel: From Baseline to Candidate, All Four Metrics Reached 1.0000! After adding setNodeTrace and supporting single-turn batch HITL modification verification, we re-executed DRY_RUN=true agents-cli eval run --config eval/eval_config.yaml: Loaded 6 cases from eval/datasets/basic-dataset.json Starting local Go ADK server... Running 6 cases against http://127.0.0.1:56649 (app=app, concurrency=4)... Wrote 6 traces to artifacts/traces/traces_20261022_223515.json Loaded 6 trace(s) from artifacts/traces/traces_20261022_223515.json Evaluating metrics: ['workflow_trajectory_accuracy', 'ocr_and_enrichment_quality', 'duplicate_and_hitl_precision', 'search_and_email_grounding'] ╭─ Evaluation Summary ───────────────────────────╮ │ Metric Score Valid │ │ ────────────────────────────────────────────── │ │ workflow_trajectory_accuracy 1.0000 6/6 │ │ ocr_and_enrichment_quality 1.0000 6/6 │ │ duplicate_and_hitl_precision 1.0000 6/6 │ │ search_and_email_grounding 1.0000 6/6 │ ╰────────────────────────────────────────────────╯ This time, Dropped 0 event(s), and all intermediate node trajectories and state_delta were 100% preserved! We immediately used the 5th step of the Quality Flywheel, agents-cli eval compare, to quantitatively compare the Baseline before the fix with the Candidate after the fix: agents-cli eval compare \ --baseline artifacts/grade_results/results_20261022_223104.json \ --candidate artifacts/grade_results/results_20261022_223515.json ╭─ Summary Comparison ─────────────────────────────────────────────────────────╮ │ Metric Baseline Candidate Delta │ │ ──────────────────────────────────────────────────────────────────────────── │ │ duplicate_and_hitl_precision 0.7333 1.0000 +0.2667 │ │ ocr_and_enrichment_quality 1.0000 1.0000 0.0000 │ │ search_and_email_grounding 1.0000 1.0000 0.0000 │ │ workflow_trajectory_accuracy 0.0000 1.0000 +1.0000 │ ╰──────────────────────────────────────────────────────────────────────────────╯ Advancing from 0.0000 / 0.7333 to all four metrics reaching 1.0000 (+1.0000 / +0.2667) is exactly the most charming value of the agents-cli eval quality flywheel—making every architectural adjustment traceable and provable! - * * VI. Dual-Track Verification: Python Custom Graders (eval/metrics.py) + Go Native Multi-turn Trajectory Tests (eval/eval_test.go) In the eval/ directory, we adopted a "Python agents-cli eval + Go go test dual-track verification architecture": 1. eval/eval_config.yaml and eval/metrics.py: Four Custom Evaluation Metrics In eval/eval_config.yaml, we registered 4 deterministic trajectory and quality graders (plus 1 card_assistant_llm_rubric available for online Vertex AI Judge models): metrics_to_run: - workflow_trajectory_accuracy - ocr_and_enrichment_quality - duplicate_and_hitl_precision - search_and_email_grounding custom_metrics: workflow_trajectory_accuracy: type: custom module: eval.metrics function: evaluate_workflow_trajectory ocr_and_enrichment_quality: type: custom module: eval.metrics function: evaluate_ocr_and_enrichment duplicate_and_hitl_precision: type: custom module: eval.metrics function: evaluate_duplicate_and_hitl search_and_email_grounding: type: custom module: eval.metrics function: evaluate_search_and_email In eval/metrics.py, we defined helper functions _extract_executed_nodes(instance) and _merged_state_delta(instance) to parse the actual node sequence traversed by the Graph Workflow and the cumulative Session State directly from agent_data.turns[].events: def _extract_executed_nodes(instance: dict[str, Any]) -> list[str]: """Extract executed workflow node names in order from `[NodeName] ...` traces.""" nodes: list[str] = [] for ev in _iter_events(instance): content = ev.get("content") or {} for part in content.get("parts") or []: if isinstance(part, dict) and isinstance(part.get("text"), str): for match in NODE_TRACE_PATTERN.findall(part["text"]): if not nodes or nodes[-1] != match: nodes.append(match) return nodes This way, evaluate_workflow_trajectory can accurately check: - Whether each node was executed in topological order (Subsequence Order)? - Whether all expected_state_keys (such as draft_card, flex_preview_json, saved_card_id, search_results) were written to state_delta? - Whether the final output simultaneously satisfies expected_response_substrings and excludes forbidden_substrings? 2. eval/eval_test.go: Go Native Multi-turn HITL Pause/Resume and /app-info Introspection Tests In addition to agents-cli eval called via HTTP SSE, we also wrote Go native TestEvalDatasetAndAppInfoIntrospection and TestPhase4WorkflowTrajectorySuite in eval/eval_test.go, which can be executed in milliseconds via go test ./... or GitHub Actions CI: - TestEvalDatasetAndAppInfoIntrospection: Verifies the Schema integrity of eval/datasets/basic-dataset.json and directly calls appinfo.BuildAppInfo to confirm that the CompanyEnrichAgent within the line_card_assistant graph can be correctly introspected via reflection by /apps/{app}/app-info. - TestPhase4WorkflowTrajectorySuite: - Case1_IngestFanOutFanInOCR - Case2_DuplicatePromotionDetection - Case3_MultiTurnHITLEditAndConfirmLoop (Actually calls StartIngest $\rightarrow$ ResumeHITL(edit) $\rightarrow$ ResumeHITL(confirm) in three turns to verify the cross-turn Pause/Resume state machine) - Case4_SearchHybridAndBroadFallbackAndEmailDraft Now, with just one line make eval, you can run both agents-cli eval run and the Go native test suite: $ make eval DRY_RUN=true agents-cli eval run --config eval/eval_config.yaml ... go test -v ./eval/... === RUN TestEvalDatasetAndAppInfoIntrospection --- PASS: TestEvalDatasetAndAppInfoIntrospection (0.00s) === RUN TestPhase4WorkflowTrajectorySuite --- PASS: TestPhase4WorkflowTrajectorySuite/Case1_IngestFanOutFanInOCR (0.00s) --- PASS: TestPhase4WorkflowTrajectorySuite/Case2_DuplicatePromotionDetection (0.00s) --- PASS: TestPhase4WorkflowTrajectorySuite/Case3_MultiTurnHITLEditAndConfirmLoop (0.00s) --- PASS: TestPhase4WorkflowTrajectorySuite/Case4_SearchHybridAndBroadFallbackAndEmailDraft (0.00s) PASS ok github.com/kkdai/line-card-assistant/eval 0.436s - * * VII. Summary and Next Preview (Phase 5 Final Chapter: Official Deployment, Terraform IaC, and Cloud Trace Observability) As of now, our LINE AI Business Card Master Roadmap (Issue #9) has completed the first four major stages: - <input type="checkbox" disabled checked> Phase 1: agents-cli environment and adk_go project Scaffolding (PR #10) - <input type="checkbox" disabled checked> Phase 2: ADK Go Graph Workflow adaptation for Agent Runtime, GCS Artifacts, Agent Platform Sessions, and Cloud SQL PostgresStore (PR #11, PR #12) - <input type="checkbox" disabled checked> Phase 3: Cloud Run LINE Webhook Gateway and Agent Runtime dual-layer decoupled architecture (PR #12) - <input type="checkbox" disabled checked> Phase 4: agents-cli eval automated evaluation and quality flywheel (PR #13) In the next article (which is also the final chapter of this series, Phase 5 — Issue #8), we will officially push the LINE AI Business Card Master, which passed the evaluation with a perfect 1.0000 score, into the production environment via agents-cli deploy. We will also fully integrate Terraform IaC (Infrastructure as Code), GitHub Actions CI/CD automated evaluation and deployment pipelines, and OpenTelemetry + Google Cloud Trace full-link observability. Stay tuned! - * * Reference Resources - GitHub Repository: kkdai/line-card-assistant - Phase 4 Pull Request: PR #13 — feat(eval): add Phase 4 agents-cli eval suite and Go workflow trajectory tests (#7) - Google Cloud agents-cli Official Documentation: The Quality Flywheel & Agent Evaluation - Google ADK Official Documentation: Graph-based Agent Workflows (adk/v2/workflow)

[Agent Platform in Practice (2)] From Monolithic Cloud Run to Google Cloud Agent Runtime: Refactoring LINE AI Business Card King !LINE Card Assistant + Google Cloud Agent Platform Architecture Recap In the previous article, "[[Agent Platform in Practice (1)] Building a LINE AI Business Card Assistant with Go and Google ADK 2.0 Graph Workflow](/2026/10/16/adk-graph-workflow-go/)", we used Golang (google.golang.org/adk/v2/workflow) and the LINE Messaging API SDK v8 to build line-card-assistant (LINE AI Business Card Assistant) from scratch. We successfully implemented and deployed the four core capabilities of ADK 2.0 Graph Workflow—Deterministic Router, AddFanOut / AddFanIn for parallel background investigation and aggregation, and NewRequestInputEvent for Human-in-the-Loop (HITL) pause and loop correction—to Google Cloud Run. However, as we evolve this monolithic LINE Bot into a Production-grade Agent Platform, we immediately encounter several architectural questions: 1. Where is the cross-instance HITL Session state stored? In the first version, we used session.InMemoryService() with local SQLite. When Cloud Run switches container instances due to auto-scaling or redeployment, the HITL pause state (Checkpoint) waiting for the user to click "✅ Confirm Save" disappears due to memory reset. 2. How can this business card Agent be called by other Agents (Agent-to-Agent, A2A)? If the company has another "CRM Sales Assistant Agent" or "Calendar Scheduling Agent" that wants to query contacts in our Business Card Assistant or ask it to write an icebreaker email via a standard A2A protocol, a monolithic server tied to the LINE Webhook /webhook entry point cannot directly provide a standard Agent interface. 3. How to systematically evaluate and observe the entire Graph? After changing a line of Prompt or adjusting a node, how do we ensure that OCR accuracy, Fan-Out parallel tool call trajectories (Tool Trajectory), and search quality haven't degraded? To bridge the gap between "writing an Agent prototype" and "operating an enterprise-grade Agent," Google officially launched a new command-line and skill toolkit—agents-cli (Google Agents CLI), serving as the unified development entry point for the Google Cloud Agent Platform (now integrated as Gemini Enterprise Agent Platform). In this second episode of the series, we will dive deep into the agents-cli official development guide, answer frequently asked questions about Agent Runtime scaling (Scale-to-Zero) and billing mechanisms, and walk you through how we completed the Phase 1 architecture upgrade (PR #10): refactoring the project into a dual-channel architecture that complies with adk_go + agent_runtime specifications without breaking the original LINE Bot functionality! * 1. In-depth Analysis of Google Cloud Agent Platform and the agents-cli Official Guide Before diving into the code, let's clarify Google's latest technical landscape for the Agent Platform through two core documents: "Getting Started - agents-cli" and "The Lifecycle - agents-cli". 1. Product Terminology Comparison: What are Agent Platform and Agent Runtime? If you've followed Google Cloud's AI product line over the past year, you might have heard of Vertex AI Agent Engine, Reasoning Engine, Agent Platform, or Gemini Enterprise Agent Platform (GEAP). In the latest agents-cli v1.10+ specifications, the official documentation has clearly converged these terms and underlying resources: | Common Developer Terms | Google Cloud Official Positioning & Underlying Resources | agents-cli Corresponding Parameter | | --- | --- | --- | | Agent Platform / Gemini Platform | Google Cloud's overall platform for hosting enterprise-grade AI Agents (covering Runtime, Sessions, Memory Bank, Agent Gateway, Agent Registry, and Eval) | Unified operation via agents-cli | | Agent Runtime / Agent Engine | A fully managed container execution environment designed specifically for AI Agents. Underlying GCP resource path: projects/{project}/locations/{location}/reasoningEngines/{id} | --deployment-target agent_runtime | | Agent Platform Sessions | A cloud-managed conversation and state persistence service (ADK automatically binds VertexAiEngineSessionService) | --session-type agent_platform_sessions | | Gemini Enterprise / Agent Registry | An enterprise Agent publishing and directory center, allowing deployed Agents to be discovered and called directly in the Gemini Enterprise interface or via A2A | agents-cli publish gemini | * 2. The Seven-Stage Development Lifecycle of agents-cli In the past, we were used to a "write code → go test → gcloud run deploy" workflow. However, in "The Lifecycle" document, agents-cli defines a Seven-Stage Quality Flywheel that integrates Coding Agent Skills (supporting Antigravity CLI, Gemini CLI, Claude Code, etc.): flowchart LR P0["Phase 0-1\nUnderstand & Recipes\n(.agents-cli-spec.md)"] --> P2["Phase 2\nScaffold\n(create / enhance)"] P2 --> P3["Phase 3\nBuild & Smoke Test\n(run / playground)"] P3 --> P4["Phase 4\nEvaluate\n(eval run Quality Flywheel)"] P4 -->|Iterative Correction| P3 P4 --> P5["Phase 5\nDeploy\n(agent_runtime / cloud_run)"] P5 --> P6["Phase 6-7\nPublish & Observe\n(Cloud Trace / BigQuery)"] 1. Phase 0 & 1 — Understand & Study Recipes: Before writing a single line of code, document the Agent's core goals, external tools, security boundaries, and deployment targets in the project root's .agents-cli-spec.md as the Single Source of Truth for the entire development cycle. 2. Phase 2 — Scaffold: Use agents-cli scaffold create or run agents-cli scaffold enhance . on an existing project to automatically generate the directory structure, configuration files (agents-cli-manifest.yaml), A2A endpoints, Session switchers, and OpenTelemetry observation modules that comply with official standards. - Supports two major official built-in templates: adk (Python) and adk_go (Golang)! 3. Phase 3 — Build & Playground: - Go projects only need to focus on implementing and exporting NewRootAgent(ctx) in app/agent.go. - During development, you can use agents-cli run "your test question" for a quick terminal smoke test, or run agents-cli playground (or make playground) to open a browser-based visual interactive interface for multi-turn conversation testing. 4. Phase 4 — Evaluate (The core of the Quality Flywheel): The official guide emphasizes: "Unit Test (go test) measures code contracts and type correctness; while agents-cli eval measures Agent behavior quality (Tool Trajectory, response quality, and safety)." 5. Phase 5 — Deploy: Use agents-cli deploy to package and deploy the project to agent_runtime, cloud_run, or gke, and automatically record the deployed resource ID in deployment_metadata.json. 6. Phase 6 & 7 — Publish & Observe: Register to the enterprise directory via agents-cli publish gemini, and combine with OpenTelemetry to automatically export all Graph node latencies, LLM Prompt/Response, and Token consumption to Cloud Trace, Cloud Logging, and BigQuery Agent Analytics. * 3. Agent Runtime vs. Cloud Run vs. GKE: Which Deployment Target Should I Choose? agents-cli supports three containerized deployment targets (all of which build based on the Dockerfile in the project root, making Go a first-class citizen on all three platforms). We have summarized the comparison based on the official Deployment Decision Matrix: | Evaluation Dimension | Agent Runtime (agent_runtime) | Cloud Run (cloud_run) | GKE (gke) | | --- | --- | --- | --- | | Infrastructure Management | Fully managed (built for Agents), automatically built from Dockerfile and mounted to reasoningEngines | Containerized serverless hosting, requires manual configuration of network and service parameters | GKE Autopilot, provides full K8s control but with the highest operational complexity | | Session State Management | Native built-in Agent Platform Sessions (automatically injects GOOGLE_CLOUD_AGENT_ENGINE_ID at runtime and enables cloud persistent Sessions) | In-Memory by default, requires additional Cloud SQL setup or manual connection to managed Sessions | In-Memory by default, requires additional Cloud SQL setup or managed Sessions | | API & Access Interface | Built-in :query / :streamQuery, Console Playground, and /api/{container_path} HTTP Passthrough and A2A | Native public or IAM-protected HTTPS URL, best for receiving external Webhooks (like LINE / Slack) | Custom K8s Ingress / Gateway | | Enterprise Integration | Native support for Managed OAuth 2.0 User Consent, Agent Identity, Agent Gateway, and PSC-I Private Networking | Supports VPC Egress, IAP, but does not support managed Agent OAuth flows | Full K8s network control | | Billing Model | Billed based on actual vCPU-hours + Memory-hours used (no compute fees when scaled to 0) | Billed based on Request duration or Instance seconds | Billed based on K8s Node Pod resources | * 4. Key Clarification: Will Instances Moved to Agent Runtime Scale to Zero? When planning to migrate line-card-assistant to Agent Runtime, a common concern is: > "According to agents-cli specifications, will instances moved to Agent Runtime never scale to 0 and constantly burn money?" The answer is: Agent Runtime supports Scale to Zero, and agents-cli adopts a clever dual-mode design: "Scale to 0 by default for development, and pre-warm 1 instance by default for production!" According to the agents-cli official deployment specifications (Sizing a Deployment): 1. When using agents-cli deploy directly (Default Scale to Zero): - The CLI's default resource allocation is identical to Cloud Run: --cpu 1, --memory 4Gi, --concurrency 8, --min-instances 0, --max-instances 10. - This means when there is no traffic to the Agent Runtime, the number of instances will automatically scale down to 0. Since Agent Runtime is billed based on actual vCPU-hours and Memory-hours, no compute fees are generated when idling at 0! 2. When using Terraform (service.tf) for production deployment (Default min_instances = 1): - When you generate Terraform IaC (deployment/terraform/single-project/service.tf) for a production environment, the template defaults min_instances to 1 to avoid cold starts for the first user. - If you want to save money in production and can tolerate a few seconds of cold start, you can specify agents-cli deploy --min-instances 0 during deployment or change min_instances back to 0 in service.tf. 💡 Why is a "Dual-layer Decoupled Architecture" best for a LINE Bot with Agent Runtime? Understanding the characteristics of Agent Runtime, you'll find that splitting the system into two layers is a perfect match: - Outer Layer: LINE Webhook Gateway (Cloud Run, min-instances=0): Since the LINE server sends Webhooks with only an X-Line-Signature HMAC signature, it cannot carry a Google Cloud IAM OAuth Token to call an IAM-protected Agent Runtime API. Use a lightweight Go Cloud Run service dedicated to "verifying the signature within 1 second, returning HTTP 200, and triggering the LINE loading animation." It scales to 0 when no one is sending business cards. - Inner Layer: Core Brain (Agent Runtime, min-instances=0): Focuses on running the ADK 2.0 Graph Workflow, managing cloud persistent VertexAiEngineSessionService (ensuring HITL pause states aren't lost across containers!), and simultaneously exposing A2A (/.well-known/agent-card.json) and the Agent Engine Playground to the enterprise! * 2. Planning the Migration Blueprint: Creating a 5-Phase Roadmap Using GitHub CLI To complete this architectural evolution methodically, we used the gh CLI in the GitHub Repository (kkdai/line-card-assistant) to establish 5 Milestones (Phases), 8 implementation Issues, and an overview tracking list Roadmap Issue #9: - Phase 1: agents-cli Environment and adk_go Project Scaffolding (The focus of this article!) - <input type="checkbox" disabled checked>#1 Upgrade google-agents-cli to 1.10+ and create .agents-cli-spec.md and agents-cli-manifest.yaml - <input type="checkbox" disabled checked>#2 Introduce adk_go + agent_runtime standard launcher architecture (app/agent.go, A2A, /app-info, Sessions, and Observability) - Phase 2: ADK Go Graph Workflow Cloud Storage and Persistence Adaptation - <input type="checkbox" disabled>#3 Deepen ADK Go v2.6+ Graph Workflow and NewRootAgent state management - <input type="checkbox" disabled>#4 Integrate Agent Platform Sessions (session/vertexai) and GCS Artifact Service to support cross-instance HITL pause/resume - <input type="checkbox" disabled>#5 Expand business card database (CardStore) to support Cloud SQL / Firestore cloud persistent storage - Phase 3: LINE Webhook Gateway and Agent Runtime Decoupled Architecture - <input type="checkbox" disabled>#6 Implement LINE Webhook Gateway (Cloud Run) and Agent Runtime dual-layer decoupled call architecture - Phase 4: agents-cli eval Automated Evaluation and Benchmarking - <input type="checkbox" disabled>#7 Establish agents-cli eval evaluation datasets and quality flywheel (OCR, Fan-Out/Fan-In, HITL loops, Hybrid Search) - Phase 5: Agent Runtime Official Deployment, IaC, and Observability - <input type="checkbox" disabled>#8 Deploy to Agent Runtime via agents-cli deploy and establish Terraform IaC, CI/CD, and Cloud Trace observability * 3. Phase 1 Hands-on Breakdown: Transforming an Existing Go Project into an adk_go + agent_runtime Dual-Channel Architecture In Phase 1 (corresponding to the merged PR #10), our goal was to: retain the original cmd/server/main.go independent LINE Webhook server while introducing the standard agents-cli (adk_go + agent_runtime) architecture in the project root, allowing the same codebase to possess both superpowers! Let's look at the key files changed in this upgrade and the new features they bring: line-card-assistant/ ├── .agents-cli-spec.md # [New] Phase 0 Architecture Spec (Single Source of Truth) ├── agents-cli-manifest.yaml # [New] agents-cli v1.10.0 project config (adk_go + agent_runtime) ├── deployment_metadata.json # [New] Records remote Agent Runtime resource ID and A2A status ├── main.go # [New] Agent Runtime / ADK WebUI / A2A standard HTTP entry point ├── apploader.go # [New] Loader supporting /ui/?app=app and rootAgent.Name() dual aliases ├── appurl.go # [New] Automatically calculates Cloud Run / Agent Runtime public URL (for A2A Card) ├── observability.go # [New] OpenTelemetry Cloud Trace / Logging / Metrics auto-mounting ├── app/ │ ├── agent.go # [New] Standard export interface: NewRootAgent(ctx) returns 12-node Graph Workflow │ └── agent_test.go # [New] Verifies Root Agent execution and search output directly via ADK Runner ├── appinfo/ │ └── appinfo.go # [New] GET /app-info endpoint, auto-parses Root Workflow and embedded CompanyEnrichAgent ├── sessions/ │ └── sessions.go # [New] Auto-switches between VertexAiEngineSessionService and local InMemoryService ├── cmd/server/main.go # [Retained] Original independent LINE Bot Webhook server └── internal/workflow/ # [Enhanced] Graph Workflow nodes upgraded to support dual-channel input and streaming Content output * 1. Creating agents-cli-manifest.yaml and .agents-cli-spec.md To make agents-cli recognize a Go project, the key is the agents-cli-manifest.yaml in the root directory: # Managed by agents-cli. Do not edit manually. acli_version: "1.10.0" project_name: "line-card-assistant" root_agent_name: "line_card_assistant" base_template: "adk_go" language: "go" framework: "google-adk" deployment_target: "agent_runtime" agent_directory: "app" region: "asia-east1" agent_guidance_filename: "GEMINI.md" is_a2a: true prototype: true generated_at: "2026-10-17T16:12:00Z" After configuration, running agents-cli info in the project root allows the CLI to immediately identify this as a Go project that supports A2A and targets deployment to the asia-east1 Agent Runtime: $ agents-cli info CLI version: 1.10.0 Project root: /Users/evanlin/Documents/line-card-assistant Project name: line-card-assistant Root agent name: line_card_assistant Language: go Deployment target: agent_runtime Agent directory: app Region: asia-east1 A2A: yes * 2. Encapsulating the Standard Root Agent Entry Point: app/agent.go In the adk_go template specification, all Agent logic should be exported via NewRootAgent(ctx context.Context) (agent.Agent, error) under the app package. Since we already used ADK 2.0's workflowagent.New to encapsulate the 12-node Graph Workflow into a standard agent.Agent in the first article, in app/agent.go, we only need to initialize the underlying CardStore (SQLite or Memory) and model.LLM (auto-detecting Vertex AI ADC or Gemini API Key), then call workflow.NewCardAssistantAgent for a seamless connection! package app import ( "context" "log/slog" "os" "strings" "github.com/kkdai/line-card-assistant/internal/config" "github.com/kkdai/line-card-assistant/internal/store" "github.com/kkdai/line-card-assistant/internal/workflow" "google.golang.org/adk/v2/agent" "google.golang.org/adk/v2/model" ) const RootAgentName = workflow.WorkflowName // "line_card_assistant" // NewRootAgent constructs the root ADK v2 Graph Workflow agent for Agent Runtime, // ADK WebUI playground, A2A server, and evaluation CLI. func NewRootAgent(ctx context.Context) (agent.Agent, error) { cfg := config.Load() var cardStore store.CardStore if strings.TrimSpace(cfg.SQLitePath) != "" { sqliteStore, err := store.NewSQLiteStore(cfg.SQLitePath) if err != nil { slog.Warn("failed to open SQLite store, falling back to in-memory store", "err", err) cardStore = store.NewMemoryStore() } else { cardStore = sqliteStore } } else { cardStore = store.NewMemoryStore() } var llm model.LLM if cfg.UseVertexAI || cfg.GoogleAPIKey != "" || os.Getenv("GOOGLE_CLOUD_AGENT_ENGINE_ID") != "" { realLLM, err := workflow.NewGeminiModel(ctx, cfg.GeminiModel, cfg.GoogleAPIKey, cfg.UseVertexAI, cfg.GoogleCloudProject, cfg.GoogleCloudLocation) if err != nil { slog.Warn("failed to initialize Gemini model, falling back to mock LLM", "err", err) llm = workflow.NewMockLLM() } else { llm = realLLM } } else { llm = workflow.NewMockLLM() } return NewRootAgentWithDeps(cardStore, llm) } func NewRootAgentWithDeps(cardStore store.CardStore, llm model.LLM) (agent.Agent, error) { return workflow.NewCardAssistantAgent(cardStore, llm) } Note a clever detail here: when deployed in Agent Runtime, the system automatically injects the GOOGLE_CLOUD_AGENT_ENGINE_ID environment variable. When running unit tests (go test ./...) without an API Key, it automatically falls back to workflow.NewMockLLM(), ensuring offline tests remain 100% fast and deterministic! * 3. Agent Runtime Standard HTTP Server and A2A Endpoints (Root main.go) Next, let's look at the new main.go in the project root. When the container is deployed to Agent Runtime (or executed locally via make playground), this entry point mounts five standard services: 1. OpenTelemetry Observability (setupObservability): Automatically exports Traces, Logs, and Metrics to Google Cloud Trace and Cloud Logging. 2. Session Service Auto-switching (sessions.SessionService()): In sessions/sessions.go, if VERTEX_AI_SEARCH_ENGINE_ID (Agent Engine ID) is detected, it automatically enables the cloud persistent NewVertexAiEngineSessionService; for local development, it seamlessly uses session.InMemoryService(). 3. ADK REST API and WebUI (/api/ and /ui/): Directly starts the ADK built-in streaming conversation API and web debugging interface via api.NewLauncher() and webui.NewLauncher(). 4. A2A (Agent-to-Agent) Protocol Server: Automatically exposes this Agent's capability description card (AgentCard) at /.well-known/agent-card.json and provides a JSON-RPC call endpoint at /a2a/app! 5. Agent Structure Introspection Endpoint (GET /app-info): Analyzes the entire Agent Tree via appinfo/appinfo.go. It not only lists the Root Graph Workflow but also registers the CompanyEnrichAgent (gemini-2.5-flash)—which is embedded in CompanyEnrichNode and equipped with geminitool.GoogleSearch{}—into the /app-info response. This allows agents-cli eval and the Google Cloud Console Playground to clearly see all sub-agents and tools within the system! * 4. Key Transformation: Enabling Graph Workflow to Support Both "LINE Webhook" and "ADK WebUI / A2A" Dual Channels! In the first version, our RouterNode assumed all input came from the LINE Webhook Handler, which pre-filled StateKeyInputType or StateKeyImageBytes into the Session State. Furthermore, terminal nodes (like FormatCarouselNode, BuildPreviewNode, SaveCardNode) only wrote results to StateKeyReplyText and StateKeyFlexCarouselJSON for the LINE Bot to retrieve and call ReplyMessage. However, when we open the ADK WebUI (make playground), the Agent Engine Console Playground, or talk directly to the Root Agent via agents-cli run "Find someone who does AI for me", the input follows the standard ADK ctx.UserContent() (containing text Part.Text or uploaded image Part.InlineData), and the output is read from ev.Content in the event stream! Therefore, we made two fully backward-compatible enhancements in internal/workflow/nodes_ingest.go and internal/workflow/nodes_search.go: (1) RouterNode Automatically Parses Images and Text from ctx.UserContent() // When uploading a business card image directly from ADK WebUI / A2A / Agent Engine Playground, // automatically extract InlineData.Data from ctx.UserContent().Parts and direct to RouteIngest! if len(getBytesState(ctx, StateKeyImageBytes)) == 0 && ctx.UserContent() != nil { for _, part := range ctx.UserContent().Parts { if part != nil && part.InlineData != nil && len(part.InlineData.Data) > 0 { setState(ctx, ev, StateKeyImageBytes, part.InlineData.Data) if getStringState(ctx, StateKeyInputType) == "" { setState(ctx, ev, StateKeyInputType, RouteIngest) } break } } } // If it's text input and the input parameter is empty, automatically extract query text or commands from ctx.UserContent().Parts if inputType == "" { sTrim, _ := input.(string) sTrim = strings.TrimSpace(sTrim) if sTrim == "" && ctx.UserContent() != nil { for _, part := range ctx.UserContent().Parts { if part != nil && strings.TrimSpace(part.Text) != "" { sTrim = strings.TrimSpace(part.Text) break } } } // Automatically determine if it's searching for a card (RouteSearch) or generating an icebreaker (action=draft_email&card_id=...) ... } (2) Terminal Nodes Synchronously Write to ev.Content In BuildPreviewNode, SaveCardNode, FormatCarouselNode, and EmailDraftNode, in addition to writing to StateDelta for the LINE Bot, we added a line: ev.Content = genai.NewContentFromText(flexJSON, genai.RoleModel) This way: - Sending or searching cards from LINE: The LINE Webhook still gets the Flex JSON from StateDelta to render the beautiful LINE card. - Testing from ADK WebUI (make playground) or Agent Engine Playground: Without needing a LINE account, you can upload a business card image or type "Find someone I met at Cloud Summit who does AI" directly in the browser. You'll see the full execution trajectory (Graph Node transitions) and the output Flex JSON / icebreaker draft directly in the conversation window! We also added a TestRootAgentDirectSearchExecution unit test in app/agent_test.go, using the standard ADK runner.Runner to send a genai.Content query to app.NewRootAgent, verifying that the business card Carousel JSON can be obtained directly from the Event Stream without going through the LINE Engine! * 5. Multi-binary Dockerfile: One Image for Both Agent Runtime and LINE Webhook Finally, to allow the same container image to be deployed to Agent Runtime via agents-cli deploy and to Cloud Run as an independent LINE Bot server, we updated the Dockerfile to compile two static executables in a multi-stage build: # Build the Agent Runtime / ADK WebUI / A2A entrypoint binary (root main.go) RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="-s -w" -o /out/agent . # Also build the standalone LINE Bot Webhook server binary (cmd/server/main.go) RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="-s -w" -o /out/line-card-assistant ./cmd/server - The default ENTRYPOINT ["/agent"] starts the Agent Runtime standard server (listening on 0.0.0.0:8080). - If it needs to run as a monolithic LINE Webhook Server on Cloud Run, simply specify the Container Command as /bin/line-card-assistant! * 4. Summary and Teaser for the Next Step Through the Phase 1 transformation (PR #10), our LINE AI Business Card Assistant (line-card-assistant) has officially evolved from a simple LINE Bot application into an adk_go Agent that complies with Google Cloud Agent Platform standards: 1. Full support for the agents-cli ecosystem: Equipped with agents-cli-manifest.yaml and .agents-cli-spec.md. 2. Dual-channel execution capability: Can interact via LINE chat and directly through the ADK WebUI via make playground, be called by other Agents via the A2A protocol, or undergo multimodal business card recognition and search testing in the Agent Engine Playground. 3. Cloud-native ready: Built-in VertexAiEngineSessionService auto-switching and OpenTelemetry Cloud Trace tracking capabilities. In the next article of this series, we will enter Phase 2 & Phase 3, where we will connect HITL pause states and image artifacts to cloud persistent storage and refactor the system into a dual-layer decoupled architecture of "Cloud Run LINE Webhook Gateway + Agent Runtime Core Brain." Stay tuned! * Reference Resources and Further Reading - First article in the series: [[Agent Platform in Practice (1)] Building a LINE AI Business Card Assistant with Go and Google ADK 2.0 Graph Workflow: From Fan-Out Parallel Retrieval to HITL Pause Loop Correction](/2026/10/16/adk-graph-workflow-go/) - Open source code and Roadmap for this project: - GitHub Repository: kkdai/line-card-assistant - Agent Runtime Migration Roadmap: Issue #9 - Phase 1 Implementation Pull Request: PR #10: feat(phase1): scaffold adk_go Agent Runtime project structure and app.NewRootAgent - Google agents-cli and Agent Platform official documentation: - Getting Started - agents-cli - The Lifecycle - agents-cli - Google ADK Graph Workflow official documentation: - Graph Workflows in ADK: Everything You Need to Know (Google Cloud Blog) - Graph-based agent workflows (adk.dev/graphs)

名片助手该在哪一步等你确认？ Evan Lin的示例：OCR后并行查重、搜公司、从近期聊天推断会面背景；等分支结束，再整理预览。 修改后重新预览，确认才保存，之后仅生成邮件草稿。 把人工确认放在补全之后、存档之前。 固定源码解读，未实测。 github.com/kkdai/line-card-assistant/blob/19296ec…

[AI in Practice] Upgrading from Gemini 3.5/3.6 to Gemini 4 Argon: Migration Notes and Pitfalls for LINE Bot Developers !Stylized promotional blogger visual with modern editorial branding and the text "Gemini 4 Argon". Background I have a LINE Bot that I use every day, linebot-helper-python. I use it daily to summarize URLs, ask follow-up questions about YouTube video content, search for map locations, and manage bookmarks. In terms of model configuration, to balance response speed and budget, many workflows were previously running on Gemini 3.5 (such as gemini-3.5-flash-lite for high-throughput, low-latency classification and data extraction) and Gemini 3.6 (such as gemini-3.6-flash for multi-step tool calling and multimodal reasoning). In mid-October, Google officially pushed Gemini 4 Argon (gemini-4-argon) to the GA stable version (paired with Python google-genai >= 2.30.1). After reading the official announcement and migration documents, the most attractive features on the surface are the big numbers: a 1M unified token window shared by input, thinking, and output; Long Decode that automatically relays a single call up to 1 million output tokens; Agentic Video Understanding which is now officially GA and saves 88% of tokens; and the Interactions API (/v1beta/interactions), which has become the default mainstream. But if your goal is the same as mine—to upgrade (or partially switch) the existing Gemini 3.5 / 3.6 in your LINE Bot to Gemini 4 Argon—don't assume it's as simple as changing model="gemini-3.6-flash" to model="gemini-4-argon". This generation has several changes in parameter specifications, token budget calculation methods, caching mechanisms, and even assumptions about waiting times that hit the pain points of LINE Bots exactly. This article organizes all the details you need to pay attention to when migrating from 3.5 / 3.6 to Gemini 4. - * * Specification Comparison: 3.5 Flash-Lite / 3.6 Flash vs. Gemini 4 Argon Before diving into settings, let's look at the positioning differences of these three models in one table: | Item | gemini-3.5-flash-lite | gemini-3.6-flash | gemini-4-argon | | --- | --- | --- | --- | | Positioning | Ultra-fast, low-cost, high-throughput classification & structured extraction | Balance of speed, multi-step Agent, and multimodality | Flagship deep reasoning, complex coding, long output, and autonomous agents | | Context Window | 1,048,576 | 1,048,576 | 1,048,576 (Shared Input + Thinking + Output) | | Max Single Output | 65,536 | 65,536 | generateContent: Non-streaming 32,768 / Streaming 65,536 <br>Interactions API: Synchronous 65,536 <br>_(Long Decode can continue up to 1M for both)_ | | Default thinking_level | "minimal" | "medium" | "medium" (Supports only low / medium / high) | | Context Caching | Supported | Supported | Implicit Caching only (Calling Explicit will error) | | temperature / top_p / top_k | Deprecated (Ignored) | Deprecated (Ignored) | Deprecated (Ignored, official requirement to remove) | | Pricing (Input / Output) | $0.30 / $2.50 (per 1M) | $1.50 / $7.50 (per 1M) | Early bird until 2026/11/30: $2.00 / $10.00 <br>_(Reverts to $4.00 / $20.00 from 12/1; 95% discount for cache hits)_ | | Free Tier | Available | Available | None (Must be linked to Paid Tier billing) | Just by looking at this table, you can identify several pitfalls you'll encounter if you only change the model name. Let's break them down one by one. - * * Pitfall 1: LINE Bots usually set max_output_tokens very small, which returns an empty string in Gemini 4 This is the first hidden trap that LINE Bot developers are most likely to fall into. Because a single LINE text message has a 5,000-character limit, and mobile screens are not suitable for long-winded content, many people (including myself in the past) are used to setting max_output_tokens to 2048 or 4096 in GenerateContentConfig as a safeguard to save money and prevent text overflow. If you set it this way on Gemini 4 Argon, when encountering a slightly complex problem, you will find the model returns finish_reason: "MAX_TOKENS", but response.text is empty, or it cuts off after barely two sentences. The reason is that Gemini 4 Argon adopts a Unified 1M Input-Output Window, and has two hard rules: 1. Thinking tokens are counted directly towards the max_output_tokens budget: Every thinking token the model spends before answering is deducted from your max_output_tokens. When a question requires deep reasoning, the model can easily burn through your 4096 limit just by thinking, hitting a wall and stopping before it even starts outputting the formal answer. 2. generateContent non-streaming single request limit decreased from 64K to 32K: In Gemini 3.5 / 3.6, the single non-streaming response limit was 65,536; in gemini-4-argon, if you continue using traditional generateContent, the non-streaming single limit has been lowered to 32,768 tokens (streaming or using the new Interactions API synchronous call remains at 65,536). Cause and Solution: The official migration guide states clearly—if your old project set max_output_tokens below 32K, please increase it to at least 32768 (or just keep the default 65536) when upgrading to gemini-4-argon. In Gemini 4, max_output_tokens is used as a "total emergency brake to prevent burning through money in extreme situations," not to control the length of the answer. - To spend fewer tokens and speed up response: Lower the thinking_level="low". - To make the output fit LINE message length: Write it in the system_instruction (e.g., "Please keep it within 400 words and use bullet points"). # ❌ Old safeguard setting often used in LINE Bots; thinking process will consume all 2048 tokens on gemini-4-argon config = types.GenerateContentConfig( max_output_tokens=2048, ) # ✅ Correct way for Gemini 4 Argon: Relax max_output_tokens, use thinking_level to control cost and latency config = types.GenerateContentConfig( max_output_tokens=32768, thinking_config=types.ThinkingConfig(thinking_level="low"), system_instruction="You are a LINE assistant. Keep answers concise and within 500 words.", ) - * * Pitfall 2: Differences in thinking_level settings—3.5-flash-lite's "minimal" cannot be moved directly If you have already changed the old-school thinking_budget (passing an integer token count) to thinking_level in Gemini 3.5 / 3.6, there is another small trap here: - On gemini-3.5-flash-lite, the default thinking level is called "minimal". - But on gemini-4-argon, only three levels are supported: "low", "medium" (default), and "high". If your config shares the same generator and you bring "minimal" into gemini-4-argon as is, or if you still have the old thinking_budget (although the current API will map it to the closest level, it will return a deprecation warning), it will make your settings messy. For LINE Bots, the practical division of these three levels in gemini-4-argon is very clear: - "low": The first choice for general LINE conversations, URL summaries, and real-time Q&A. Lowest token consumption and lowest Time to First Token (TTFT). - "medium" (default): Suitable for scenarios requiring multiple tool calls (Function Calling), reading long videos, or writing code. - "high": Deep math, complex itinerary optimization, long-form analysis. Note: Setting this to "high" makes it almost impossible to finish within the valid time of a LINE replyToken. You must use an asynchronous push architecture. - * * Pitfall 3: LINE Webhook's few-second lifespan vs. official recommendation of "20-minute Timeout" When reading the Gemini 4 Argon migration document, one item made me laugh out loud: > Increase client timeouts: Set HTTP client timeouts to at least 15–20 minutes per call. The official documentation even added a note in the Long Decode section: when facing difficult tasks, gemini-4-argon might spend an entire 32K or 64K hop entirely on thinking, only outputting the real answer in the next hop (under generateContent, the new SDK google-genai >= 2.30.1 has automatic_continuation=True enabled by default, which uses continuation_token to automatically loop for you on the client side; under the Interactions API, to exceed the single sync limit of 65536, you must enable background=True to let the server side run multiple hops). This is perfectly fine when running a CLI script on your own computer; but in a LINE Bot + Cloud Run architecture, waiting synchronously on the client side is a disaster: 1. LINE's replyToken has an extremely short lifespan and won't wait two minutes for you to think. 2. LINE users staring at a screen with no response for more than 10 seconds will start pressing the button a second or third time. 3. If you wait synchronously for automatic_continuation to finish several hops within the same Webhook request, Cloud Run connections and workers will quickly become deadlocked. Cause and Solution: To put Gemini 4 into a LINE Bot, you must do two things architecturally: 1. Don't phase out 3.5 / 3.6; instead, create a "Fast/Slow Model Router" The official documentation emphasizes that gemini-3.6-flash and gemini-3.5-flash-lite are still the main models for stable online supply. The healthiest LINE Bot architecture is not "switching everything to gemini-4-argon," but rather: - First-line instant replies / Intent classification / Simple structured extraction: Continue to hand these to gemini-3.5-flash-lite ($0.30/1M, extremely fast) or gemini-3.6-flash ($1.50/1M, concise, saves steps in tool calling). - Heavy reasoning / Long video follow-up / Complex tasks: Only then route to gemini-4-argon. 2. Use the new Interactions API with background=True + webhook_config for heavy tasks In this generation, Google has labeled our familiar generateContent as (Legacy) and is pushing the Gemini Interactions API (/v1beta/interactions) across its documentation. The greatest value of the Interactions API for LINE Bots is its native support for background=True and Webhooks (supporting project-level Static Webhooks or Dynamic webhook_config included in a single request): 1. A user asks a question on LINE that requires gemini-4-argon with "high" thinking or long output. 2. The LINE Bot first calls the LINE Loading Animation API and uses replyToken to reply instantly: "_This is a deep question. I am performing deep reasoning in the background and will push the result to you once finished!_" 3. Simultaneously call client.interactions.create(..., background=True, webhook_config=...), handing the task to Google's server to run multiple hops slowly. You can even put the line_user_id directly into user_metadata so Google returns it as-is during the callback. 4. When Google finishes and sends an interaction.completed event to your /webhooks/gemini endpoint, use client.interactions.get(interaction_id) to retrieve the result (if multiple Long Decode hops over 64K were run, all segments will be fully recorded in the model_output of interaction.steps). Then, calmly call LINE's push_message to push the result into the chat room. from google import genai client = genai.Client() def dispatch_heavy_line_task(line_user_id: str, prompt: str): # Hand over to Google server for background execution (even multi-round 64K hops for Long Decode will be automatically completed on the server side) interaction = client.interactions.create( model="gemini-4-argon", input=prompt, background=True, generation_config={ "max_output_tokens": 131072, "thinking_level": "high", }, webhook_config={ "uris": ["https://your-cloud-run.a.run.app/webhooks/gemini"], "user_metadata": {"line_user_id": line_user_id}, }, ) return interaction.id - * * Pitfall 4: Explicit Caching Errors—Prompt order determines if you save 95% If your LINE Bot contains a full product FAQ, long System Instructions, or large documents uploaded by users, you might have previously used client.caches.create() to create an Explicit Cache and then put cached_content into GenerateContentConfig to save money. After upgrading to gemini-4-argon, this code must be completely removed. gemini-4-argon does not support Explicit Caching. If you call caches.create or include cached_content in the config, the API will return an Error. Instead, it uses Implicit Caching, which is enabled by default: - As long as a single Prompt length reaches 4,096 tokens or more, and the preceding content (Prefix) matches a previous request, the system will automatically hit the cache. - Cache-hit Input Tokens receive a 95% discount (priced at 0.05x)—at early bird prices, this drops from $2.00 / 1M to $0.10 / 1M, which is cheaper than gemini-3.5-flash-lite! (Even when Long Decode automatically continues to the next hop, the repeated Context from before will automatically apply the 95% cache discount.) Key impact on LINE Bots: Since this has become automatic Prefix Matching, the order in which you assemble your Prompt determines the thickness of your bill: - Always put at the beginning (Immutable): Long system_instruction, shared background knowledge base, Few-shot examples, fixed background documents for that group. - Always put at the end (Changes every time): current_time timestamp, LINE user name/status, the current message. If you accidentally write "Current time: 2026-10-15 21:40:02" on the first line of your System Prompt, the first token of every request will be different, and the 95% cache discount will immediately drop to zero. - * * Pitfall 5: Strict rules tightening since 3.5 / 3.6—Sampling parameters, Prefill, and Thought Signatures These three rules have actually been in effect since gemini-3.5-flash-lite and gemini-3.6-flash, but if you are organizing your codebase across versions, be sure to clean them up together: 1. Remove temperature, top_p, top_k, candidate_count These four sampling parameters are all deprecated and ignored on Gemini 3.5 / 3.6 and Gemini 4 Argon (official notice says future generations will return HTTP 400 if these parameters are included). If you used to set temperature=0.0 to ensure stable output format, delete them all now and switch to Structured Outputs (response_schema) and clear system_instruction. 2. Prohibit Prefilled Model Turns (The last turn cannot be role: "model") A common Prompt technique used to be: to prevent the model from saying "Sure! Here is the summary..." or to force it to start outputting JSON from {, you would put {"role": "model", "parts": [{"text": "{"}]} as the last entry in the contents array. Now, if the last turn of non-empty content in the payload is the model role, the API will return an HTTP 400 Error. To suppress the opening remarks, please write it properly into the system_instruction. 3. Multi-turn conversations and Function Calling must preserve thought_signature (or just use previous_interaction_id) When a LINE Bot runs multi-step tool calls (e.g., first check bookmark DB $\rightarrow$ then fetch web content $\rightarrow$ finally summarize), the Part returned by the model will contain a thought_signature. If you manually maintain the conversation history array but only extract and reassemble the text and function_call, missing the thought_signature, the quality of multi-turn tool calls will degrade significantly or even error out. For those who don't want to manage thought_signature serialization themselves, the easiest way to upgrade to Gemini 4 is to use the Interactions API's previous_interaction_id: # Only remember the last interaction.id for that LINE user in Redis; let Google's server handle history state and thought_signature interaction = client.interactions.create( model="gemini-4-argon", input=user_message, previous_interaction_id=last_interaction_id, ) - * * The real weapon worth upgrading your LINE Bot for: Agentic Video Understanding GA After discussing all the pitfalls, let's talk about a positive upgrade for gemini-4-argon (and 3.8-flash) that is most noticeable for LINE Bots: Agentic Video Understanding is officially GA. One core feature of my LINE Bot is "drop a YouTube link and ask follow-up questions about the video content." Previously, traditional video understanding (static mode) involved taking a screenshot of the entire video every second (1 FPS) and stuffing it into the Context Window. For a 30-60 minute long lecture or Podcast video, even if the Context could fit it, the token consumption was staggering. After switching to Agentic mode (Note: it still defaults to static on all models including gemini-4-argon, so it must be explicitly enabled), the model's behavior changes to: 1. First read the audio transcript and timeline structure. 2. Based on the user's question, decide which timestamps to "scrub" to. 3. Dynamically load high-resolution and high-frame-rate frames only for key scenes to look closely. Whether you use the new Interactions API or the original generate_content, remember to add the corresponding parameters (parameter names and casing differ slightly between the two): # Method A: Using the new Interactions API (parameter is processing="agentic") interaction = client.interactions.create( model="gemini-4-argon", input=[ { "type": "video", "uri": youtube_url, "mime_type": "video/mp4", "processing": "agentic", # Remember to enable explicitly; default is still static }, { "type": "text", "text": "On the page where the speaker shows the architecture diagram, what are the three bottlenecks mentioned? Please include [MM:SS] timestamps.", }, ], generation_config={ "max_output_tokens": 32768, "thinking_level": "medium", }, ) # Method B: Using traditional generate_content (parameter is media_processing="AGENTIC") response = client.models.generate_content( model="gemini-4-argon", contents=[ types.Part.from_uri( file_uri=youtube_url, mime_type="video/mp4", media_processing="AGENTIC", # Remember to enable explicitly; default is still STATIC ), "On the page where the speaker shows the architecture diagram, what are the three bottlenecks mentioned? Please include [MM:SS] timestamps.", ], config=types.GenerateContentConfig( max_output_tokens=32768, thinking_config=types.ThinkingConfig(thinking_level="medium"), ), ) Official tests show that for long videos over 5-10 minutes and "needle in a haystack" detail questions, token usage is saved by up to 88%, and accuracy is improved by 7%. This point alone completely makes up for the higher unit price of gemini-4-argon. - * * Upgrade Checklist Finally, here is a PR self-check list for LINE Bot projects before upgrading from Gemini 3.5 / 3.6 to gemini-4-argon: 1. SDK and Billing Confirmation: Upgrade google-genai to >= 2.30.1; confirm the project associated with the API Key is linked to a Paid Tier (Free Tier and Google AI Pro cannot call gemini-4-argon). 2. Remove Sampling Parameters and Prefill: Search the entire project and remove temperature, top_p, top_k, candidate_count, and any prefilled prompts ending with role="model". 3. Remove Explicit Caching: Remove caches.create and cached_content; check Prompt assembly logic to ensure fixed, long System Instructions are placed at the beginning to trigger the 95% discount of Implicit Caching. 4. Increase max_output_tokens: Increase any limit smaller than 32768 to at least 32768 (to avoid empty responses caused by thinking tokens consuming the budget); move word count limits to system_instruction. 5. Check thinking_level Enum Values: Replace 3.5-flash-lite specific "minimal" or old thinking_budget with "low", "medium", or "high". 6. Fast/Slow Routing and Async Push: Tasks that need an instant reply within a few seconds for LINE replyToken should be set to thinking_level="low" or continue to be handled by 3.5-flash-lite / 3.6-flash; deep reasoning tasks should use the Interactions API (background=True + webhook_config) paired with LINE push_message. 7. Enable Agentic Mode for Video Understanding: When processing YouTube or long videos, add media_processing="AGENTIC" to Part.from_uri (or "processing": "agentic" for the Interactions API). If you use a Coding Agent that supports Skills (like Antigravity or Claude Code), Google also provides a migration Skill (/gemini-api-dev migrate my app to Gemini 4 Argon) that can help automatically scan for these parameter changes. However, architectural levels like "LINE replyToken timeout routing" and "Prompt prefix cache order" still need to be designed manually.