If your pipeline or agent runs longer than your deploy cycle, design what the retry remembers before you design the happy path
wrote up a durable RAG ingestion pipeline a while back, and I think the main lesson applies to long-running agents just as much.
anything that runs for hours will see a deploy, an OOM or a rate limit somewhere in the middle. for a 5,000 document ingestion run that means dying at document 3,000, and a plain retry going back to document 0 and re-embedding everything it already paid for. an agent making a few hundred model calls over an hour has the same shape, with more expensive steps.
two things from the build I'd carry into any long-running agent:
- record each step when it completes. every stage in the tutorial is wrapped in Inngest's step.run(), so the retry starts at the document that broke and the failure only costs the 2,000 documents that were left.
- make waiting cheap. some steps have to wait on a person. step.wait_for_event() suspends the run for up to seven days without holding a worker open, and when the reviewer approves it picks up exactly where it stopped. in the tutorial that's the gate for documents that parsed to garbage. in an agent it's the approval step.
it also helps in the boring case. you fix a parser bug, deploy, and replay the failed runs. the embed steps that already succeeded come back from the journal, so you don't pay for them twice.
honest limit: if re-running your whole job is cheap, none of this is worth adding a runtime for. it starts to pay when a full re-run costs real tokens or hours, or when partial failure is the normal case for you.
full write-up with code: github.com/NirDiamant/agents-towards-production/t…
built on Inngest: inngest.link
1 point
Mentions
Build an Autonomous Web Research Analyst with Firecrawl, LangChain & M This video demonstrates building an autonomous web research analyst by integrating Firecrawl CLI for data extraction, LangChain for AI reasoning, Payload for workflow orchestration, and Tremor for monitoring. We'll connect these tools to create a system that autonomously researches topics and delivers insights. The focus is on practical implementation rather than theoretical concepts. The final product handles end-to-end research workflows with minimal human intervention. Repositories used in this architecture: - firecrawl-cli: firecrawl/cli - agents-towards-production: NirDiamant/agents-towards-production - langchain: langchain-ai/langchain - payload: payloadcms/payload - tremor: tremorlabs/tremor This episode explains one integrated product architecture, including interfaces, data flow, deployment, security, and failure handling. Subscribe for daily open-source product builds: youtube.com #GitHub #OpenSource #SoftwareArchitecture #Programming
If your pipeline or agent runs longer than your deploy cycle, design what the retry remembers before you design the happy path wrote up a durable RAG ingestion pipeline a while back, and I think the main lesson applies to long-running agents just as much. anything that runs for hours will see a deploy, an OOM or a rate limit somewhere in the middle. for a 5,000 document ingestion run that means dying at document 3,000, and a plain retry going back to document 0 and re-embedding everything it already paid for. an agent making a few hundred model calls over an hour has the same shape, with more expensive steps. two things from the build I'd carry into any long-running agent: - record each step when it completes. every stage in the tutorial is wrapped in Inngest's step.run(), so the retry starts at the document that broke and the failure only costs the 2,000 documents that were left. - make waiting cheap. some steps have to wait on a person. step.wait_for_event() suspends the run for up to seven days without holding a worker open, and when the reviewer approves it picks up exactly where it stopped. in the tutorial that's the gate for documents that parsed to garbage. in an agent it's the approval step. it also helps in the boring case. you fix a parser bug, deploy, and replay the failed runs. the embed steps that already succeeded come back from the journal, so you don't pay for them twice. honest limit: if re-running your whole job is cheap, none of this is worth adding a runtime for. it starts to pay when a full re-run costs real tokens or hours, or when partial failure is the normal case for you. full write-up with code: github.com/NirDiamant/agents-towards-production/t… built on Inngest: inngest.link
10 Agents Towards Production How to ship agents with tutorials on memory, deployment, monitoring NirDiamant/agents-towards-production