temporalio

What is workflow orchestration, and where should a .NET app run it? Workflow orchestration means that one component owns the sequence of steps, the state and the failures of a multi-step process. That component is the orchestrator. For a business process inside one .NET product, run the orchestrator as a library in your app. Keep its state in your own database. Data teams use the same term for a second job. For them, the term means running data pipelines on tools such as Apache Airflow. Mike Lukinov, co-founder of Optimajet, wrote What Is Workflow Orchestration. The post gives a rule for where to run the orchestrator. If the process belongs to one product and its data should stay in that product's database, put the orchestrator inside the product. The post also says when a separate server or a cloud service is the better place. Workflow Engine by Optimajet is a .NET workflow engine that runs as a library inside your app. The post compares Workflow Engine with Temporal, Camunda 8, AWS Step Functions and the data pipeline tools. It also lists the vendor pages used for these comparisons, with the date each page was read. Workflow Engine Free needs no license key. Its license covers personal use and internal commercial use, with attribution. It runs up to 10 workflow definitions (schemes) and 4 execution threads. A public web app or a SaaS product needs the Subscription, SaaS or Enterprise Edition of Workflow Engine NEO. Workflow Engine NEO is a separate product based on Workflow Engine. It adds an HTTP API and multitenancy. The facts about Workflow Engine were checked against version 23.0.0 and its documentation on 27 October 2026. What an orchestrator does For a business process inside an application, the orchestrator is software called a workflow engine. Workflow Engine by Optimajet is one product of that kind. Take a purchase request. Someone submits it, and a manager approves it within a week. Then the system places the order through a supplier's API. Each request is a case. A case is one business item that moves through the process. Without an orchestrator, each service calls the next one. The state of the request is spread across several tables. A scheduled job searches for requests that got stuck. With an orchestrator, each request has one record. There is one place to ask where the request is. An orchestrator has five duties: - Sequence. It decides which step runs next, and which steps can run in parallel. In Workflow Engine, the sequence lives in the process definition, which is called a scheme. A scheme is an XML file. - State. It saves the current step and the data of each case between steps. In Workflow Engine, the state is a row in the application's own database, so the case survives a restart. - Waiting. It pauses a case until a person acts, a timer fires or another system answers. In Workflow Engine, a waiting case holds no execution thread. - Failure handling. It retries a step, times it out, moves the case to another path, or hands the case to a person. In Workflow Engine, the exception handlers of a step can retry a failed call and then move the case to manual review. - History. It records each move, the time of the move, and who or what caused it. In Workflow Engine, GetProcessHistoryAsync returns that history. Two jobs share the name Data pipeline orchestration moves datasets on a schedule. Application workflow orchestration runs business cases that wait for people. It is the job of a workflow engine, such as Workflow Engine. The two jobs need different tools. | | Data pipeline orchestration | Application workflow orchestration | |---|---|---| | What moves | Datasets, from source to warehouse | One business case, such as an order, a request or a loan | | Typical shape | A graph of tasks with no loops (a DAG). It runs on a schedule, and each run takes minutes to hours. | A process instance that can wait days for a person | | Who acts | Mostly machines | Machines and people | | When a step fails | Rerun the task, or backfill (start the runs that were missed) | Retry, time out, take another path, or route the case to a person. The case is kept. | | Typical tools | Apache Airflow, Dagster, Prefect | Workflow engines such as Temporal, Camunda, AWS Step Functions and Workflow Engine by Optimajet | For a nightly ETL chain (extract, transform, load), pick a data tool. Apache Airflow, Dagster and Prefect define pipelines in Python and are built for that job. Kubernetes uses the word orchestration for a third thing. Container orchestration keeps containers running in the state you declare. It knows nothing about approvals. What an embedded engine does with waits, deadlines and failures An embedded engine is a workflow engine that runs as a library inside your application. In Workflow Engine, the engine's object inside the application is WorkflowRuntime. It runs on the application's own database. The purchase request goes from the Draft step to the Review step, where a manager approves or rejects it. Then it goes to the Ordering step, which calls the supplier's API, and then to Ordered. A restart. If you stop the app while the request waits for approval, the request is still in Review when the app starts again. A waiting request has the status Idled. It holds no execution thread. All it holds is its row in the database. The restarted application is a new operating-system process. It loads the request with the same step, the same deadline and the same two commands, Approve and Reject. A command is what a person executes to move a request to another step. A deadline. A timer on the Review step can take its value from a process parameter. A process parameter is a value stored with the request. So each request carries its own review window. When the timer fires, a transition moves the request from Review to Escalated. By default, the engine checks timers about once a second. It does this only while WorkflowRuntime is started. If a timer's deadline passes while the app is down, the timer fires after the app starts again. A failure. A step runs code called an action. The step also declares what happens when its action fails. When the manager executes Approve, the request moves into Ordering. So the supplier call runs inside the Approve command. If that call throws an HttpRequestException, the exception handlers of the Ordering step can retry it twice. After that, they move the request to a step for manual review. Without handlers, the Approve command fails with a ProcessExecutionException. The request stays in Review with the status Error. Nothing is lost, but someone has to go and find the request. An embedded engine still leaves four things for you to handle: - A crash in the middle of a step. By default, the engine does not rerun an interrupted step. The process gets the status Error or Terminated. Your recovery policy decides what happens next. - Side effects. A real order needs an idempotency key. This key is a value that makes a repeated call harmless. - The next step for a person. Escalation and manual-review steps need commands for the manager. Without them, a case stops there. - More than one server. To run several servers on one database, you need either Workflow Engine Complete or Workflow Engine NEO. Workflow Engine Complete is a paid Edition of Workflow Engine. When you need an orchestrator, and when you do not An orchestrator is one more component to learn and to keep up to date. For small jobs, it costs more than it saves. One job on a schedule needs only a cron job or a hosted service. One unit of work with a retry policy needs only a queue and a background worker. Two services that work together with no waits and no people need only a direct call with a retry policy. Mike Lukinov's rule of thumb is that you need an orchestrator when one of these is true: - A step waits for a person, or a process takes days. - One process touches three or more systems. - Someone keeps asking where a case is stuck, and nobody can answer quickly. - The process changes more often than you want to release code. Where the orchestrator runs For an application workflow, the orchestrator runs in one of three places. The first is a separate server, such as Temporal. The second is a cloud provider's service, such as AWS Step Functions. The third is a library in your app, such as Workflow Engine by Optimajet. The place you choose decides where the process state lives and what you operate. It also decides what it costs to move away later. | | A separate server | A cloud provider's service | A library in your app | |---|---|---|---| | Examples | Temporal, Camunda 8, Orkes Conductor | AWS Step Functions, Azure Durable Functions, Azure Logic Apps | Workflow Engine by Optimajet, Elsa Workflows, Workflow Core | | Where state lives | In the server's own storage | In the provider's store. With Durable Functions, in a backend you choose. | In your application's database | | What you operate | A cluster, its storage and its upgrades, unless you buy the vendor's cloud plan | Nothing extra, within the provider's limits | Your application and its database. The database also holds the engine's tables. They get migrations on major upgrades. Workflow Engine Free runs on one server. | | How your code talks to it | Through an SDK. Your workers poll the server for tasks. | Through the provider's SDK or API | Through method calls in the same process | | Cost model | Temporal (MIT) and Conductor OSS (Apache 2.0) are free to self-host. Camunda 8 Self-Managed needs a license in production. Each vendor also sells a cloud plan. | You pay for what you use. Step Functions Standard charges per state transition. Logic Apps Consumption charges per action. | Elsa Workflows and Workflow Core use the MIT license. Workflow Engine Free covers personal and internal commercial use, with attribution. A public web app or a SaaS product needs an Edition of Workflow Engine NEO. There is no fee per execution. | | Moving away | You rewrite the definitions. Temporal workflows are code written against its SDK. Camunda 8 models are BPMN (a standard process notation) with Camunda's extensions. Conductor definitions use Conductor's own JSON format. | You rewrite the definitions. Step Functions uses the Amazon States Language. Durable Functions workflows are code written against its SDK. | You rewrite the schemes, which are in the engine's own XML format. The process data is already in your database. | | Fits best | Many services in several languages, or one process that starts thousands of parallel tasks | An application that already lives on one cloud | One product that keeps its own processes and data | In Mike Lukinov's view, a separate server is worth its operating cost when services in several languages share the same processes, or when one process starts thousands of parallel tasks. That is where Temporal is strong. A cloud service is the fastest way to start when the application already lives on that cloud. Its cost grows with every execution. For one product whose data stays in its own database, the library is the simplest place. The reason is that you add the library to an application you already run. The post takes its facts about these products from the following pages: - Temporal: the .NET SDK (Temporalio 1.20.0), workers and license, read on 14 and 27 October 2026. - Camunda: the licenses, read on 14 October. - Orkes Conductor: the C# SDK (conductor-csharp 3.0.0), read on 14 October. - Azure Durable Functions: the storage providers, read on 11 October. - AWS Step Functions and Azure Logic Apps: the Step Functions pricing and Logic Apps pricing pages, read on 27 October. - Elsa Workflows 3.9.0, read on 11 October, and Workflow Core 3.21.0, read on 1 October. The full post also compares orchestration with automation, job scheduling and choreography. It sorts the tools into seven categories, from data pipeline orchestrators to robotic process automation. The categories are in the tools section of What Is Workflow Orchestration.

sdk-python ⭐ 1.2k A Python framework for writing long-running business logic as durable workflows. 🔗 temporalio/sdk-python #opensource t.co

Kustomize JSON patch add: fields a Helm chart does not template A chart can only set the fields its templates write. When a Kubernetes field you need is missing from them, the value you add to values.yaml is silently ignored, unless the chart has a values schema that rejects it. This post sets nine such fields from open issues on the rendered manifests, first with a Kustomize patch over helm template, then with Kapitan's mutate generator. Most of them are RFC 6902 JSON Patch operations, and mutate's json6902 takes the same operations as Kustomize, so the two versions of each patch read almost the same. Each patch was rendered from the chart version named in its section with Helm 4.3.1, Kustomize 5.8.3, and Kapitan 0.36.3 with mutate 0.1.0. In every case the Kustomize output and the mutate output set the same value and changed nothing else in the render. The mutate output was installed on kind clusters on Kubernetes 1.37.0 with three worker nodes, and each Kustomize output's patched object passed a server-side dry run there. Charts from HTTP repositories matched the sha256 their index.yaml lists. The guide Patch a Helm chart compares the places such a patch can live. The class for each chart pins its version, like this one for Vault, and the target adds mutate as a second compile entry on the chart's output_path, as the guide shows: parameters: kapitan: dependencies: - type: helm source: https://helm.releases.hashicorp.com chart_name: vault version: 0.34.1 output_path: charts/vault compile: - input_type: helm input_paths: [charts/vault] output_path: vault helm_values_files: [values/vault.yaml] helm_params: name: vault namespace: vault Vault Helm chart: Service appProtocol {#vault-service-appprotocol} hashicorp/vault-helm#1108, opened in April 2025 and at 9 thumbs up, asks for appProtocol: https on the Vault Service. Without it, Gateway API implementations such as GKE's Gateway controller talk plain HTTP to a Vault that serves HTTPS, according to the issue and the GKE documentation it links. No maintainer has replied. Chart 0.34.1, released on 13 August 2026, writes the ports in server-service.yaml without it: ports: - name: {{ include "vault.scheme" . }} port: {{ .Values.server.service.port }} targetPort: {{ .Values.server.service.targetPort }} {{- if and (.Values.server.service.nodePort) (eq (.Values.server.service.type | toString) "NodePort") }} nodePort: {{ .Values.server.service.nodePort }} {{- end }} - name: https-internal port: 8201 targetPort: 8201 The port takes its name from the scheme, https once global.tlsDisable is false, which this values file sets: global: tlsDisable: false resources: - helm.yaml patches: - target: kind: Service name: vault patch: |- - op: test path: /spec/ports/0/name value: https - op: add path: /spec/ports/0/appProtocol value: https parameters: mutations: vault-app-protocol: match: {apiVersion: v1, kind: Service, name: vault} json6902: - {op: test, path: /spec/ports/0/name, value: https} - {op: add, path: /spec/ports/0/appProtocol, value: https} test makes both patches refuse to mark a plain HTTP port as https. With the chart's default tlsDisable: true the first port is named http, and both stopped: Error: testing value /spec/ports/0/name failed: test failed mutate: mutation vault-app-protocol: vault/templates/server-service.yaml: json6902 test of '/spec/ports/0/name' failed The Kustomize target name: vault matched the Service vault only, not vault-internal or vault-agent-injector-svc. The test applied the chart's Services to kind, without the Vault server, and the stored Service carries the field: https 8200 appProtocol=https CloudNativePG cluster chart: topologySpreadConstraints {#cloudnative-pg-topology-spread-constraints} cloudnative-pg/charts#817, opened in February 2026, asks for topologySpreadConstraints on the Cluster that the cluster chart renders. The CloudNativePG operator supports the field, but the chart's cluster.yaml only templates affinity. Pull request #818, opened the same day, adds it. It has had no review and now conflicts with the chart's main branch. In its thread, a user pointed out that the label cnpg.io/cluster alone also matches pooler and Job pods, and suggested adding cnpg.io/podRole: instance to the selector. On node-local storage that selector has a catch. CloudNativePG creates each instance's volume through an initdb or join Job, whose pod gets the instance's constraints but not the label cnpg.io/podRole, so the constraint does not count it. kind's local-path volumes are bound to the node of the first pod that mounts them. In a run with that selector, the join Job of the third instance landed on the node of the second, the volume followed it, and the third instance stayed Pending: 0/4 nodes are available: 1 node(s) didn't match pod topology spread constraints, 1 node(s) had untolerated taint(s), 2 node(s) didn't match PersistentVolume's node affinity. preemption: 0/4 nodes are available: 1 No preemption victims found for incoming pod, 3 Preemption is not helpful for scheduling. A selector that counts the Job pods too and leaves out only poolers spread all three instances. Until a chart release has the value, chart 0.9.0 takes this patch: resources: - helm.yaml patches: - target: group: postgresql.cnpg.io kind: Cluster name: pg-cluster patch: |- - op: add path: /spec/topologySpreadConstraints value: - maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: cnpg.io/cluster: pg-cluster matchExpressions: - key: cnpg.io/podRole operator: NotIn values: [pooler] parameters: mutations: cluster-topology-spread: match: {apiVersion: postgresql.cnpg.io/v1, kind: Cluster, name: pg-cluster} json6902: - op: add path: /spec/topologySpreadConstraints value: - maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: cnpg.io/cluster: pg-cluster matchExpressions: - key: cnpg.io/podRole operator: NotIn values: [pooler] With operator chart 0.29.1 on the cluster and the release named pg, the Cluster pg-cluster became Ready on a fresh cluster. Its three instance pods carry the constraint and run on three different workers: pg-cluster-1 node=set-fields-worker2 ready=true spread=kubernetes.io/hostname pg-cluster-2 node=set-fields-worker ready=true spread=kubernetes.io/hostname pg-cluster-3 node=set-fields-worker3 ready=true spread=kubernetes.io/hostname Harbor Helm chart: annotations on the core Deployment {#harbor-core-deployment-annotations} goharbor/harbor-helm#2221, opened in August 2025, asks for annotations and labels on the components' Deployments, beyond the pod annotations the chart takes. One commenter needs an annotation on harbor-core for Reloader. A collaborator suggested a Helm hook, which the reporter cannot use with Argo CD. Three pull requests for the values were closed unmerged by the stale bot. In chart 1.19.4, released on 16 October 2026, core-dpl.yaml writes no annotations into the Deployment's metadata, and core.podAnnotations, whose comment in values.yaml reads "Additional deployment annotations", goes to the pod template only: metadata: name: {{ template "harbor.core" . }} namespace: {{ .Release.Namespace | quote }} labels: {{ include "harbor.labels" . | indent 4 }} component: core app.kubernetes.io/component: core An annotation is a key in a map, which a strategic merge patch and mutate's merge add without touching other keys: resources: - helm.yaml patches: - patch: |- apiVersion: apps/v1 kind: Deployment metadata: name: harbor-core namespace: harbor annotations: reloader.stakater.com/auto: "true" parameters: mutations: harbor-core-annotations: match: {apiVersion: apps/v1, kind: Deployment, name: harbor-core} merge: metadata: annotations: reloader.stakater.com/auto: "true" Harbor generates secrets on every render, so two renders differ in their Secrets and in the Deployments' checksum annotations. The comparison of the renders left those out. On kind, every Harbor Deployment became Available, and the stored harbor-core carries the annotation: harbor-core annotations: {"deployment.kubernetes.io/revision":"1","reloader.stakater.com/auto":"true"} ECK operator chart: ServiceMonitor labels and interval {#eck-operator-servicemonitor} elastic/cloud-on-k8s#7717, opened in April 2024, asks for labels and the interval on the operator's ServiceMonitor, because a Prometheus from kube-prometheus-stack selects ServiceMonitors by its release label. A commenter confirmed in August 2026 that 3.5.0 ignores serviceMonitor.labels and keeps a second ServiceMonitor by hand. The chart has no values schema, so the key is silently dropped. In 3.5.0, released on 4 August 2026, service-monitor.yaml renders the ServiceMonitor only with metrics in secure mode, with the chart's own labels and a fixed interval: {{- if and .Values.config.metrics.secureMode.enabled .Values.serviceMonitor.enabled }} apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: {{ include "eck-operator.fullname" . }} namespace: {{ ternary .Values.serviceMonitor.namespace .Release.Namespace (not (and (.Values.serviceMonitor) (empty .Values.serviceMonitor.namespace))) }} labels: {{- include "eck-operator.labels" . | nindent 4 }} spec: namespaceSelector: matchNames: - {{ .Release.Namespace }} selector: matchLabels: app.kubernetes.io/name: {{ include "eck-operator.name" . }}-metrics-service app.kubernetes.io/instance: {{ .Release.Name }} endpoints: - port: https path: /metrics scheme: https interval: 30s These values turn the metrics endpoint and secure mode on: config: metrics: port: "8080" secureMode: enabled: true resources: - helm.yaml patches: - target: group: monitoring.coreos.com kind: ServiceMonitor name: elastic-operator patch: |- - op: add path: /metadata/labels/release value: kube-prometheus-stack - op: test path: /spec/endpoints/0/interval value: 30s - op: replace path: /spec/endpoints/0/interval value: 60s parameters: mutations: eck-servicemonitor: match: {apiVersion: monitoring.coreos.com/v1, kind: ServiceMonitor, name: elastic-operator} json6902: - {op: add, path: /metadata/labels/release, value: kube-prometheus-stack} - {op: test, path: /spec/endpoints/0/interval, value: 30s} - {op: replace, path: /spec/endpoints/0/interval, value: 60s} add on /metadata/labels/release adds one key to the existing labels. The test of 30s stops the patch if a release changes the interval. On kind, with the ServiceMonitor CRD of kube-prometheus-stack 92.11.0, the operator became ready and the stored ServiceMonitor has both values: release=kube-prometheus-stack interval=60s Temporal Helm chart: Service trafficDistribution {#temporal-service-trafficdistribution} temporalio/helm-charts#897, opened in April 2026, asks for spec.trafficDistribution on the server Services, at least the frontend. Its author sets the field today with a separate server-side apply after each install. No maintainer has replied. Chart 1.7.0, released on 12 September 2026, writes type, ports and selector in server-service.yaml and has no values schema, so server.frontend.service.trafficDistribution is ignored. The chart needs a database to render, and this values file names one that the test never contacted: server: config: persistence: datastores: default: sql: pluginName: postgres12 databaseName: temporal connectAddr: db.example.internal:5432 visibility: sql: pluginName: postgres12 databaseName: temporal_visibility connectAddr: db.example.internal:5432 resources: - helm.yaml patches: - target: kind: Service name: temporal-frontend patch: |- - op: add path: /spec/trafficDistribution value: PreferSameZone parameters: mutations: temporal-traffic-distribution: match: {apiVersion: v1, kind: Service, name: temporal-frontend} json6902: - {op: add, path: /spec/trafficDistribution, value: PreferSameZone} PreferClose, the value in the issue, is the older name for PreferSameZone, and the Kubernetes documentation lists it as deprecated. The field is generally available since Kubernetes 1.33. The value PreferSameZone is enabled by default from 1.34, so a 1.33 cluster needs PreferClose. Without a database the Temporal servers cannot start, so the test applied the chart's Services only, and the API server stored the field: temporal-frontend trafficDistribution: PreferSameZone MetalLB Helm chart: the speaker without host networking {#metallb-speaker-hostnetwork} metallb/metallb#3005, opened in April 2026, asks for hostNetwork: false on the speaker DaemonSet. The reporter runs FRR on every node next to MetalLB, and FRR does not accept iBGP peering with an address in its own network namespace, so the speaker has to run in the pod network and peer with the node's FRR. Another user peers the speakers with Calico the same way. The maintainers asked for the use case. Pull request #3099 adds a speaker.hostNetwork value, and on it the maintainer fedepaol wrote "I think it makes sense to document this, mentioning that l2 won't be able to work." It is not merged. Chart 0.16.1 hard-codes the field in speaker.yaml: hostNetwork: true By default the chart runs BGP in a separate frr-k8s DaemonSet, which keeps host networking whatever the speaker does. The test therefore uses MetalLB's native mode, in which the speaker itself speaks BGP: frrk8s: enabled: false speaker: frr: enabled: false Flipping hostNetwork alone is not enough in 0.16.1. The speaker serves its health endpoint on 127.0.0.1 inside its own network namespace (speaker/main.go), and the chart's probes ask for 127.0.0.1, which the kubelet resolves in the node's network namespace. With only the field changed, no speaker became ready on kind, and every probe was refused: Readiness probe failed: Get "http://127.0.0.1:17472/readyz": dial tcp 127.0.0.1:17472: connect: connection refused Liveness probe failed: Get "http://127.0.0.1:17472/healthz": dial tcp 127.0.0.1:17472: connect: connection refused Pull request #3099 changes the field only, so it would leave the same probes in place. The patch below also removes both probes. The speaker then has no health check, and the kubelet restarts it only when the process exits: resources: - helm.yaml patches: - target: kind: DaemonSet name: metallb-speaker patch: |- - op: test path: /spec/template/spec/hostNetwork value: true - op: replace path: /spec/template/spec/hostNetwork value: false - op: test path: /spec/template/spec/containers/0/name value: speaker - op: remove path: /spec/template/spec/containers/0/livenessProbe - op: remove path: /spec/template/spec/containers/0/readinessProbe parameters: mutations: metallb-speaker-pod-network: match: {apiVersion: apps/v1, kind: DaemonSet, name: metallb-speaker} json6902: - {op: test, path: /spec/template/spec/hostNetwork, value: true} - {op: replace, path: /spec/template/spec/hostNetwork, value: false} - {op: test, path: /spec/template/spec/containers/0/name, value: speaker} - {op: remove, path: /spec/template/spec/containers/0/livenessProbe} - {op: remove, path: /spec/template/spec/containers/0/readinessProbe} On kind, a speaker ran on each of the four nodes with its own pod IP. Without a readiness probe, ready means that the container runs: metallb-speaker-6nsjd hostNetwork=false pod IP is the node IP: no ready=true restarts=0 metallb-speaker-6rr28 hostNetwork=false pod IP is the node IP: no ready=true restarts=0 metallb-speaker-8z7rr hostNetwork=false pod IP is the node IP: no ready=true restarts=0 metallb-speaker-hc622 hostNetwork=false pod IP is the node IP: no ready=true restarts=0 L2 mode answers ARP requests for the service address on the node's interface, which a speaker in the pod network does not have. With an L2 pool from the kind network and an nginx LoadBalancer Service, the test compared both settings from the host: speaker in the pod network: the LoadBalancer address from the host: not reachable speaker in the host network (plain chart): the LoadBalancer address from the host: reachable The test did not set up a BGP peer, so it shows that the speakers run in the pod network, not that they announce routes over BGP from there. Consul Helm chart: tolerations on the hook Jobs {#consul-hook-job-tolerations} hashicorp/consul-k8s#4469, opened in January 2025 and at 7 thumbs up, asks for tolerations and a nodeSelector on the Job that generates the gossip encryption key. When every node carries a taint, that Job and tls-init stay Pending, and the thread names a third Job, tls-init-cleanup. The workaround offered there is to helm pull the chart and edit its templates, which a commenter who installs from HashiCorp's repository cannot do. Pull request #4625, opened in June 2025, adds the values and has had no review. In chart 2.0.4, released on 11 September 2026, none of the three Jobs sets either field in its pod spec, and 2.1.0-rc2 does not either. All three are Helm hooks, as tls-init-job.yaml shows for tls-init: annotations: "helm.sh/hook": pre-install,pre-upgrade "helm.sh/hook-weight": "1" "helm.sh/hook-delete-policy": hook-succeeded,before-hook-creation The gossip key Job runs at the same hooks, tls-init-cleanup at pre-delete. These values turn on TLS and the gossip key, and global.name gives the Jobs the names from the issue: global: name: consul tls: enabled: true gossipEncryption: autoGenerate: true The test tainted and labelled all three workers with example.com/purpose=infra and set the chart's own tolerations and nodeSelector values for the server, the connect injector and the webhook cert manager. global.acls.tolerations reaches the consul-gateway-resources Job, which a commenter also mentions, and the two gateway cleanup Jobs, even with ACLs off. Those three have no nodeSelector value. With the plain chart, both pre-install Jobs stayed Pending: consul-tls-init-fh7nh: 0/4 nodes are available: 4 node(s) had untolerated taint(s). preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling. The three Jobs carry app: consul whatever the release is called, and component with their own name, so one Kustomize target with a set-based selector matches all three: resources: - helm.yaml patches: - target: group: batch kind: Job labelSelector: app=consul,component in (tls-init,tls-init-cleanup,gossip-encryption-autogenerate) patch: |- - op: add path: /spec/template/spec/tolerations value: - key: example.com/purpose operator: Equal value: infra effect: NoSchedule - op: add path: /spec/template/spec/nodeSelector value: example.com/purpose: infra mutate's match takes one value per key, so each Job gets a mutation of its own, and the three share the operations through an inventory key: parameters: hook_scheduling: - op: add path: /spec/template/spec/tolerations value: - {key: example.com/purpose, operator: Equal, value: infra, effect: NoSchedule} - op: add path: /spec/template/spec/nodeSelector value: example.com/purpose: infra mutations: consul-tls-init: match: {apiVersion: batch/v1, kind: Job, labels: {app: consul, component: tls-init}} json6902: ${hook_scheduling} consul-tls-init-cleanup: match: {apiVersion: batch/v1, kind: Job, labels: {app: consul, component: tls-init-cleanup}} json6902: ${hook_scheduling} consul-gossip-encryption-autogenerate: match: {apiVersion: batch/v1, kind: Job, labels: {app: consul, component: gossip-encryption-autogenerate}} json6902: ${hook_scheduling} Without a cluster to ask, helm template renders the chart's two PodDisruptionBudgets as policy/v1beta1, which Kubernetes 1.25 removed, and the first server-side dry run refused them. Both renders therefore pass --api-versions policy/v1/PodDisruptionBudget, which Kapitan takes as api_versions in helm_params. Helm hooks with the patched Jobs kubectl apply of the whole render would also start tls-init-cleanup, which deletes the Secrets consul-ca-cert, consul-ca-key and consul-server-cert. The test applied the mutate output in the order of helm install: the pre-install hooks, the templates, then the post-install Job. The server and the connect injector became ready. The pre-delete hooks, applied last as helm uninstall would run them, removed the three Secrets. Every Job completed on a tainted worker: consul-tls-init node=cac-consul-worker2 taint=example.com/purpose=infra:NoSchedule complete=True consul-gossip-encryption-autogenerate node=cac-consul-worker taint=example.com/purpose=infra:NoSchedule complete=True consul-tls-init-cleanup node=cac-consul-worker2 taint=example.com/purpose=infra:NoSchedule complete=True For helm install itself, Helm 4 passes hooks to the post-renderer together with the templates. Helm 3 never did. With Helm 3.22.1 and the same Kustomize patch as a post-renderer script, helm template printed the three Jobs without the fields. A Helm 4 post-renderer is a plugin, and this one runs the Kustomize patch from the directory it is given: name: kustomize-pr version: 0.1.0 type: postrenderer/v1 apiVersion: v1 runtime: subprocess runtimeConfig: platformCommand: - command: "${HELM_PLUGIN_DIR}/kustomize.sh" #!/usr/bin/env bash # stdin: Helm's render, hooks included. $1: the directory with kustomization.yaml. set -euo pipefail tmp=$(mktemp -d) trap 'rm -rf "$tmp"' EXIT cp "$1/kustomization.yaml" "$tmp/" cat > "$tmp/helm.yaml" kustomize build "$tmp" helm plugin install ./kustomize-pr helm install consul consul --repo https://helm.releases.hashicorp.com --version 2.0.4 \ -n consul -f consul-values.yaml --post-renderer kustomize-pr --post-renderer-args "$PWD" --wait The plugin's output equalled the Kustomize build. With Helm 4.3.1 on kind, the release installed, and helm uninstall ran tls-init-cleanup, each Job on a tainted worker: Successfully assigned consul/consul-gossip-encryption-autogenerate-jdm6k to cac-consul-worker Successfully assigned consul/consul-tls-init-qt7ql to cac-consul-worker3 Successfully assigned consul/consul-tls-init-cleanup-wkt87 to cac-consul-worker APISIX ingress controller chart: pod labels {#apisix-ingress-controller-pod-labels} apache/apisix-helm-chart#978, opened in June 2026 and at 11 thumbs up, asks for a podLabels value in the apisix-ingress-controller chart, so that the controller's pods can carry labels for observability, scheduling or policies. Nobody has commented. The apisix chart in the same repository got podLabels through pull request #965, merged on 30 September 2026. The controller chart takes deployment.podAnnotations, but in 1.4.0, released on 17 September 2026, deployment.yaml gives the pod template the selector labels only: labels: {{- include "apisix-ingress-controller-manager.selectorLabels" . | nindent 8 }} The chart's labelsOverride replaces these selector labels, so a label set there also lands in the Deployment's spec.selector and in the selectors of both Services. Over the installed chart, the API server refused the change: The Deployment "apisix-ingress-controller" is invalid: spec.selector: Invalid value: {"matchLabels":{"app.kubernetes.io/instance":"apisix-ingress-controller","app.kubernetes.io/name":"apisix-ingress-controller","team":"platform"}}: field is immutable add on a path below the pod template's labels sets one label there and leaves the selector alone: resources: - helm.yaml patches: - target: group: apps kind: Deployment name: apisix-ingress-controller patch: |- - op: add path: /spec/template/metadata/labels/team value: platform parameters: mutations: apisix-pod-labels: match: {apiVersion: apps/v1, kind: Deployment, name: apisix-ingress-controller} json6902: - {op: add, path: /spec/template/metadata/labels/team, value: platform} A key with a slash such as example.com/team goes into the path as example.com~1team, and both tools then set the label example.com/team. The chart generates its webhook certificate on every render, so the comparison of the renders left out that Secret and the webhook configuration's caBundle. helm template renders the chart's CRDs only with --include-crds, which Kapitan's helm input passes by default. On kind, the controller became ready, and its pod carries the label: apisix-ingress-controller-85b5bb8fb4-9m4kc labels: {"app.kubernetes.io/instance":"apisix-ingress-controller","app.kubernetes.io/name":"apisix-ingress-controller","pod-template-hash":"85b5bb8fb4","team":"platform"} Cilium Helm chart: ipFamilyPolicy on cilium-ingress {#cilium-ingress-ipfamilypolicy} cilium/cilium#27003, opened in July 2023, asks for ipFamilyPolicy and ipFamilies on the Service of Cilium's ingress controller. Without them the Service gets the cluster's primary family only, so on a dual-stack cluster the Ingresses answer on IPv4 or on IPv6, not on both. The maintainer youngnick wrote in July 2023 that LB Service customisation was planned for that cycle, and pinned the issue in December 2023. Chart 1.20.3, released on 15 October 2026, renders the Service with its ports, type, loadBalancerClass, allocateLoadBalancerNodePorts, loadBalancerIP and externalTrafficPolicy, and 1.21.0-pre.3 renders the same. The chart has this Service in shared mode only. In dedicated mode the operator creates a Service per Ingress, which a patch of the chart does not reach. The ingress controller needs Cilium's kube-proxy replacement, so the test ran Cilium as the CNI of a dual-stack kind cluster without kube-proxy: kubeProxyReplacement: true k8sServiceHost: cac-cilium-control-plane k8sServicePort: 6443 ipam: mode: kubernetes ipv6: enabled: true ingressController: enabled: true loadbalancerMode: shared resources: - helm.yaml patches: - target: kind: Service name: cilium-ingress patch: |- - op: add path: /spec/ipFamilyPolicy value: RequireDualStack - op: add path: /spec/ipFamilies value: [IPv4, IPv6] parameters: mutations: cilium-ingress-dual-stack: match: {apiVersion: v1, kind: Service, name: cilium-ingress} json6902: - {op: add, path: /spec/ipFamilyPolicy, value: RequireDualStack} - {op: add, path: /spec/ipFamilies, value: [IPv4, IPv6]} The chart generates the Hubble certificates on every render, so the comparison of the renders left out those Secrets. The plain chart gave the Service one IPv4 cluster IP. The mutate output, applied over it, changed the existing Service in place, and 60 seconds later the operator had not reverted it: ipFamilyPolicy=SingleStack ipFamilies=["IPv4"] clusterIPs=["10.96.179.251"] loadBalancer=198.51.100.0 ipFamilyPolicy=RequireDualStack ipFamilies=["IPv4","IPv6"] clusterIPs=["10.96.179.251","fd00:10:96::b12c"] loadBalancer=198.51.100.0 2001:db8:: LB-IPAM had a pool with one block of each family from the documentation ranges and gave the Service an address of each. An Ingress for an echo backend then answered on all four addresses from a node, with curl in the node's network namespace: 10.96.179.251 from node: 200, from pod: 000 fd00:10:96::b12c from node: 200, from pod: 200 198.51.100.0 from node: 200, from pod: 000 2001:db8:: from node: 200, from pod: 200 From a pod, requests over IPv4 timed out on this test host, with the plain chart as well and on an IPv4-only kind cluster too. The patch does not cause it, and the test does not cover that path. ipFamilies with both families needs a dual-stack cluster, whatever the policy. On an IPv4-only kind cluster the server-side dry run refused the patched Service: * spec.ipFamilyPolicy: Invalid value: "RequireDualStack": this cluster is not configured for dual-stack services * spec.ipFamilies[1]: Invalid value: "IPv6": not configured on this cluster With PreferDualStack it still refused ipFamilies[1]. PreferDualStack without ipFamilies passed there and got IPv4 only. Originally published at kadet-hub.org.

Your AI workflow can survive a worker crash. Retried API calls can still duplicate records. Temporal: Document → LLM Activity → Checks → Approval → API Activity Replay reuses recorded results. External writes need idempotency. temporalio/temporal t.co

Your logs might be hiding your most important errors. Shared rate limit → one noisy log line drains it → rare errors silently dropped. Found this in @temporalio and opened my first PR: one token bucket per message. Go check your throttled logger 👇 temporal#12680

@_alejandroao Well, it's a great time to learn Temporal: temporalio/pi-temporal More details here: temporal.io

temporal ⭐ 24k Temporal is an open-source platform for developers to build fault-tolerant, scalable workflows. 🔗 temporalio/temporal #opensource t.co

Temporal: durable execution platform (23.7k stars) What it does All text below is quoted directly from the project README. >Temporal is a durable execution platform that enables developers to build scalable applications without sacrificing productivity or reliability. Introduction: - The Temporal server executes units of application logic called Workflows in a resilient manner that automatically handles intermittent failures, and retries failed operations. - Temporal is a mature technology that originated as a fork of Uber's Cadence. - It is developed by Temporal Technologies, a startup by the creators of Cadence. - Execute the following commands to start a pre-built image along with all the dependencies. - Use Temporal CLI to interact with the running Temporal server. - Try Temporal Web UI by opening localhost for viewing your sample workflows executing on Temporal. - This repository contains the source code of the Temporal server. To implement Workflows, Activities and Workers, use one of the supported languages. >brew install temporal && temporal server start-dev Repository temporalio/temporal Repository stats |Metric|Value| |:-|:-| |Stars|23,742| |Forks|2,012| |Last release|v1.32.1 (2026-10-08)| |Languages|Go 99.5%, Shell 0.2%, PLpgSQL 0.1%| |Contributors|316| |AI contributor accounts (Claude / Codex / Cursor / Copilot)|Copilot: 2, cursoragent: 7, claude: 251| |AI co-authored commits|Claude: 256, Cursor: 7, Copilot: 2| Main contributor |Field|Value| |:-|:-| |Name|Wenquan Xing| |GitHub|@wxing1292| |Commits to this repo|1,045 (top contributor)| |Followers|86| |Public repos|6| |On GitHub since|2014-09-13|

chaptermill: a podcast's back catalog to show notes for 3¢ an audio hour, on an old PC I kept crashing on purpose This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle What I Built Most podcasts that have been running for a few years have the same problem: hundreds of old episodes with no transcript, no chapters and no show notes worth reading. That archive is what new listeners stumble on through search, and it is the part nobody wants to pay per-hour SaaS prices to fix. chaptermill takes a podcast's RSS feed and, on a machine you already own, turns the back catalog into the files a producer actually delivers: - a timestamped transcript (SRT, VTT, JSON), - chapters as a Podcasting 2.0 chapters.json and as YouTube chapter lines, - 2–4 pull quotes, each checked word for word against the transcript before it is accepted, - Markdown show notes with a summary, topics and a credit line. Who pays for it: freelance podcast producers (show notes are a standard deliverable) and independent shows with a big archive. The hosted tools price by the audio hour. On their pricing pages today, Castmagic's Hobby plan is $19/month billed annually for 30 transcribed hours in total, and its Team plan, 400 hours, is $139/month billed annually; Podsqueeze's Agency Lite plan is $89/month for 600 minutes per month. A 300-episode archive of half-hour shows is 150 hours. With chaptermill, at the rate of my demo run, the same archive costs about $4.40 in open-weight model calls and a day of runtime on a 2015-era desktop. The side hustle writes itself: a flat-fee "archive pass" for small shows, where the marginal cost is cents. For scale: Hacker Public Radio, whose episodes I used for the demo, numbers its episodes past 4,700. At the average length of the nine I processed (28.5 minutes) that is roughly 2,250 hours, or about $65 of model calls at my demo run's rate. That is an estimate from a sample of nine, not a quote. To be fair to the incumbents: Apple Podcasts and Spotify now generate chapters for some shows inside their own apps. That helps listeners on those apps. It does not give the producer a file to put in the feed, on YouTube or on the show's website, and it does not touch the archive in one go. Demo **Live: bowen1314.github.io !The chaptermill demo page: run numbers at the top, the episode list on the left, and one episode's player, summary and clickable chapters on the right Nine recent episodes of Hacker Public Radio (a community podcast released under CC BY-SA 4.0), processed on my home server (one run, plus a re-run of two episodes that I explain below), with nothing edited by hand. Pick an episode; press a chapter or a quote and the player jumps there; open the full transcript and the current line follows the audio. Every file is downloadable. Further down the page is the part I am proudest of: the report from a second run during which a script SIGKILLed the worker 9 times and the Temporal server once, while the worker injected model-API 503s, cut-off downloads and failing tool calls on purpose. Code {% embed Bowen1314/chaptermill %} MIT licensed. About 1,900 lines of Python plus 560 lines of tests, including workflow tests on Temporal's time-skipping test server. How I Built It Three open pieces do all the work: | Piece | What it does | License | |---|---|---| | faster-whisper base.en | transcription, locally, on 2 CPU threads | MIT | | Qwen3-235B-A22B-Instruct-2507 | the show-notes agent, through Nebius Token Factory's OpenAI-compatible API | Apache-2.0 | | Temporal (dev server + Python SDK) | durable orchestration, retries, signals, queries | MIT | The machine is a shared home server: an Intel i3-6100 (2 cores), 3.8 GB of RAM and one spinning disk, also running a dozen other services. The shape of it A CatalogWorkflow reads the feed and starts one child EpisodeWorkflow per episode, two at a time, inside a dollar budget. Each episode workflow downloads the audio, cuts it into 5-minute chunks (one activity per chunk, on its own task queue so Whisper never starves the rest), merges the transcript, then runs the agent loop and writes the files. The agent gets a one-line-per-minute outline and three tools: read_transcript(start, end), search_transcript(query) and submit_notes(...). Every model call and every tool call is a separate Temporal activity, so each one has its own timeout and retry policy, and a crash in turn 6 resumes at turn 6: for _turn in range(agent.MAX_TURNS): await self._wait_for_cap() # durable wait for a raise_cap signal if over budget resp = await workflow.execute_activity_method( Activities.llm_turn, LlmRequest(model=inp.model, messages=messages, tools=agent.TOOLS, ...), start_to_close_timeout=timedelta(minutes=3), heartbeat_timeout=timedelta(seconds=30), # a dead worker is noticed in 30 s retry_policy=LLM_RETRY, # 429/5xx retried, other 4xx fail fast ) messages.append(resp.message) for call in resp.message.get("tool_calls") or []: if call["function"]["name"] == "submit_notes": v = await workflow.execute_activity_method(Activities.validate_notes, ...) out = "Accepted." if v.ok else "Rejected. Fix these and call submit_notes again:\n- " + "\n- ".join(v.errors) ... # read_transcript / search_transcript: one activity each messages.append({"role": "tool", "tool_call_id": call["id"], "content": out}) Audio, transcripts and chunk results stay on disk; workflow history carries file paths and the agent's conversation, nothing bigger. The model is checked, not trusted submit_notes does not end the loop. A validator compares the submission with the transcript, and if anything is off, the error goes back to the model as the tool result. In the demo run it rejected 4 submissions; the model's next submission fixed each of them. Two of them are worth showing, because they are opposite failures: - HPR4749: the model's first quote silently cut a sentence out of the middle ("And from the website, I'm gonna quote it here.") and stitched the rest together. Rejected; the resubmission quoted the host exactly. - HPR4747: the model wrote "in place of cat", which is what the host said (the Unix command). Whisper had transcribed it as "CAD". The validator rejected the correct word, and the final quote says "CAD". That second one is the honest limit of this design: quotes are guaranteed to match the transcript, not the audio. A custom vocabulary for Whisper is the obvious next step. Making it durable, for real Temporal's AI cookbook patterns map almost one to one onto this pipeline: - Retries in one place. The HTTP client's own retries are off. Errors are classified: 408/409/429/5xx are retryable, other 4xx fail fast, and a Retry-After header becomes the activity's next_retry_delay. - Idempotent model calls. Each reply is written to a ledger and to a cache keyed by a hash of the request before the activity returns, so if the worker dies a millisecond later, the retry reads the cache instead of buying the reply again. - Chunks as checkpoints. A finished chunk is a file. Whisper runs at temperature 0, so a re-run chunk produces the same text. - Heartbeats on every long activity, including a side thread that heartbeats while the CPU activity is stuck loading the model. - Budget as a durable wait. Episodes that would go over the run's budget wait in workflow.wait_condition until a raise_budget signal arrives, for as long as it takes; pause/resume signals and a progress query come for free. Then I tried to break it. scripts/chaos.py started a three-episode catalog and, every 45 to 110 seconds, SIGKILLed the worker, kept it down for 5 to 20 seconds and started it again; on the third kill it took the Temporal server down too. On top of that, the worker failed 30% of model calls with a fake 503, cut off 50% of audio downloads and broke 15% of transcript tool calls, each only on an activity's first two attempts. The result: | | | |---|---| | Kills | 9 SIGKILLs of the worker at random moments, one of them together with the Temporal server | | Episodes finished | 3 of 3, in 12.5 minutes, with no manual intervention | | Transcripts vs. the undisturbed run | identical, segment for segment (584 segments) | | Activity attempts | 88 for 67 activities: 8 retried transcription chunks, 8 retried model calls, 3 retried downloads, 2 retried tool calls | | Model replies received | 18, none bought twice | | Requests in flight at a kill | 1: its reply died with the process and the retry asked again (the provider may have billed both) | | Fake 503s absorbed | 6 | | Model cost | $0.021 | Each kill cost at most the one chunk or model call in progress. The unluckiest chunk, in HPR4740, was killed three times and finished on its fourth attempt; nothing before it was redone. The box itself added chaos I had not planned. During the clean run, another service on the same disk was writing about 16 MB/s, and Linux reported I/O pressure between 40% and 87%. The Temporal dev server keeps everything in SQLite, so every history write queued behind that disk, and a few tasks were handed to my worker after the worker had stopped waiting for them. Temporal's answer was a timeout and a retry; the run finished anyway, with 20 workflow-task timeouts in its history. Halfway through I restarted the server with SQLite in WAL mode and synchronous=normal (still safe when the process is killed, which is the failure I care about here): before the switch 19 of 78 workflow tasks timed out, after it 1 of 222. Not a controlled experiment, but a big enough difference that the chaos script now starts the server that way. I also learned not to fsync inside an async activity: on that disk a single fsync blocked the event loop the workflows share for seconds. Disk-heavy activities now run as sync activities on a thread pool. Numbers from the demo run | | | |---|---| | Episodes | 9 Hacker Public Radio episodes, 4.3 h of audio, 38,318 words | | Wall clock | 44 minutes, two episodes at a time | | Transcription | 26 minutes for 4.3 h of audio (9.8× real time), Whisper base.en on 2 CPU threads | | Agent | 72 turns, 91 tool calls, 4 rejected submissions, all fixed | | Model replies paid for | 74, none bought twice: $0.125 in total, $0.029 per audio hour | | Difference from HPR's own transcripts | 12.6% of words (HPR's are also made with Whisper and not proofread, so this is two machine transcripts disagreeing, not an error rate) | One honest miss: the first run finished 7 of 9. On the two longest interviews the agent spent all ten of its turns reading the transcript window by window and never called submit_notes. I added a "three turns left, submit now" message and re-ran just those two. Every transcript chunk and the first nine model replies of each came straight from cache; only the last turn was new, at about a third of a cent each. The $0.125 above includes the wasted turns. Why Does Open Innovation Matter? Cost. The expensive half of this job, turning audio into text, costs nothing per minute when the model runs on your own CPU. OpenAI lists its Whisper API at an estimated $0.006 per minute, $0.36 per audio hour; my entire language-model bill was $0.029 per audio hour. Open weights move the dominant cost from "per minute, forever" to an old PC's spare time. Privacy. The audio never leaves the machine. Only transcript text goes to the model endpoint, and because the model is open-weight, a producer whose clients will not allow even that can point chaptermill at their own server. (I want to be precise: for the demo I used a hosted endpoint; I did not run the 235B model at home.) Reproducibility and no lock-in. The Whisper weights are a file on my disk that nobody can update or retire under me, so running an episode again gives the same transcript; the chaos run shows it, with transcripts identical segment for segment to the undisturbed run. The agent talks to any OpenAI-compatible server, so switching model or provider is a config change, not a rewrite. I did try Gemma 3 27B first and it ignored the tool definitions on that endpoint, so "any" means any model whose tool calling works. Fine-tuning for a niche. The "cat"/"CAD" miss above is exactly the kind of error open weights let you fix yourself. A producer whose shows are full of jargon (Unix commands, drug names, player names) can fine-tune base.en on a few hours of their own corrected transcripts, or at least hand Whisper a vocabulary prompt, and run the result on the same CPU. A closed transcription API may let you pass a prompt, but you cannot retrain it on your show. I have not done this yet; it is next on the list. The orchestration is open too. Temporal's dev server ran on the same old box, with a SQLite file, no account and no cloud. Prize Categories Temporal. The category asks for an agent that "survives failures, retries flaky tool calls, and picks up where it left off". chaptermill's agent is a Temporal workflow where every model call and tool call is an activity, and the demo page shows the evidence from a run where I killed the worker 9 times and the server once.

most system design pattern for large projects applicable to rust 1:1 as for skills, did similar research 3 months ago, and couldn’t find anything comprehensive … except this link github.com/leonardomso/rust-skills/tree/master/ru… ended up asking claude for code review 3 time and logging best practices in markdown gives compound effect as a heavy temporal.io user i was never really bothered with async tokio rust gotchas temporalio/sdk-rust python libs written in rust under hood is a good source of rustish domain specific best practices too … if you are ok to see lot of “unsafe” code and clever tricks