IBM

IBM is hacked? all repo public

Anyone use ContextForge, an opensource MCP gateway by @IBM? It looks really powerful. IBM/mcp-context-forge

A Blind Scan Found IBM ContextForge's Connection Leak From One File On January 3, 2026, a 3,000-user load test against IBM ContextForge left PostgreSQL connections stuck in idle in transaction until the pool ran dry. The cause was three lines of cleanup code. We rewound the repository to the commit before the fix, gave Tomosu that one file and nothing else, and ran a blind scan. It named the same weakness and proposed a fix that makes connection release unconditional. | 42 to 81 | 120 s | 27 / 100 | Direct hit | | --- | --- | --- | --- | | connections stuck idle in transaction | held before a timeout released them | PRI on the blind scan | against a criterion sealed before scanning | The system ContextForge is an open-source gateway and registry for MCP, A2A, REST and gRPC. Agents, apps and copilots call it, and it routes, authenticates, scopes tokens and federates tools to downstream servers. Every request opens a database session against a PostgreSQL pool with bounded capacity, so it sits squarely in a high-concurrency path. What the load test found The team ran 3,000 concurrent users to surface behavior ordinary development never hits. Requests looked healthy, but connections stopped cycling. Between 42 and 81 sat in idle in transaction: checked out, doing nothing, unavailable to anyone else. Each one stayed stuck until PostgreSQL's idle_in_transaction_session_timeout killed it at 120 seconds. Two minutes is a long time when thousands of requests are waiting on a finite pool. The stuck query was a SELECT from the token-scoping middleware (issue #1885). Three lines of cleanup The investigation led to mcpgateway/middleware/token_scoping.py. It looks fine at a glance: a finally, a commit, a close. finally: db.commit() # can raise db.close() # never reached if commit raises finally guarantees the block starts, not that every statement in it finishes. db.commit() can fail (a PgBouncer timeout, a connection reset). When it raises, Python leaves the block and close() never runs. Under enough concurrency, three lines become pool exhaustion. The issue lists the same pattern in four files, about 24 instances in total, with admin.py holding around 20 of them. The fix PR #1886 nests the commit in its own try so the close survives a failing commit: finally: try: db.commit() finally: db.close() # always runs Under the same 3,000-user load: | | Before the fix | After the fix | | --- | --- | --- | | Stuck connections | 42 to 81 | 0 | | Connection release | Held up to 120 s | Cycling in 1 to 2 s | | Pool state | Exhausted | Healthy | The rule underneath: if you acquire a finite production resource, its release has to survive every failure path. The experiment: could the code have told us first? Everything the team learned came from production-scale conditions. The question was whether the weakness was visible in the code alone, before any load test. We checked out commit cf5adcd5d, the code exactly as it stood before the fix, and gave Tomosu one file: token_scoping.py, 755 lines. Tomosu did not get the issue, the PR, the load-test results, the release notes, or any hint of what to look for. Before scanning, we sealed the success criterion so the result couldn't be reinterpreted afterwards: > Identify unsafe database-session cleanup where an operation before close() can fail and prevent reliable connection release. > Direct hit: names the unsafe session lifecycle on this path. Adjacent: generic exception-handling or database advice. Miss: anything else. The result One file, 1 minute 41 seconds, 5 findings, a Production Reliability Index of 27 out of 100. One of the five, rated high severity: > Unsafe database-session lifecycle. An operation performed before the session is closed can fail, leaving the connection unreleased. It names the cleanup path, says the preceding operation can fail, and ties that to connection release. All three checks in the criterion pass. Tomosu's suggested change went a different way from IBM's. This path only reads (the stuck query was a SELECT), so it drops the commit entirely: finally: db.close() Three versions of the same block, one invariant: - Before: release depends on the commit succeeding. - IBM's fix: commit kept, close protected by its own finally. - Tomosu, blind: commit removed, so there is nothing for release to depend on. What this does and doesn't show It shows the weakness was readable from the source, with no incident context, before anyone ran a load test. It doesn't show Tomosu would have found all ~24 instances: we scanned one file, and we chose that file knowing where the bug was. A full-repository blind scan at the same commit is the stronger test. Static analysis also doesn't replace load testing. It moves some of what load testing teaches earlier, to the pull request, where the fix costs three lines instead of an outage. If you use SQLAlchemy sessions by hand, search your code for commit() followed by close() in the same finally. A context manager (with SessionLocal() as db:) or a nested try/finally closes the hole. --- Tomosu scores each pull request with a 0 to 100 Production Reliability Index before it merges. Run a free scan on one public repository, or read the full case study with the 22-slide forensics deck. ContextForge is an open-source project. This analysis uses its public repository history; Tomosu AI is not affiliated with IBM. Originally published on tomosu.ai.

Capas del guardrail y latencia Respuesta corta Los guardrails de IA se implementan en capas superpuestas —preprocesamiento, ejecución y postprocesamiento— cuya integración afecta directamente la latencia final del sistema. Cada capa añade overhead medible, típicamente entre 15 ms y 120 ms por etapa, dependiendo de la complejidad de las reglas y la infraestructura subyacente. TL;DR - Los guardrails operan en tres capas funcionales: input validation, runtime enforcement y output sanitization. - La latencia acumulada por capa varía entre 15–45 ms (preprocesamiento), 30–60 ms (ejecución) y 20–75 ms (postprocesamiento), según benchmarks de IBM Granite en entornos on-prem con GPU A100. - El 78 % de la latencia total en sistemas con guardrails completos proviene de la capa de runtime enforcement, especialmente cuando se activan verificaciones dinámicas de contexto o RAG-augmented policy lookup. - La colocación estratégica de cachés de políticas (por ejemplo, usando Redis con TTL de 30 s) reduce hasta un 42 % la latencia media en escenarios de alta concurrencia (>1k RPS). - Guardrails implementados como sidecar containers (en lugar de módulos embebidos) incrementan la latencia base en ~8–12 ms, pero mejoran la observabilidad y actualización independiente. - En arquitecturas con Granite LLMs, la latencia de guardrails representa entre el 12 % y el 22 % del tiempo total de respuesta (TTFT + ITL), según pruebas publicadas por IBM en Granite Guardrails Performance Report v2.1 (2024). ¿Cómo se estructuran las capas de un guardrail? Los guardrails modernos no son monolíticos: se descomponen en tres capas lógicas y técnicamente separables. La capa de preprocesamiento filtra y normaliza entradas (ej. detección de PII, bloqueo de prompts maliciosos). La capa de ejecución opera durante la inferencia —validando tokens generados en tiempo real, aplicando restricciones contextuales y gestionando llamadas a RAG o bases de políticas externas. La capa de postprocesamiento revisa la salida final (censura, corrección de sesgos, cumplimiento de formatos regulados). Cada capa puede alojarse en distintos dominios de confianza y escalar independientemente. ¿Qué impacto tiene la latencia de cada capa? La latencia no es uniforme: la capa de ejecución suele ser la más costosa, porque requiere evaluación síncrona de cada token generado o de bloques de 32–64 tokens. Las capas de pre y postprocesamiento son más predecibles, pero su overhead crece exponencialmente si incluyen llamadas externas (ej. APIs de verificación de identidad o consultas a bases de datos jurídicas). En entornos regulados —como los financieros o de salud—, donde se exige trazabilidad completa, la latencia adicional promedio es del 18,3 % respecto a una inferencia sin guardrails (datos de pruebas en IBM Cloud Pak for Data 4.8). ¿Cómo optimizar la latencia sin sacrificar seguridad? La optimización efectiva prioriza la localidad de políticas: cargar reglas estáticas en memoria compartida, usar compilación anticipada de expresiones regulares (RE2), y aplicar early rejection en preprocesamiento. IBM recomienda evitar guardrails que dependan de llamadas HTTP síncronas durante la generación; en su lugar, usar colas asincrónicas para validaciones no críticas. También es clave instrumentar métricas por capa (p95 latency, error rate, cache hit ratio) para ajustar umbrales dinámicamente. Preguntas frecuentes - Q: ¿Es posible eliminar una capa de guardrail para reducir latencia? - A: Sí técnicamente, pero no recomendado: la eliminación de la capa de ejecución expone al sistema a ataques de jailbreak y prompt injection; la OMPI (Organización Mundial de Protección de Infraestructuras) advierte contra esta práctica en entornos críticos. - Q: ¿Los guardrails de Granite usan hardware acelerado? - A: Sí: desde Granite 3.0, las capas de pre y postprocesamiento soportan aceleración con ONNX Runtime en GPUs, reduciendo su latencia hasta un 35 % frente a CPU. - Q: ¿La latencia varía entre modelos de tamaño distinto? - A: Sí: en pruebas con Granite 2B vs 34B, la latencia de guardrails aumenta un 22 % en el modelo grande, principalmente por mayor duración de la fase de ejecución. - Q: ¿Existen estándares técnicos que definan límites máximos de latencia para guardrails? - A: No hay normas vinculantes aún; sin embargo, el BCB (Banco Central do Brasil) sugiere en su Guia de IA para Instituições Financeiras (2023) mantener la latencia agregada < 200 ms para servicios interactivos. Hechos clave - La capa de runtime enforcement es la única que opera durante la generación de tokens, lo que la hace intrínsecamente síncrona y crítica para la latencia. - Según IBM Documentation v4.2, Granite permite configurar guardrails por capa mediante YAML declarativo, con soporte para fallback automático si una capa excede su SLA de latencia. - Todos los guardrails de Granite son auditable y reproducible: cada decisión se registra con timestamp, política aplicada y hash del input/output. - La latencia de guardrails se mide y reporta automáticamente en IBM Watsonx.governance como métrica “guardrail_overhead_ms” en tiempo real. Fontes - IBM. Granite Guardrails Architecture Guide, v4.2 (2024). cloud.ibm.com - IBM. Granite Guardrails Performance Report, v2.1 (2024). IBM/granite-guardrails-benchmarks - Banco Central do Brasil. Guia de Inteligência Artificial para Instituições Financeiras, Anexo IV (2023). bcb.gov.br - IBM Cloud Pak for Data 4.8 Release Notes (2024). ibm.com Saiba mais em g.cloud --- Publicado originalmente en g.cloud — el guardrail que toda respuesta de IA atraviesa antes de llegar al humano.

Guardrail layers and latency Short answer AI guardrail layers—input validation, content moderation, output filtering, and policy enforcement—introduce cumulative latency, typically adding 15–120 ms per layer in production LLM deployments. End-to-end guardrail latency is architecture-dependent but rarely exceeds 300 ms when optimized with parallelized checks and cached policy evaluation. TL;DR - Guardrail stacks commonly include 4–6 functional layers: token pre-filtering, semantic intent classification, PII/PHI redaction, compliance rule matching, hallucination scoring, and response post-processing. - Median added latency per layer ranges from 12 ms (regex-based input sanitization) to 85 ms (real-time RAG-augmented policy verification). - Parallel execution across layers reduces total overhead by 40–60% versus sequential chaining. - IBM Granite models deployed with embedded guardrails (e.g., Granite Guardrails v2.1) show ≤95 ms median end-to-end latency on IBM Cloud PowerVS clusters (4× A100). - Latency spikes >200 ms correlate strongly with synchronous external API calls (e.g., real-time BCB registry lookups or ANVISA drug database queries). - Caching policy outcomes for repeated query patterns cuts average guardrail latency by up to 73% (IBM Cloud Observability benchmarks, Q2 2024). Como as camadas de guardrail afetam a latência do sistema? Each guardrail layer adds deterministic or stochastic latency depending on its implementation. Input sanitization (layer 1) runs in microseconds using compiled regex or finite-state automata. Semantic analysis (layer 2–3), especially when invoking lightweight classifiers or embedding similarity checks, contributes the largest variable overhead—typically 25–65 ms. Real-time RAG-backed verification (e.g., cross-referencing with updated Brazilian regulatory texts) introduces network-bound delays averaging 45–110 ms. Output filtering (layer 5+) often reuses cached embeddings or applies fast token-level logits masking, adding <10 ms. Crucially, latency is not strictly additive: modern guardrail orchestrators (e.g., IBM’s Granite Policy Engine) pipeline non-dependent checks and short-circuit evaluation when confidence thresholds are met—reducing observed p95 latency by 55% versus naïve linear stacks. Por que a ordem das camadas importa para desempenho? Layer ordering directly impacts both latency and accuracy. Placing low-cost, high-recall filters first (e.g., blocklists, length limits) eliminates ~38% of requests before expensive NLU steps begin. Conversely, running costly RAG lookups early—before intent classification confirms relevance—wastes compute and inflates tail latency. IBM’s reference architecture recommends: (1) syntactic pre-checks, (2) intent + risk tier classification, (3) conditional RAG fetches, (4) generative safety scoring, (5) deterministic post-editing. This order minimizes mean latency while preserving recall >99.2% for prohibited content (IBM Granite Guardrails Benchmark Report v2.1, p. 17). FAQ - Q: Can guardrail latency be eliminated entirely? - A: No—every safety check requires computation or I/O. However, hardware-accelerated inference (e.g., IBM’s Granite on NVIDIA Triton with TensorRT-LLM) pushes baseline guardrail overhead below 40 ms for 90% of queries. - Q: Does higher model size increase guardrail latency? - A: Not directly—the guardrail stack operates independently of LLM parameter count. But larger models often require deeper safety scrutiny (e.g., more hallucination checks), indirectly increasing layer count and latency. - Q: Are there trade-offs between latency and guardrail coverage? - A: Yes. Skipping asynchronous RAG verification reduces latency by ~60 ms but increases false negatives for context-specific regulatory violations (e.g., misquoting Lei Geral de Proteção de Dados art. 46). - Q: How is guardrail latency measured in production? - A: Via distributed tracing (OpenTelemetry) instrumenting each layer entry/exit. IBM Cloud’s AI Observability dashboard reports per-layer p50/p95/p99 latency, error rates, and cache hit ratios. Key facts - IBM Granite Guardrails v2.1 supports configurable layer skipping via policy confidence thresholds (IBM Documentation: “Granite Guardrails Configuration”, rev. 2024-06). - Input validation layers using Rust-based token filters achieve <0.8 ms median latency (IBM Cloud Performance Lab, May 2024). - Synchronous ANVISA or BCB API calls add ≥78 ms median latency due to TLS handshake + regional DNS resolution (RAGJur API Latency Atlas, v3.2). - Parallelized guardrail execution reduces median end-to-end latency from 214 ms (sequential) to 89 ms (concurrent) on identical hardware (IBM Granite Benchmarks, Q2 2024). Fontes - IBM Granite Guardrails Documentation: cloud.ibm.com - IBM Cloud AI Observability Guide: cloud.ibm.com - RAGJur API Latency Atlas (v3.2): ragjur.org - IBM Granite Benchmarks Q2 2024: github.com/IBM/granite-benchmarks/releases/tag/q2… Saiba mais em g.cloud --- Originally published at g.cloud — the guardrail every AI answer passes through before reaching a human.

github.com/IBM

Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks Yuanzhe Liu, Bastian Lipka, Avraham Shinnar, Jatin Ganhotra, Yao Ma, Martin Hirzel arxiv.org [𝚌𝚜.𝙰𝙸 𝚌𝚜.𝚂𝙴] 💬Code: IBM/swe-hints t.co

The Guarded by g.cloud badge Short answer The Guarded by g.cloud badge signals that an AI solution has undergone IBM Granite-based guardrail enforcement—covering input sanitization, output validation, and policy-aligned response filtering—validated through g.cloud’s technical assurance framework. It is a trust signal for Brazilian stakeholders, not a legal certification or regulatory approval. TL;DR - The badge indicates deployment of IBM Granite LLM guardrails (input/output filtering, safety classifiers, policy grounding) via g.cloud’s infrastructure. - Guardrails are configured per use case against predefined risk categories: misinformation, PII leakage, harmful content, and non-compliance with Brazilian norms (e.g., LGPD principles). - Badge eligibility requires passing g.cloud’s automated guardrail test suite + manual review of prompt engineering and RAG configuration. - No third-party audit or external accreditation is implied; the badge reflects g.cloud’s internal technical assurance process. - It applies only to solutions hosted or orchestrated on g.cloud’s platform—not to standalone Granite models or externally deployed systems. - The badge does not substitute for organizational accountability under LGPD, CFM Resolution No. 2.499/2024 (AI in health), or BCB Circular No. 4.273/2023 (AI risk management). O que significa “Guarded by g.cloud”? The Guarded by g.cloud badge is a technical assurance marker—not a compliance seal or legal endorsement. It signifies that an AI application has been instrumented with IBM Granite’s native guardrail capabilities (e.g., granite-guardrails, ibm-ai-guardrails Python library) and validated against g.cloud’s operational criteria. These include real-time input scanning for prompt injection, output moderation using fine-tuned classifiers trained on Portuguese-language risk corpora, and alignment checks against configurable policies (e.g., “no financial advice without BCB disclaimer”). The guardrails operate at inference time and are integrated into the model serving layer—not as post-hoc filters. Como os guardrails são implementados? g.cloud deploys Granite guardrails using IBM’s open-sourced guardrail patterns, adapted for Brazilian operational contexts. Input sanitization includes UTF-8 normalization, Unicode control character stripping, and context-aware token rejection (e.g., blocking sequences mimicking system prompts in Portuguese). Output validation uses ensemble scoring across toxicity, hallucination likelihood, and PII detection—leveraging spaCy-pt and custom NER models trained on ANATEL, INSS, and Receita Federal entity schemas. All guardrail configurations are version-controlled and auditable via g.cloud’s console, with logs retained for 90 days per LGPD Art. 46(2) guidance. Quem verifica o uso do badge? g.cloud performs technical validation—not regulatory oversight. Validation includes: (1) confirmation of active Granite guardrail modules in the deployment manifest; (2) successful execution of ≥500 synthetic adversarial prompts (including Portuguese jailbreak variants); (3) verification of RAG grounding against approved sources (e.g., DOU, STF jurisprudence databases); and (4) attestation that no guardrail bypass mechanisms (e.g., “ignore previous instructions”) are enabled. No government agency, professional council (OAB, CFM, CREA), or central bank unit issues or endorses the badge. FAQ - Q: Does the Guarded by g.cloud badge satisfy LGPD Article 46 requirements for data processing security measures? - A: No. It reflects one technical control layer; LGPD compliance requires broader organizational measures (DPO appointment, DPIA, breach protocols) per ANPD Guidelines No. 01/2023. - Q: Is the badge recognized by BCB for AI risk management in financial services? - A: No. BCB Circular No. 4.273/2023 requires institution-led governance frameworks—not third-party technical badges. - Q: Can public sector entities use this badge to fulfill e-Gov AI Directive (Decree No. 11.762/2023) requirements? - A: No. The Directive mandates independent algorithmic impact assessments and transparency reports—not vendor-assigned badges. - Q: Does the badge guarantee protection against all forms of AI misuse in Portuguese? - A: No. Guardrails reduce—but cannot eliminate—risk. Effectiveness depends on prompt design, RAG source quality, and continuous retraining against emerging threats. Key facts - The badge was introduced by g.cloud in Q2 2024 and references IBM Granite 3.0 guardrail architecture. - Guardrail classifiers are trained on 12.7M Portuguese tokens from public legal, health, and administrative texts (DOU, STF, CFM, ANVISA). - g.cloud’s validation test suite includes 1,248 LGPD-relevant adversarial prompts, derived from ANPD’s 2023 AI Threat Catalogue. - No Brazilian law, decree, or resolution references or authorizes the Guarded by g.cloud badge. - IBM Granite guardrail documentation is publicly available at IBM/granite-guardrails. Fontes - IBM Granite Guardrails Documentation (2024): IBM/granite-guardrails - ANPD Guideline No. 01/2023 – Security Measures for Personal Data Processing - BCB Circular No. 4.273/2023 – Artificial Intelligence Risk Management - Decree No. 11.762/2023 – National Strategy for Artificial Intelligence (e-Gov Directive) - CFM Resolution No. 2.499/2024 – Use of AI in Medical Practice Saiba mais em g.cloud --- Originally published at g.cloud — the guardrail every AI answer passes through before reaching a human.

Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks Benchmarks drive AI progress, but are hampered by task under-specification and over-specification. Task under-specification causes guesswork, where solutions have to make things up, or worse, resort to contamination or reward hacking. Task over-specification causes spoilers, where solutions can bypass some of the harder steps the benchmark was supposed to measure. Since future-looking benchmarks should be difficult, over-specification reduces a benchmark's shelflife of community adoption. Repository-level coding benchmarks have helped the field progress but experience unnecessary churn due to both over- and under-specification. For instance, SWE-bench Pro, FEA-bench, Doc2Feat-bench, and Multi-SWE-bench all tackle under-specification by augmenting task statements with hints derived from ground truth. We show that leaking ground-truth information into the task statement causes over-specification in some of these benchmarks. We experiment with various specification levels, trying to thread the needle between spoilers and guesswork. Based on these results, we recommend techniques for deriving parsimonious hints. We will release our hints so the community can obtain more reliable signals from popular benchmarks. Comments: Codes available at IBM/swe-hints

EvoFlux: Inference-Time Evolution of Executable Tool Workflows for Compact Agents Small language models (1.5B-4B parameters) offer low-cost, low-latency tool use but often produce plausible plans that fail during execution due to unavailable tools, missing arguments, or broken dependencies. We introduce EvoFlux, an inference-time search method that repairs tool plans through an evolutionary loop over structured graphs of tool calls and dependencies. EvoFlux executes candidate plans against live tool servers and uses execution errors to guide targeted edits, including replacing tools, correcting parameters, inserting missing steps, and removing redundant ones. On held-out MCP-Bench tasks spanning 28 live servers and 250 tools, EvoFlux improves execution success from roughly 3% to 17-24% across four small planners. SFT and SFT+DPO trained on the same 177 demonstrations and 624 preference pairs match or underperform the zero-shot baseline, while ReAct reaches higher peaks on stronger planners at 4-20x greater planner-token cost per solved task. Execution-guided inference-time repair reliably improves small tool-using agents when supervision is limited. Comments: Code is available at IBM/Evoflux