src/**/*.js が拾うファイル、ツールごとに違う問題
はじめに
src//.ts、!/.test.js、*.{js,ts}。設定ファイルを書いていると、こうした「glob」パターンを毎日のように目にします。
ただ、同じ src/**/*.js でも、ESLint では拾われるファイルが TypeScript の include では拾われない、ということが実際に起きます。この記事では、
glob とは何か(どこまでが標準で、どこからが拡張か)
Node.js でよく使われる glob(node-glob)・minimatch・brace-expansion がそれぞれ何をしているか
.gitignore / ESLint / tsconfig / Vite / Jest で挙動がどう違うか
を、手元で動かせるコードと実行結果つきでまとめます。
動作確認した環境
項目
バージョン
Node.js
v20.19.2
npm
11.21.1
glob
14.0.3
minimatch
11.0.3(makeRe の確認のみ 10.2.6)
brace-expansion
5.0.12
ESLint
10.14.0
TypeScript
7.0.2
Vite
8.3.5
Jest
30.5.2
git
2.47.3
globとは
glob は、* や ? を使ってファイルのパス名をまとめて指定するパターン記法です。Linux の man ページ glob(7) によると、起源は UNIX V6 の /etc/glob というプログラムで、まもなくシェルの組み込み機能になりました。
ls *.ts
POSIX1 で定められている基本の記号は次の3つです。
記号
意味
*
任意の文字列(空文字列を含む)。パス区切りの / にはマッチしない
?
任意の1文字(/ を除く)
[...]
括弧内のいずれか1文字。[!...] で否定、[a-z] で範囲
よく使う ** と {a,b} は、実はこの基本セットに含まれていません。
**(globstar): 0個以上のディレクトリにまたがってマッチする拡張です。bash でも shopt -s globstar を有効にしたときだけ使えます。
{a,b}(ブレース展開): bash のマニュアルでは、ファイル名展開とは別の「文字列を生成する仕組み」として説明されています。
先頭の !(否定パターン): POSIX の記法に出てくる ! は [!abc] のように括弧の中だけです。先頭につけるパターンは定められた記法ではありません。
つまり「glob」と一口に言っても、ツールごとに方言が生まれる余地があります。
Node.js においては、.gitignore に書く node_modules/ や .log、ESLint の設定の files: ["/.ts"]、tsconfig.json の "include": ["src//*"]、テストランナーが *.test.ts を探すときの指定も、すべて glob パターンです。どれも「このパターンに合うファイルはどれか」を判定していますが、それぞれのツールが採用する判定処理には差異があるのです。
この記事ではまず、そもそもglobパターンの判定処理のイメージを掴むために、手で確かめやすい glob・minimatch・brace-expansion というglobパターンを扱うライブラリを代表例として、取り上げます。
代表的な3つのライブラリで仕組みを見る
Node.js でよく使われる3つのパッケージ。次のような依存関係でもあります(npm view で確認)。
glob@14.0.3
└─ minimatch ^11.0.2
└─ brace-expansion ^5.0.12
パッケージ
やること
ディスクを見るか
glob
パターンに合うファイルをディレクトリをたどって探す
見る
minimatch
1つのパス文字列がパターンに合うかを判定する
見ない
brace-expansion
{a,b} や {1..3} を文字列の配列に展開する
見ない
処理の流れはおおむね次のとおりです。
brace-expansion が src/.{js,ts} を src/.js と src/*.ts に展開する
minimatch がそれぞれを / で区切り、区切りごとの照合器を作る
glob がディレクトリをたどりながら、各パスを minimatch で判定する
以下のコードは、空のディレクトリで順に実行すれば再現できます。
サンプルコードはGitHubで公開しています: ikushimayujiro/glob-demo
準備:デモ用のファイル構成
mkdir glob-demo && cd glob-demo
npm init -y
npm i glob@14.0.3 minimatch@11.0.3 minimatch10@npm:minimatch@10.2.6 brace-expansion@5.0.12
minimatch10 は、後で makeRe を見るために v10 系を別名で入れています。
!
minimatch 10.2.6 は、2026年10月31日に公開された GHSA-g3hv-75wg-m249(minimatch <11.0.1 が対象)に該当するため、インストール時に npm audit の警告が出ます。makeRe を確かめるためにわざと古い版を入れているので、このデモ以外では使わないでください。
00-setup.sh
#!/usr/bin/env bash
set -eu
rm -rf fixture && mkdir -p fixture/src/{deep/nested,.config,tests}
for f in src/a.js src/b.ts src/.hidden.js src/.config/c.js src/deep/nested/d.js \
src/app.test.js src/tests/e.js src/.config/f.test.js; do
echo "export const x = 1;" > "fixture/$f"
done
fixture/src/
├── .config/
│ ├── c.js
│ └── f.test.js
├── .hidden.js
├── tests/
│ └── e.js
├── a.js
├── app.test.js
├── b.ts
└── deep/nested/d.js
ドットで始まるファイル(.hidden.js)とディレクトリ(.config/)をわざと混ぜています。
glob:実際にファイルを探す
01-glob.mjs
import { glob } from 'glob'
const cwd = new URL('./fixture/', import.meta.url).pathname
const show = async (pattern, opts = {}) => {
const files = (await glob(pattern, { cwd, posix: true, ...opts })).sort()
console.log(${pattern} ${JSON.stringify(opts)}\n ->, files)
}
await show('src/*.js')
await show('src//*.js')
await show('src//*.js', { dot: true })
await show('src//*.{js,ts}')
await show('src//.js', { ignore: ['/.test.js', 'src/__tests__/'] })
src/*.js {}
-> [ 'src/a.js', 'src/app.test.js' ]
src//*.js {}
-> [ 'src/__tests__/e.js', 'src/a.js', 'src/app.test.js', 'src/deep/nested/d.js' ]
src//*.js {"dot":true}
-> [ 'src/.config/c.js', 'src/.config/f.test.js', 'src/.hidden.js',
'src/tests/e.js', 'src/a.js', 'src/app.test.js', 'src/deep/nested/d.js' ]
src//*.{js,ts}
-> [ 'src/__tests__/e.js', 'src/a.js', 'src/app.test.js', 'src/b.ts', 'src/deep/nested/d.js' ]
src//.js {"ignore":["/.test.js","src/__tests__/"]}
-> [ 'src/a.js', 'src/deep/nested/d.js' ]
(出力は読みやすいよう改行を詰めています)
ポイントは2つあります。
- は / をまたがないので、src/*.js は deep/nested/d.js を拾いません。
既定では、ドットで始まるファイルやディレクトリはマッチしません。dot: true を付けると拾います。README にも「パターン側で明示的に . を書けば、ドットファイルにマッチする」とあります。
minimatch:1つのパスを判定する
minimatch はファイルシステムを見ません。文字列とパターンを受け取って true / false を返すだけです。
02-minimatch.mjs
import { minimatch } from 'minimatch'
const cases = [
['src/a.js', 'src/*.js'],
['src/deep/nested/d.js', 'src/*.js'],
['src/deep/nested/d.js', 'src//*.js'],
['src/.hidden.js', 'src/*.js'],
['src/.hidden.js', 'src/*.js', { dot: true }],
['src/.config/c.js', 'src//*.js'],
['src/b.ts', 'src/*.{js,ts}'],
['src/b.ts', 'src/*.{js,ts}', { nobrace: true }],
['src/a.js', '!src/*.ts'],
['src/a.js', 'src/?.js'],
['src/a.js', 'src/[ab].js'],
]
for (const [path, pattern, opts] of cases) {
console.log(String(minimatch(path, pattern, opts)).padEnd(5), path.padEnd(22), pattern, opts ? JSON.stringify(opts) : '')
}
console.log('\nbraceExpand("src/.{js,ts}") ->', minimatch.braceExpand('src/.{js,ts}'))
true src/a.js src/*.js
false src/deep/nested/d.js src/*.js
true src/deep/nested/d.js src//*.js
false src/.hidden.js src/*.js
true src/.hidden.js src/*.js {"dot":true}
false src/.config/c.js src//*.js
true src/b.ts src/*.{js,ts}
false src/b.ts src/*.{js,ts} {"nobrace":true}
true src/a.js !src/*.ts
true src/a.js src/?.js
true src/a.js src/[ab].js
braceExpand("src/.{js,ts}") -> [ 'src/.js', 'src/*.ts' ]
** も既定ではドットで始まるディレクトリ(.config/)には入りません。
nobrace: true にするとブレース展開が行われず、{js,ts} は文字どおりの文字列として扱われます。
minimatch 単体では、先頭の ! は「このパターンに合わなければ true」という否定になります。src/a.js は src/*.ts に合わないので true です。
minimatch.braceExpand() の結果からわかるように、ブレースはマッチングの前に展開されて、複数のパターンになります。
makeRe:正規表現になる(v10まで)
minimatch v10 系までは、パターンを正規表現に変換して照合していました。makeRe() で中身を見られます。
03-makere.cjs
const mm10 = require('minimatch10') // minimatch@10.2.6
const mm11 = require('minimatch') // minimatch@11.0.3
for (const p of ['.js', 'src/*/.js', '.{js,ts}']) {
console.log(p.padEnd(14), '->', mm10.makeRe(p))
}
console.log('\nminimatch@11 に makeRe はある? ->', typeof mm11.makeRe, '/', typeof mm11.minimatch.makeRe)
.js -> /^(?!.)[^/]?.js$/
src/**/.js -> /^src(?:\/|\/(?:(?!(?:\/|^).).)?\/)(?!.)[^/]*?.js$/
.{js,ts} -> /^(?:(?!.)[^/]?.js|(?!.)[^/]*?.ts)$/
minimatch@11 に makeRe はある? -> undefined / undefined
正規表現を読むと、ここまでの挙動がそのまま書かれています。
[^/]?: は / 以外の文字列
(?!.):先頭がドットならマッチしない
*.{js,ts}:ブレース展開した2つのパターンを | でつないだもの
一方、2026年10月に出た minimatch v11 では、照合に正規表現を使わなくなりました。changelog によると、パスの区切りごとに @isaacs/nonrecursive-glob-match の Matcher で照合する方式に変わり、makeRe() は削除されています。README では、正規表現はバックトラックによる ReDoS(正規表現によるサービス拒否)を起こしうるため、という理由が説明されています。
ESLint 10.14.0 は minimatch ^10.2.6 に依存しているので、プロジェクトによっては v10 系と v11 系が両方入っていることもあります。
brace-expansion:文字列を展開する
04-brace.mjs
import { expand } from 'brace-expansion'
console.log(expand('file.{js,ts}'))
console.log(expand('img{1..3}.png'))
console.log(expand('{a,b}{x,y}'))
console.log(expand('src/{components,hooks}/**/*.{ts,tsx}'))
[ 'file.js', 'file.ts' ]
[ 'img1.png', 'img2.png', 'img3.png' ]
[ 'ax', 'ay', 'bx', 'by' ]
[
'src/components//*.ts',
'src/components//*.tsx',
'src/hooks//*.ts',
'src/hooks//*.tsx'
]
{a,b}{x,y} が4件になるように、ブレースを並べると展開数は掛け算で増えます。小さい規模で確かめてみます。
for (const n of [1, 2, 4, 8, 12, 16]) {
const r = expand('{a,b}'.repeat(n))
const totalChars = r.reduce((s, x) => s + x.length, 0)
console.log(n=${String(n).padStart(2)} 入力${String(n * 5).padStart(3)}文字 -> ${String(r.length).padStart(6)}件 (理論値 2^n=${2 ** n}) 合計${totalChars}文字)
}
n= 1 入力 5文字 -> 2件 (理論値 2^n=2) 合計2文字
n= 2 入力 10文字 -> 4件 (理論値 2^n=4) 合計8文字
n= 4 入力 20文字 -> 16件 (理論値 2^n=16) 合計64文字
n= 8 入力 40文字 -> 256件 (理論値 2^n=256) 合計2048文字
n=12 入力 60文字 -> 4096件 (理論値 2^n=4096) 合計49152文字
n=16 入力 80文字 -> 65536件 (理論値 2^n=65536) 合計1048576文字
入力80文字で、出力は65,536件・合計約100万文字になりました。件数は 2^n、合計文字数は n × 2^n で増えます。
!
ここでは n=16 までにとどめています。これより大きい値を試すとメモリを大量に使うので、手元でも同じくらいの規模にとどめてください。
brace-expansion には、この増え方を抑える上限オプションがあります(README)。
max:展開する件数の上限(既定 100,000)
maxLength:展開結果の合計文字数の上限(既定 4,000,000)。5.0.8 で追加
console.log(expand('{1..100}', { max: 10 }).length) // 10
const capped = expand('{a,b}'.repeat(12), { maxLength: 1000 })
console.log(capped.length, capped.reduce((s, x) => s + x.length, 0)) // 83 996
上限に達すると、エラーにはならず結果が途中で打ち切られます。
いざ、ツールごとの違い
それではいよいよ、本題。
同じ fixture/ に対して、各ツールが実際にどのファイルを拾うかを試しました(デモの 05-tools.sh)。
ツール
設定項目
使用ライブラリ
ドットファイル
**
ブレース {a,b}
先頭 !
git
.gitignore
独自(fnmatch(3) 準拠の記述)
- でマッチする
対応
非対応(文字どおり扱う)
前のパターンで除外したものを再び含める。ただし親ディレクトリが除外されていると戻せない
ESLint(flat config)
files / ignores
minimatch
マッチする(ignores はドキュメントに明記、files は実測)
対応
対応
ignores 内で除外の取り消しに使える
TypeScript
include / exclude
独自(* ? **/ のみ記載)
ワイルドカードではマッチしない。src/.config/* のように明示すれば拾う
**/ 対応
非対応(該当なしでエラー)
記載なし
Vite
import.meta.glob()
tinyglobby
既定ではマッチしない。exhaustive:true で指定
対応
対応
除外パターンとして使える
Jest
testMatch
micromatch
マッチする
対応
対応
除外。配列の順序が効く(否定は後ろに置く)
実測の出力(抜粋)は次のとおりです。
=== git (.gitignore)
-- .gitignore: src/*.js
src/a.js ignored
src/.hidden.js ignored
src/deep/nested/d.js not-ignored
-- .gitignore: src/*.{js,ts}
src/a.js not-ignored
src/b.ts not-ignored
=== TypeScript (tsconfig include) Version 7.0.2
-- include: ["src//*"]
src/__tests__/e.js
src/a.js
src/app.test.js
src/b.ts
src/deep/nested/d.js
-- include: ["src//*.{js,ts}"]
error TS18003: No inputs were found in config file 'tsconfig.json'. ...
-- include: ["src/.config/*"]
src/.config/c.js
src/.config/f.test.js
=== ESLint (flat config files: ['src/**/*.{js,ts}']) v10.14.0
src/.config/c.js
src/.config/f.test.js
src/.hidden.js
src/tests/e.js
src/a.js
src/app.test.js
src/b.ts
src/deep/nested/d.js
=== Jest (testMatch) 30.5.2
-- default testMatch
src/.config/f.test.js
src/tests/e.js
src/app.test.js
=== Vite (import.meta.glob) vite/8.3.5
plain ./src/tests/e.js, ./src/a.js, ./src/app.test.js, ./src/deep/nested/d.js
brace ./src/tests/e.js, ./src/a.js, ./src/app.test.js, ./src/b.ts, ./src/deep/nested/d.js
negated ./src/tests/e.js, ./src/a.js, ./src/deep/nested/d.js
exhaustive ./src/.config/c.js, ./src/.config/f.test.js, ./src/.hidden.js, ...
読み取れることをまとめます。
ドットファイル: ESLint と Jest は .config/ の中まで拾いますが、TypeScript と Vite は拾いません。.storybook/ のような設定ディレクトリが lint やテストの対象に入る・入らないの差は、ここから来ます。
ブレース展開: .gitignore と tsconfig の include では使えません。tsconfig で "src/**/*.{ts,tsx}" と書くと、ファイルが1つも見つからずエラーになりました。
否定: .gitignore の ! は「再び含める」、Jest の testMatch では「除外」で、しかも順序が効きます。同じ記号でも意味が違います。
まとめ:なぜツールごとに方言が生まれるのか
ここまで見てきた違いは、大きく3つの原因に分けられます。
1. パターン判別のライブラリが違う
git と TypeScript は独自の実装、ESLint は minimatch、Vite は tinyglobby、Jest は micromatch を使っています。
2. 同じライブラリでも、渡すオプションが違う
ESLint は minimatch を使っていますが、ドットファイルにもマッチします。minimatch は既定ではドットファイルを拾いませんが、ESLint が dot: true を付けて呼び出しているからです。
3. 設定ファイル側の文法が違う
先頭の ! の意味は、ライブラリだけでなく、各ツールの設定ファイルの文法にもよります。.gitignore では「前のパターンで除外したものを再び含める」、Jest の testMatch では「除外」で、しかも配列の順序が効きます。
同じ glob パターンでも、どのライブラリを、どのオプションで、どんな文法の中で使っているかによって結果が変わるということですね。迷ったら、各ツールのファイル一覧を出すコマンド(git check-ignore、tsc --listFilesOnly、eslint --inspect-config、jest --listTests)で実際の結果を確認するのが確実です。
おまけ:glob と minimatch を書いた人
最後に少し寄り道です。この記事で何度も出てきた glob と minimatch は、npm registry の author 欄を見ると同じ人の名前になっています。
npm view glob author # Isaac Z. Schlueter <i@izs.me> (blog.izs.me)
npm view minimatch author # Isaac Z. Schlueter <i@izs.me> (blog.izs.me)
Isaac Z. Schlueter さん(GitHub では isaacs)は、npm を作った人です。本人の GitHub プロフィールには「npm inventor, founder npm, Inc. Former Node BDFL.」と書かれています。
author が Isaac Z. Schlueter さんになっているパッケージには、ほかにも次のようなものがあります。
パッケージ
用途
週間ダウンロード数(2026-10-23〜10-29)
minimatch
glob パターンの照合
951,240,180
lru-cache
LRU キャッシュ
789,165,914
glob
ファイル検索
536,547,167
rimraf
rm -rf 相当の削除
214,603,009
tar
tar アーカイブの読み書き
119,327,291
(npm の downloads API で取得)
package.json の ^5.0.8 のようなバージョン範囲を解釈する semver も、LICENSE に「Copyright (c) Isaac Z. Schlueter and Contributors」とあります(現在の author 欄は GitHub Inc.、リポジトリは npm organization の管理です)。
普段使っている npm を作った人が、node_modules によく入っている小さな部品もたくさん書いている、ということになります。npm CLI 自身が、glob・minimatch・semver・tar に依存しているのを見ると、開発の中で自身が必要になったものを、片っ端から生み出していった結果なのかもしれません。
参考資料
glob(7) - Linux manual page: man7.org
Bash Reference Manual - Brace Expansion: gnu.org
Bash Reference Manual - The Shopt Builtin(globstar / dotglob): gnu.org
isaacs/node-glob README: isaacs/node-glob
isaacs/minimatch README / changelog: isaacs/minimatch , github.com/isaacs/minimatch/blob/main/changelog.md
juliangruber/brace-expansion README: juliangruber/brace-expansion
gitignore Documentation: git-scm.com
ESLint - Configuration Files: eslint.org
ESLint v10.14.0 リリースノート: github.com/eslint/eslint/releases/tag/v10.14.0
TSConfig Reference - include: typescriptlang.org
Vite - Glob Import: vite.dev
tinyglobby documentation: superchupu.dev
Jest - testMatch: jestjs.io
脚注
POSIX(Portable Operating System Interface)は、UNIX 系 OS の間でアプリケーションを移植しやすくするための標準規格です。最新版の POSIX.1-2024 は IEEE Std 1003.1-2024 と The Open Group の Base Specifications Issue 8 を兼ねていて、OS のインターフェースに加えてシェルや共通コマンドも定めています。glob の記号は、シェルの章の「2.14 Pattern Matching Notation」で規定されています。出典:The Open Group Base Specifications Issue 8、同 2.14 Pattern Matching Notation、IEEE 1003.1-2024 ↩︎
Building an In-Memory Semantic Cache for LLM Endpoints with Node.js and pgvector
If you run an LLM-backed endpoint (a support bot, a docs assistant, a "chat with your data" feature), you've probably noticed something in your logs: people ask the same questions over and over, just worded differently.
- "How do I reset my password?"
- "I forgot my password, what do I do?"
- "password reset steps"
A traditional cache keyed on the exact prompt string treats these as three unrelated requests, so you pay for three LLM calls, three times the latency, and three times the tokens.
A semantic cache fixes this. Instead of matching strings, it matches meaning: embed the incoming prompt, find the nearest previously answered prompt, and if it's close enough, return the stored answer without ever calling the LLM.
In this post we'll build one in Node.js and TypeScript with a two-tier design:
- L1: in-process memory. Microsecond lookups for exact repeats.
- L2: Postgres + pgvector. Shared across instances, durable, with similarity search via an HNSW index.
We'll also cover the part most tutorials skip: when a cache hit is actually a wrong answer, and how to design around it.
---
How a semantic cache works
Request ─▶ normalize ─▶ L1 exact lookup ──hit──▶ return
│ miss
▼
embed(prompt) ◀── ~50-300 ms, small cost
│
▼
L2 nearest neighbor (pgvector)
│
similarity ≥ threshold? ──yes──▶ return cached answer (+ promote to L1)
│ no
▼
call the LLM
│
▼
store in L1 + L2 (embedding + answer) ─▶ return
The economics work because embedding a prompt is much cheaper and faster than generating a completion. A miss costs you one extra embedding call plus a DB query. A hit saves a full generation. If your hit rate is decent, you come out well ahead on both cost and p50 latency.
Real numbers depend entirely on your traffic, so measure them (we'll add metrics later) rather than trusting anyone's blog post, including this one.
---
Step 1: Set up pgvector
Enable the extension and create the cache table:
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE llm_cache (
id BIGSERIAL PRIMARY KEY,
scope TEXT NOT NULL, -- tenant + model + prompt version (see below)
prompt_hash TEXT NOT NULL, -- hash of normalized prompt, for exact match
prompt TEXT NOT NULL, -- original text, useful for debugging and audits
embedding vector(1536) NOT NULL,
response JSONB NOT NULL,
hit_count INTEGER NOT NULL DEFAULT 0,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
expires_at TIMESTAMPTZ NOT NULL
);
-- Exact-match lookups
CREATE UNIQUE INDEX llm_cache_scope_hash_idx ON llm_cache (scope, prompt_hash);
-- Approximate nearest neighbor search using cosine distance
CREATE INDEX llm_cache_embedding_idx
ON llm_cache USING hnsw (embedding vector_cosine_ops);
-- For cleanup jobs
CREATE INDEX llm_cache_expires_idx ON llm_cache (expires_at);
Notes:
- vector(1536) must match your embedding model's output dimension. Check your model's docs, and keep it consistent. Mixing models in one column breaks similarity.
- HNSW gives fast approximate search and doesn't require training data first (unlike IVFFlat). It needs pgvector 0.5.0 or newer.
- <=> is pgvector's cosine distance operator. Cosine similarity is 1 - distance.
The scope column is the most important design decision
A cached answer is only valid for the context it was generated in. Put everything that changes the correct answer into scope:
function buildScope(opts: {
tenantId: string;
model: string;
promptVersion: string; // bump when you change your system prompt
toolsetVersion?: string;
}) {
return [opts.tenantId, opts.model, opts.promptVersion, opts.toolsetVersion ?? '-'].join('|');
}
Without this, tenant A's answer can be served to tenant B (a data leak), or a stale answer generated by an old system prompt keeps being served after you "fixed" the bot. Bumping promptVersion is your cheap, instant "invalidate everything" button.
---
Step 2: The cache class
Install dependencies:
npm i pg pgvector lru-cache
Here's the core implementation:
// semantic-cache.ts
import crypto from 'node:crypto';
import { Pool } from 'pg';
import pgvector from 'pgvector/pg';
import { LRUCache } from 'lru-cache';
export interface CachedResponse {
text: string;
model: string;
usage?: { inputTokens: number; outputTokens: number };
}
export interface CacheLookup {
hit: boolean;
tier?: 'l1' | 'l2';
similarity?: number;
response?: CachedResponse;
}
export class SemanticCache {
// L1: exact-match, in-process. Key = scope + prompt hash.
private l1 = new LRUCache<string, CachedResponse>({
max: 5_000,
ttl: 10 * 60 * 1000, // 10 minutes
});
// Single-flight: if two identical requests arrive at once, only one calls the LLM
private inflight = new Map<string, Promise<CachedResponse>>();
constructor(
private pool: Pool,
private embed: (text: string) => Promise<number[]>,
private opts = {
similarityThreshold: 0.92,
ttlSeconds: 24 * 60 * 60,
},
) {}
static normalize(prompt: string): string {
return prompt.trim().toLowerCase().replace(/\s+/g, ' ');
}
private hash(text: string) {
return crypto.createHash('sha256').update(text).digest('hex');
}
async lookup(scope: string, prompt: string): Promise<CacheLookup & { embedding?: number[] }> {
const normalized = SemanticCache.normalize(prompt);
const promptHash = this.hash(normalized);
const l1Key = `${scope}:${promptHash}`;
// --- L1: exact match in memory ---
const l1Hit = this.l1.get(l1Key);
if (l1Hit) return { hit: true, tier: 'l1', similarity: 1, response: l1Hit };
// --- L2a: exact match in Postgres (no embedding needed) ---
const exact = await this.pool.query(
`SELECT response FROM llm_cache
WHERE scope = $1 AND prompt_hash = $2 AND expires_at > now()`,
[scope, promptHash],
);
if (exact.rows[0]) {
this.l1.set(l1Key, exact.rows[0].response);
return { hit: true, tier: 'l2', similarity: 1, response: exact.rows[0].response };
}
// --- L2b: semantic match via pgvector ---
const embedding = await this.embed(normalized);
const { rows } = await this.pool.query(
`SELECT id, response, 1 - (embedding <=> $2) AS similarity
FROM llm_cache
WHERE scope = $1 AND expires_at > now()
ORDER BY embedding <=> $2
LIMIT 1`,
[scope, pgvector.toSql(embedding)],
);
const best = rows[0];
if (best && best.similarity >= this.opts.similarityThreshold) {
// Fire-and-forget stats update; don't block the response
this.pool
.query('UPDATE llm_cache SET hit_count = hit_count + 1 WHERE id = $1', [best.id])
.catch(() => {});
this.l1.set(l1Key, best.response); // promote so the exact phrasing is fast next time
return { hit: true, tier: 'l2', similarity: best.similarity, response: best.response, embedding };
}
return { hit: false, embedding };
}
async store(
scope: string,
prompt: string,
response: CachedResponse,
embedding?: number[],
): Promise<void> {
const normalized = SemanticCache.normalize(prompt);
const promptHash = this.hash(normalized);
this.l1.set(`${scope}:${promptHash}`, response);
const vec = embedding ?? (await this.embed(normalized));
await this.pool.query(
`INSERT INTO llm_cache (scope, prompt_hash, prompt, embedding, response, expires_at)
VALUES ($1, $2, $3, $4, $5, now() + ($6 || ' seconds')::interval)
ON CONFLICT (scope, prompt_hash)
DO UPDATE SET response = EXCLUDED.response,
embedding = EXCLUDED.embedding,
expires_at = EXCLUDED.expires_at`,
[scope, promptHash, prompt, pgvector.toSql(vec), response, String(this.opts.ttlSeconds)],
);
}
/** Wrap an LLM call with cache lookup, single-flight, and write-back. */
async getOrCompute(
scope: string,
prompt: string,
compute: () => Promise<CachedResponse>,
): Promise<CacheLookup & { response: CachedResponse }> {
const found = await this.lookup(scope, prompt);
if (found.hit) return found as CacheLookup & { response: CachedResponse };
const key = `${scope}:${this.hash(SemanticCache.normalize(prompt))}`;
let pending = this.inflight.get(key);
if (!pending) {
pending = compute()
.then(async (res) => {
// Don't let a cache write failure fail the user's request
await this.store(scope, prompt, res, found.embedding).catch((e) =>
console.error('cache store failed', e),
);
return res;
})
.finally(() => this.inflight.delete(key));
this.inflight.set(key, pending);
}
return { hit: false, response: await pending };
}
}
A few things worth calling out:
- Exact match before embedding. Many repeats are literally identical. Skipping the embedding call for those saves both time and money.
- Single-flight. If 50 users ask the same new question at once (say, right after an outage), you want one LLM call, not 50. The inflight map handles that within a process.
- The cache must never break the request. Failures on store are logged and swallowed. You should also wrap lookup so a database outage degrades to "just call the LLM" (see the Express example below).
- Parameterized queries throughout. pgvector.toSql() serializes the array safely for the vector type.
---
Step 3: Wire it into an endpoint
// server.ts
import express from 'express';
import { Pool } from 'pg';
import pgvector from 'pgvector/pg';
const pool = new Pool({ connectionString: process.env.DATABASE_URL });
pool.on('connect', (client) => pgvector.registerTypes(client));
const cache = new SemanticCache(pool, embedText);
const app = express();
app.use(express.json());
app.post('/api/ask', async (req, res) => {
const { question } = req.body;
const tenantId = req.user.tenantId; // from your auth middleware
const scope = buildScope({
tenantId,
model: process.env.LLM_MODEL!,
promptVersion: 'support-bot-v7',
});
const started = performance.now();
try {
const result = await cache.getOrCompute(scope, question, () =>
callLlm({ system: SYSTEM_PROMPT, user: question }),
);
metrics.timing('ask.latency_ms', performance.now() - started, {
cache: result.hit ? result.tier! : 'miss',
});
metrics.increment('ask.cache', { result: result.hit ? 'hit' : 'miss' });
res.json({ answer: result.response.text, cached: result.hit });
} catch (err) {
// If the cache layer itself blows up, fall back to the LLM directly
console.error('cache path failed, calling LLM directly', err);
const direct = await callLlm({ system: SYSTEM_PROMPT, user: question });
res.json({ answer: direct.text, cached: false });
}
});
One caveat on that catch: it also catches errors from the LLM call itself, so you may double-call the LLM if the first call failed. In production, separate "cache infrastructure errors" from "compute errors" (for example by having lookup return a miss on DB failure internally).
---
The hard part: choosing a similarity threshold
The threshold is the knob that trades hit rate against correctness, and there's no universal right value. It depends on your embedding model, your domain, and how costly a wrong answer is.
- Too low: you serve wrong answers confidently.
- Too high: you rarely hit, and the cache is expensive overhead.
Embedding similarity measures topical closeness, not intent equivalence. These pairs are all highly similar to an embedding model but need different answers:
| Prompt A | Prompt B | Same answer OK? |
|---|---|---|
| "How do I cancel my subscription?" | "How do I not cancel my subscription?" | ❌ |
| "What's the refund policy for annual plans?" | "What's the refund policy for monthly plans?" | ❌ |
| "Is feature X available in Python?" | "Is feature X available in Go?" | ❌ |
| "How do I reset my password?" | "I forgot my password" | ✅ |
Calibrate with your own data
Don't guess. Build a small labeled set from real logs: pairs of prompts marked "same intent" or "different intent". Then sweep thresholds:
// calibrate.ts
interface Pair { a: string; b: string; sameIntent: boolean }
function cosine(a: number[], b: number[]) {
let dot = 0, na = 0, nb = 0;
for (let i = 0; i < a.length; i++) {
dot += a[i] * b[i]; na += a[i] ** 2; nb += b[i] ** 2;
}
return dot / (Math.sqrt(na) * Math.sqrt(nb));
}
async function sweep(pairs: Pair[]) {
const scored = [];
for (const p of pairs) {
const [ea, eb] = await Promise.all([embedText(p.a), embedText(p.b)]);
scored.push({ sim: cosine(ea, eb), sameIntent: p.sameIntent });
}
for (let t = 0.80; t <= 0.99; t += 0.01) {
const hits = scored.filter((s) => s.sim >= t);
const falsePositives = hits.filter((s) => !s.sameIntent).length;
const truePositives = hits.length - falsePositives;
console.log(
`threshold=${t.toFixed(2)} hits=${hits.length} ` +
`correct=${truePositives} WRONG=${falsePositives}`,
);
}
}
Choose the lowest threshold where WRONG is acceptably close to zero for your risk tolerance. For a customer-facing bot that quotes prices or policies, be very conservative. For a casual "explain this concept" endpoint, you can be looser.
Defense in depth beyond the threshold
1. Segment by scope. Different products, plans, or languages shouldn't share cache entries.
2. Keep entities out of the fuzzy match. If prompts contain order IDs, names, or numbers, either extract them into the scope/key or skip semantic matching when they're present, because the nearest neighbor of "status of ORD-123456" is "status of ORD-654321".
3. Per-category thresholds. Stricter for transactional or factual queries, looser for general explanations.
4. Log near-misses. Record hits between 0.88 and 0.95 along with the matched prompt for periodic human review. It's the best source of calibration data you'll get.
---
What you should not cache
A cache hit means "the answer from last time is still correct". That's false for many requests:
- Personalized answers ("what's my balance?"). Different users, different truths.
- Time-sensitive answers ("what's the status of my order?", "what's today's date?").
- Responses that used tools with live data. Caching the final text freezes that data in place.
- Non-deterministic or creative requests where the user expects variety ("give me another idea").
- Multi-turn context. If the answer depends on earlier messages, the last user message alone is not the cache key. Either only cache stateless first turns, or embed a summary of the conversation context.
- Anything with sensitive data, unless you've thought through retention, access control, and your compliance obligations. Cached prompts and answers are stored data.
Implement this as an explicit policy function rather than scattered ifs:
function isCacheable(req: AskRequest): boolean {
if (req.conversation.length > 1) return false; // multi-turn
if (req.usedTools) return false; // live data involved
if (req.user.requiresPersonalization) return false;
if (/\b(today|now|current|latest|my)\b/i.test(req.question)) return false; // crude, tune it
return true;
}
---
Performance and operations
HNSW tuning
At query time, hnsw.ef_search controls the recall/speed tradeoff (the default is 40; higher means better recall but slower):
SET hnsw.ef_search = 100;
For a cache you can usually tolerate slightly imperfect recall, since a missed neighbor just means one extra LLM call.
Filtering with HNSW
Our query filters by scope. With HNSW, filtering is applied after the index scan, so a highly selective filter can return fewer rows than you asked for (or none, even though matches exist). Options:
- Use pgvector's iterative index scans (available in newer versions, via hnsw.iterative_scan)
- Use partial indexes or table partitioning per large tenant
- Keep scopes coarse enough that most scans find candidates
Check the pgvector README for the options available in your installed version.
Expiry and cleanup
expires_at filtering in queries keeps stale entries from being served, but they still occupy space and index memory. Run a periodic cleanup:
DELETE FROM llm_cache WHERE expires_at < now() - interval '1 day';
Run it in batches, or use partitioning by time if the table is large. Consider also evicting entries with zero hits after N days. They're not earning their keep.
Embedding latency is on your critical path
Every cache miss pays for an embedding call and the LLM call. Mitigations:
- Skip semantic lookup for very short or very long prompts (tune the bounds)
- Set a tight timeout on the embedding call. If it's slow, treat it as a miss and go straight to the LLM
- Reuse the embedding computed during lookup when storing (we do this via the returned embedding)
Streaming responses
If you stream tokens to users, you can't cache until the stream completes. Buffer the full text as it streams, then store it afterwards. On a hit, you can either return the whole answer instantly or simulate streaming by chunking the cached text, which keeps your frontend code identical for both paths.
A note on "in-memory" scaling
The L1 cache is per-process. With many instances, each warms up independently, and invalidation (via promptVersion in the scope) works because new scopes simply never match old keys. If you need cross-instance L1 coherence, that's a sign to add Redis as a shared tier, but start without it. L1 + pgvector covers most workloads, and every extra tier is another thing to keep consistent.
---
Metrics that tell you if it's working
Instrument from day one:
- Hit rate by tier (l1, l2, miss) and by scope or category
- Latency by outcome (hit vs miss). Compare p50 and p95.
- Embedding latency and error rate
- Similarity distribution of hits. If most hits cluster right at your threshold, you're likely over-matching.
- Estimated tokens saved, from the usage stored on each cached response
- User feedback signals (thumbs down, re-asks) segmented by cached: true/false. If cached answers get disproportionately downvoted, raise the threshold.
That last one is the real test. A cache with a great hit rate and worse user satisfaction is a regression.
---
Putting it together
┌───────────────────────────────┐
Request ──▶ │ isCacheable? ── no ─────────┼──▶ LLM
└──────────────┬────────────────┘
│ yes
┌──────────▼──────────┐
│ L1: in-process LRU │── hit ──▶ response (µs)
└──────────┬──────────┘
│ miss
┌──────────▼──────────┐
│ L2: exact (Postgres)│── hit ──▶ response (ms)
└──────────┬──────────┘
│ miss
┌──────────▼──────────┐
│ embed + pgvector kNN│── ≥ threshold ──▶ response
└──────────┬──────────┘
│ miss
┌──────────▼──────────┐
│ single-flight → LLM │──▶ store L1 + L2 ──▶ response
└─────────────────────┘
Key takeaways
1. A semantic cache matches meaning, not strings. Embed the prompt, find the nearest cached prompt, and reuse the answer above a similarity threshold.
2. Two tiers are enough. An in-process LRU for exact repeats and Postgres with pgvector (HNSW) for shared, durable semantic matches.
3. Put everything that affects correctness into the scope: tenant, model, prompt version. This prevents data leaks and gives you instant invalidation.
4. Calibrate the threshold with labeled data from your own traffic. Similar embeddings don't guarantee equivalent intent.
5. Be deliberate about what's cacheable. Personalized, time-sensitive, tool-backed, and multi-turn requests usually aren't.
6. Make the cache fail-open. It should only ever make your endpoint faster, never less available.
7. Measure user-facing quality, not just hit rate.
Have you built a semantic cache, or tried one and abandoned it? I'd love to hear what threshold and embedding model worked for you in the comments.
---
Further reading: the pgvector README (HNSW options, iterative scans, distance operators), pgvector-node, and lru-cache.
Mentions
src/**/*.js が拾うファイル、ツールごとに違う問題 はじめに src//.ts、!/.test.js、*.{js,ts}。設定ファイルを書いていると、こうした「glob」パターンを毎日のように目にします。 ただ、同じ src/**/*.js でも、ESLint では拾われるファイルが TypeScript の include では拾われない、ということが実際に起きます。この記事では、 glob とは何か(どこまでが標準で、どこからが拡張か) Node.js でよく使われる glob(node-glob)・minimatch・brace-expansion がそれぞれ何をしているか .gitignore / ESLint / tsconfig / Vite / Jest で挙動がどう違うか を、手元で動かせるコードと実行結果つきでまとめます。 動作確認した環境 項目 バージョン Node.js v20.19.2 npm 11.21.1 glob 14.0.3 minimatch 11.0.3(makeRe の確認のみ 10.2.6) brace-expansion 5.0.12 ESLint 10.14.0 TypeScript 7.0.2 Vite 8.3.5 Jest 30.5.2 git 2.47.3 globとは glob は、* や ? を使ってファイルのパス名をまとめて指定するパターン記法です。Linux の man ページ glob(7) によると、起源は UNIX V6 の /etc/glob というプログラムで、まもなくシェルの組み込み機能になりました。 ls *.ts POSIX1 で定められている基本の記号は次の3つです。 記号 意味 * 任意の文字列(空文字列を含む)。パス区切りの / にはマッチしない ? 任意の1文字(/ を除く) [...] 括弧内のいずれか1文字。[!...] で否定、[a-z] で範囲 よく使う ** と {a,b} は、実はこの基本セットに含まれていません。 **(globstar): 0個以上のディレクトリにまたがってマッチする拡張です。bash でも shopt -s globstar を有効にしたときだけ使えます。 {a,b}(ブレース展開): bash のマニュアルでは、ファイル名展開とは別の「文字列を生成する仕組み」として説明されています。 先頭の !(否定パターン): POSIX の記法に出てくる ! は [!abc] のように括弧の中だけです。先頭につけるパターンは定められた記法ではありません。 つまり「glob」と一口に言っても、ツールごとに方言が生まれる余地があります。 Node.js においては、.gitignore に書く node_modules/ や .log、ESLint の設定の files: ["/.ts"]、tsconfig.json の "include": ["src//*"]、テストランナーが *.test.ts を探すときの指定も、すべて glob パターンです。どれも「このパターンに合うファイルはどれか」を判定していますが、それぞれのツールが採用する判定処理には差異があるのです。 この記事ではまず、そもそもglobパターンの判定処理のイメージを掴むために、手で確かめやすい glob・minimatch・brace-expansion というglobパターンを扱うライブラリを代表例として、取り上げます。 代表的な3つのライブラリで仕組みを見る Node.js でよく使われる3つのパッケージ。次のような依存関係でもあります(npm view で確認)。 glob@14.0.3 └─ minimatch ^11.0.2 └─ brace-expansion ^5.0.12 パッケージ やること ディスクを見るか glob パターンに合うファイルをディレクトリをたどって探す 見る minimatch 1つのパス文字列がパターンに合うかを判定する 見ない brace-expansion {a,b} や {1..3} を文字列の配列に展開する 見ない 処理の流れはおおむね次のとおりです。 brace-expansion が src/.{js,ts} を src/.js と src/*.ts に展開する minimatch がそれぞれを / で区切り、区切りごとの照合器を作る glob がディレクトリをたどりながら、各パスを minimatch で判定する 以下のコードは、空のディレクトリで順に実行すれば再現できます。 サンプルコードはGitHubで公開しています: ikushimayujiro/glob-demo 準備:デモ用のファイル構成 mkdir glob-demo && cd glob-demo npm init -y npm i glob@14.0.3 minimatch@11.0.3 minimatch10@npm:minimatch@10.2.6 brace-expansion@5.0.12 minimatch10 は、後で makeRe を見るために v10 系を別名で入れています。 ! minimatch 10.2.6 は、2026年10月31日に公開された GHSA-g3hv-75wg-m249(minimatch <11.0.1 が対象)に該当するため、インストール時に npm audit の警告が出ます。makeRe を確かめるためにわざと古い版を入れているので、このデモ以外では使わないでください。 00-setup.sh #!/usr/bin/env bash set -eu rm -rf fixture && mkdir -p fixture/src/{deep/nested,.config,tests} for f in src/a.js src/b.ts src/.hidden.js src/.config/c.js src/deep/nested/d.js \ src/app.test.js src/tests/e.js src/.config/f.test.js; do echo "export const x = 1;" > "fixture/$f" done fixture/src/ ├── .config/ │ ├── c.js │ └── f.test.js ├── .hidden.js ├── tests/ │ └── e.js ├── a.js ├── app.test.js ├── b.ts └── deep/nested/d.js ドットで始まるファイル(.hidden.js)とディレクトリ(.config/)をわざと混ぜています。 glob:実際にファイルを探す 01-glob.mjs import { glob } from 'glob' const cwd = new URL('./fixture/', import.meta.url).pathname const show = async (pattern, opts = {}) => { const files = (await glob(pattern, { cwd, posix: true, ...opts })).sort() console.log(${pattern} ${JSON.stringify(opts)}\n ->, files) } await show('src/*.js') await show('src//*.js') await show('src//*.js', { dot: true }) await show('src//*.{js,ts}') await show('src//.js', { ignore: ['/.test.js', 'src/__tests__/'] }) src/*.js {} -> [ 'src/a.js', 'src/app.test.js' ] src//*.js {} -> [ 'src/__tests__/e.js', 'src/a.js', 'src/app.test.js', 'src/deep/nested/d.js' ] src//*.js {"dot":true} -> [ 'src/.config/c.js', 'src/.config/f.test.js', 'src/.hidden.js', 'src/tests/e.js', 'src/a.js', 'src/app.test.js', 'src/deep/nested/d.js' ] src//*.{js,ts} -> [ 'src/__tests__/e.js', 'src/a.js', 'src/app.test.js', 'src/b.ts', 'src/deep/nested/d.js' ] src//.js {"ignore":["/.test.js","src/__tests__/"]} -> [ 'src/a.js', 'src/deep/nested/d.js' ] (出力は読みやすいよう改行を詰めています) ポイントは2つあります。 - は / をまたがないので、src/*.js は deep/nested/d.js を拾いません。 既定では、ドットで始まるファイルやディレクトリはマッチしません。dot: true を付けると拾います。README にも「パターン側で明示的に . を書けば、ドットファイルにマッチする」とあります。 minimatch:1つのパスを判定する minimatch はファイルシステムを見ません。文字列とパターンを受け取って true / false を返すだけです。 02-minimatch.mjs import { minimatch } from 'minimatch' const cases = [ ['src/a.js', 'src/*.js'], ['src/deep/nested/d.js', 'src/*.js'], ['src/deep/nested/d.js', 'src//*.js'], ['src/.hidden.js', 'src/*.js'], ['src/.hidden.js', 'src/*.js', { dot: true }], ['src/.config/c.js', 'src//*.js'], ['src/b.ts', 'src/*.{js,ts}'], ['src/b.ts', 'src/*.{js,ts}', { nobrace: true }], ['src/a.js', '!src/*.ts'], ['src/a.js', 'src/?.js'], ['src/a.js', 'src/[ab].js'], ] for (const [path, pattern, opts] of cases) { console.log(String(minimatch(path, pattern, opts)).padEnd(5), path.padEnd(22), pattern, opts ? JSON.stringify(opts) : '') } console.log('\nbraceExpand("src/.{js,ts}") ->', minimatch.braceExpand('src/.{js,ts}')) true src/a.js src/*.js false src/deep/nested/d.js src/*.js true src/deep/nested/d.js src//*.js false src/.hidden.js src/*.js true src/.hidden.js src/*.js {"dot":true} false src/.config/c.js src//*.js true src/b.ts src/*.{js,ts} false src/b.ts src/*.{js,ts} {"nobrace":true} true src/a.js !src/*.ts true src/a.js src/?.js true src/a.js src/[ab].js braceExpand("src/.{js,ts}") -> [ 'src/.js', 'src/*.ts' ] ** も既定ではドットで始まるディレクトリ(.config/)には入りません。 nobrace: true にするとブレース展開が行われず、{js,ts} は文字どおりの文字列として扱われます。 minimatch 単体では、先頭の ! は「このパターンに合わなければ true」という否定になります。src/a.js は src/*.ts に合わないので true です。 minimatch.braceExpand() の結果からわかるように、ブレースはマッチングの前に展開されて、複数のパターンになります。 makeRe:正規表現になる(v10まで) minimatch v10 系までは、パターンを正規表現に変換して照合していました。makeRe() で中身を見られます。 03-makere.cjs const mm10 = require('minimatch10') // minimatch@10.2.6 const mm11 = require('minimatch') // minimatch@11.0.3 for (const p of ['.js', 'src/*/.js', '.{js,ts}']) { console.log(p.padEnd(14), '->', mm10.makeRe(p)) } console.log('\nminimatch@11 に makeRe はある? ->', typeof mm11.makeRe, '/', typeof mm11.minimatch.makeRe) .js -> /^(?!.)[^/]?.js$/ src/**/.js -> /^src(?:\/|\/(?:(?!(?:\/|^).).)?\/)(?!.)[^/]*?.js$/ .{js,ts} -> /^(?:(?!.)[^/]?.js|(?!.)[^/]*?.ts)$/ minimatch@11 に makeRe はある? -> undefined / undefined 正規表現を読むと、ここまでの挙動がそのまま書かれています。 [^/]?: は / 以外の文字列 (?!.):先頭がドットならマッチしない *.{js,ts}:ブレース展開した2つのパターンを | でつないだもの 一方、2026年10月に出た minimatch v11 では、照合に正規表現を使わなくなりました。changelog によると、パスの区切りごとに @isaacs/nonrecursive-glob-match の Matcher で照合する方式に変わり、makeRe() は削除されています。README では、正規表現はバックトラックによる ReDoS(正規表現によるサービス拒否)を起こしうるため、という理由が説明されています。 ESLint 10.14.0 は minimatch ^10.2.6 に依存しているので、プロジェクトによっては v10 系と v11 系が両方入っていることもあります。 brace-expansion:文字列を展開する 04-brace.mjs import { expand } from 'brace-expansion' console.log(expand('file.{js,ts}')) console.log(expand('img{1..3}.png')) console.log(expand('{a,b}{x,y}')) console.log(expand('src/{components,hooks}/**/*.{ts,tsx}')) [ 'file.js', 'file.ts' ] [ 'img1.png', 'img2.png', 'img3.png' ] [ 'ax', 'ay', 'bx', 'by' ] [ 'src/components//*.ts', 'src/components//*.tsx', 'src/hooks//*.ts', 'src/hooks//*.tsx' ] {a,b}{x,y} が4件になるように、ブレースを並べると展開数は掛け算で増えます。小さい規模で確かめてみます。 for (const n of [1, 2, 4, 8, 12, 16]) { const r = expand('{a,b}'.repeat(n)) const totalChars = r.reduce((s, x) => s + x.length, 0) console.log(n=${String(n).padStart(2)} 入力${String(n * 5).padStart(3)}文字 -> ${String(r.length).padStart(6)}件 (理論値 2^n=${2 ** n}) 合計${totalChars}文字) } n= 1 入力 5文字 -> 2件 (理論値 2^n=2) 合計2文字 n= 2 入力 10文字 -> 4件 (理論値 2^n=4) 合計8文字 n= 4 入力 20文字 -> 16件 (理論値 2^n=16) 合計64文字 n= 8 入力 40文字 -> 256件 (理論値 2^n=256) 合計2048文字 n=12 入力 60文字 -> 4096件 (理論値 2^n=4096) 合計49152文字 n=16 入力 80文字 -> 65536件 (理論値 2^n=65536) 合計1048576文字 入力80文字で、出力は65,536件・合計約100万文字になりました。件数は 2^n、合計文字数は n × 2^n で増えます。 ! ここでは n=16 までにとどめています。これより大きい値を試すとメモリを大量に使うので、手元でも同じくらいの規模にとどめてください。 brace-expansion には、この増え方を抑える上限オプションがあります(README)。 max:展開する件数の上限(既定 100,000) maxLength:展開結果の合計文字数の上限(既定 4,000,000)。5.0.8 で追加 console.log(expand('{1..100}', { max: 10 }).length) // 10 const capped = expand('{a,b}'.repeat(12), { maxLength: 1000 }) console.log(capped.length, capped.reduce((s, x) => s + x.length, 0)) // 83 996 上限に達すると、エラーにはならず結果が途中で打ち切られます。 いざ、ツールごとの違い それではいよいよ、本題。 同じ fixture/ に対して、各ツールが実際にどのファイルを拾うかを試しました(デモの 05-tools.sh)。 ツール 設定項目 使用ライブラリ ドットファイル ** ブレース {a,b} 先頭 ! git .gitignore 独自(fnmatch(3) 準拠の記述) - でマッチする 対応 非対応(文字どおり扱う) 前のパターンで除外したものを再び含める。ただし親ディレクトリが除外されていると戻せない ESLint(flat config) files / ignores minimatch マッチする(ignores はドキュメントに明記、files は実測) 対応 対応 ignores 内で除外の取り消しに使える TypeScript include / exclude 独自(* ? **/ のみ記載) ワイルドカードではマッチしない。src/.config/* のように明示すれば拾う **/ 対応 非対応(該当なしでエラー) 記載なし Vite import.meta.glob() tinyglobby 既定ではマッチしない。exhaustive:true で指定 対応 対応 除外パターンとして使える Jest testMatch micromatch マッチする 対応 対応 除外。配列の順序が効く(否定は後ろに置く) 実測の出力(抜粋)は次のとおりです。 === git (.gitignore) -- .gitignore: src/*.js src/a.js ignored src/.hidden.js ignored src/deep/nested/d.js not-ignored -- .gitignore: src/*.{js,ts} src/a.js not-ignored src/b.ts not-ignored === TypeScript (tsconfig include) Version 7.0.2 -- include: ["src//*"] src/__tests__/e.js src/a.js src/app.test.js src/b.ts src/deep/nested/d.js -- include: ["src//*.{js,ts}"] error TS18003: No inputs were found in config file 'tsconfig.json'. ... -- include: ["src/.config/*"] src/.config/c.js src/.config/f.test.js === ESLint (flat config files: ['src/**/*.{js,ts}']) v10.14.0 src/.config/c.js src/.config/f.test.js src/.hidden.js src/tests/e.js src/a.js src/app.test.js src/b.ts src/deep/nested/d.js === Jest (testMatch) 30.5.2 -- default testMatch src/.config/f.test.js src/tests/e.js src/app.test.js === Vite (import.meta.glob) vite/8.3.5 plain ./src/tests/e.js, ./src/a.js, ./src/app.test.js, ./src/deep/nested/d.js brace ./src/tests/e.js, ./src/a.js, ./src/app.test.js, ./src/b.ts, ./src/deep/nested/d.js negated ./src/tests/e.js, ./src/a.js, ./src/deep/nested/d.js exhaustive ./src/.config/c.js, ./src/.config/f.test.js, ./src/.hidden.js, ... 読み取れることをまとめます。 ドットファイル: ESLint と Jest は .config/ の中まで拾いますが、TypeScript と Vite は拾いません。.storybook/ のような設定ディレクトリが lint やテストの対象に入る・入らないの差は、ここから来ます。 ブレース展開: .gitignore と tsconfig の include では使えません。tsconfig で "src/**/*.{ts,tsx}" と書くと、ファイルが1つも見つからずエラーになりました。 否定: .gitignore の ! は「再び含める」、Jest の testMatch では「除外」で、しかも順序が効きます。同じ記号でも意味が違います。 まとめ:なぜツールごとに方言が生まれるのか ここまで見てきた違いは、大きく3つの原因に分けられます。 1. パターン判別のライブラリが違う git と TypeScript は独自の実装、ESLint は minimatch、Vite は tinyglobby、Jest は micromatch を使っています。 2. 同じライブラリでも、渡すオプションが違う ESLint は minimatch を使っていますが、ドットファイルにもマッチします。minimatch は既定ではドットファイルを拾いませんが、ESLint が dot: true を付けて呼び出しているからです。 3. 設定ファイル側の文法が違う 先頭の ! の意味は、ライブラリだけでなく、各ツールの設定ファイルの文法にもよります。.gitignore では「前のパターンで除外したものを再び含める」、Jest の testMatch では「除外」で、しかも配列の順序が効きます。 同じ glob パターンでも、どのライブラリを、どのオプションで、どんな文法の中で使っているかによって結果が変わるということですね。迷ったら、各ツールのファイル一覧を出すコマンド(git check-ignore、tsc --listFilesOnly、eslint --inspect-config、jest --listTests)で実際の結果を確認するのが確実です。 おまけ:glob と minimatch を書いた人 最後に少し寄り道です。この記事で何度も出てきた glob と minimatch は、npm registry の author 欄を見ると同じ人の名前になっています。 npm view glob author # Isaac Z. Schlueter <i@izs.me> (blog.izs.me) npm view minimatch author # Isaac Z. Schlueter <i@izs.me> (blog.izs.me) Isaac Z. Schlueter さん(GitHub では isaacs)は、npm を作った人です。本人の GitHub プロフィールには「npm inventor, founder npm, Inc. Former Node BDFL.」と書かれています。 author が Isaac Z. Schlueter さんになっているパッケージには、ほかにも次のようなものがあります。 パッケージ 用途 週間ダウンロード数(2026-10-23〜10-29) minimatch glob パターンの照合 951,240,180 lru-cache LRU キャッシュ 789,165,914 glob ファイル検索 536,547,167 rimraf rm -rf 相当の削除 214,603,009 tar tar アーカイブの読み書き 119,327,291 (npm の downloads API で取得) package.json の ^5.0.8 のようなバージョン範囲を解釈する semver も、LICENSE に「Copyright (c) Isaac Z. Schlueter and Contributors」とあります(現在の author 欄は GitHub Inc.、リポジトリは npm organization の管理です)。 普段使っている npm を作った人が、node_modules によく入っている小さな部品もたくさん書いている、ということになります。npm CLI 自身が、glob・minimatch・semver・tar に依存しているのを見ると、開発の中で自身が必要になったものを、片っ端から生み出していった結果なのかもしれません。 参考資料 glob(7) - Linux manual page: man7.org Bash Reference Manual - Brace Expansion: gnu.org Bash Reference Manual - The Shopt Builtin(globstar / dotglob): gnu.org isaacs/node-glob README: isaacs/node-glob isaacs/minimatch README / changelog: isaacs/minimatch , github.com/isaacs/minimatch/blob/main/changelog.md juliangruber/brace-expansion README: juliangruber/brace-expansion gitignore Documentation: git-scm.com ESLint - Configuration Files: eslint.org ESLint v10.14.0 リリースノート: github.com/eslint/eslint/releases/tag/v10.14.0 TSConfig Reference - include: typescriptlang.org Vite - Glob Import: vite.dev tinyglobby documentation: superchupu.dev Jest - testMatch: jestjs.io 脚注 POSIX(Portable Operating System Interface)は、UNIX 系 OS の間でアプリケーションを移植しやすくするための標準規格です。最新版の POSIX.1-2024 は IEEE Std 1003.1-2024 と The Open Group の Base Specifications Issue 8 を兼ねていて、OS のインターフェースに加えてシェルや共通コマンドも定めています。glob の記号は、シェルの章の「2.14 Pattern Matching Notation」で規定されています。出典:The Open Group Base Specifications Issue 8、同 2.14 Pattern Matching Notation、IEEE 1003.1-2024 ↩︎
CVE-2026-118773: CVE-2026-118773: Asynchronous Control Flow Bypass and Uncontrolled Resource Consumption in node-glob CVE-2026-118773: Asynchronous Control Flow Bypass and Uncontrolled Resource Consumption in node-glob > Vulnerability ID: CVE-2026-118773 > CVSS Score: 6.3 > Published: 2026-10-30 CVE-2026-118773 (GHSA-3g3m-2fjr-6pw4) is an uncontrolled resource consumption vulnerability in the node-glob npm package affecting versions 9.0.0 through 14.0.1. Due to missing return statements in the AbortSignal check routines within the directory traversal walker, aborted operations reject the caller promise as expected but continue executing background filesystem reads and pattern matching. Applications that rely on AbortSignal or AbortSignal.timeout() to enforce execution budgets on user-influenced glob operations remain susceptible to CPU, event loop, and disk I/O exhaustion. TL;DR Missing return statements in node-glob's AbortSignal handling logic cause directory traversal operations to continue running in the background even after an operation is aborted or times out. Upgrading to glob version 14.0.2 resolves the issue by properly terminating the control flow upon signal abortion. --- ⚠️ Exploit Status: POC Technical Details - Vulnerability ID: CVE-2026-118773 / GHSA-3g3m-2fjr-6pw4 - Weakness Class: CWE-400 (Uncontrolled Resource Consumption) - Affected Component: glob (node-glob) >= 9.0.0, < 14.0.2 - CVSS v4.0 Score: 6.3 Medium (CVSS:4.0/AV:N/AC:L/AT:P/PR:N/UI:N/VC:N/VI:N/VA:L/SC:N/SI:N/SA:N) - EPSS Score: 0.35% (28.69th Percentile) - Impact: Post-timeout background filesystem traversal leading to I/O and CPU exhaustion - Exploit Status: Proof-of-Concept (Unit test reproduction in patch commit) - CISA KEV Status: Not Listed Affected Systems - node-glob (npm package: glob) - glob: >= 9.0.0, < 14.0.2 (Fixed in: 14.0.2) Code Analysis Commit: 4682e8c fix: do not continue walking after signal abort diff --git a/src/walker.ts b/src/walker.ts index a675fee2..34f07665 100644 --- a/src/walker.ts +++ b/src/walker.ts @@ -155,7 +155,6 @@ export abstract class GlobUtil<O extends GlobWalkerOpts = GlobWalkerOpts> { } } onResume(fn: () => unknown) { - if (this.signal?.aborted) return /* c8 ignore start */ if (!this.paused) { fn() @@ -281,7 +280,7 @@ export abstract class GlobUtil<O extends GlobWalkerOpts = GlobWalkerOpts> { walkCB(target: Path, patterns: Pattern[], cb: () => unknown) { /* c8 ignore start */ - if (this.signal?.aborted) cb() + if (this.signal?.aborted) return cb() /* c8 ignore stop */ this.walkCB2(target, patterns, new Processor(this.opts), cb) } @@ -293,7 +292,7 @@ export abstract class GlobUtil<O extends GlobWalkerOpts = GlobWalkerOpts> { cb: () => unknown, ) { if (this.#childrenIgnored(target)) return cb() - if (this.signal?.aborted) cb() + if (this.signal?.aborted) return cb() if (this.paused) { this.onResume(() => this.walkCB2(target, patterns, processor, cb)) return @@ -365,7 +364,7 @@ export abstract class GlobUtil<O extends GlobWalkerOpts = GlobWalkerOpts> { walkCBSync(target: Path, patterns: Pattern[], cb: () => unknown) { /* c8 ignore start */ - if (this.signal?.aborted) cb() + if (this.signal?.aborted) return cb() /* c8 ignore stop */ this.walkCB2Sync(target, patterns, new Processor(this.opts), cb) } @@ -377,7 +376,7 @@ export abstract class GlobUtil<O extends GlobWalkerOpts = GlobWalkerOpts> { cb: () => unknown, ) { if (this.#childrenIgnored(target)) return cb() - if (this.signal?.aborted) cb() + if (this.signal?.aborted) return cb() if (this.paused) { this.onResume(() => this.walkCB2Sync(target, patterns, processor, cb), Exploit Details - GitHub: Official regression test case in test/signal.ts demonstrating continued fs.readdir and fs.readdirSync calls after AbortController.abort() is triggered. Mitigation Strategies - Upgrade node-glob to version 14.0.2 or later to ensure AbortSignal immediately terminates directory traversal routines. - Validate and sanitize untrusted glob patterns by rejecting unbounded globstar (**) expressions and constraining traversal depth via the maxDepth option. - Scope filesystem searches to narrow working directories (cwd) and configure explicit ignore patterns for large directory trees such as node_modules. - Isolate potentially expensive glob operations inside dedicated worker threads or child processes that can be forcibly terminated via worker.terminate() or SIGKILL on timeout. Remediation Steps: 1. Audit project dependency trees using 'npm ls glob' or 'pnpm why glob' to identify direct and transitive installations of glob >= 9.0.0 and < 14.0.2. 2. Update package.json manifests to require glob@^14.0.2 or apply package manager overrides/resolutions for transitive dependencies. 3. Run regression tests to verify compatibility with glob v14.0.2 and confirm that aborted operations no longer generate background disk I/O. References - GitHub Security Advisory GHSA-3g3m-2fjr-6pw4 - Fix Commit: do not continue walking after signal abort - node-glob Release v14.0.2 - NVD Vulnerability Detail: CVE-2026-118773 - CVE.org Record: CVE-2026-118773 --- Read the full report for CVE-2026-118773 on our website for more details including interactive diagrams and full exploit analysis.
CVE-2026-118764: CVE-2026-118764: Asymmetric Resource Consumption and Denial of Service in minimatch Pattern Preprocessing CVE-2026-118764: Asymmetric Resource Consumption and Denial of Service in minimatch Pattern Preprocessing > Vulnerability ID: CVE-2026-118764 > CVSS Score: 6.9 > Published: 2026-10-30 CVE-2026-118764 (also tracked as GHSA-g3hv-75wg-m249) is an asymmetric resource consumption vulnerability (CWE-405) in the npm package minimatch prior to version 11.0.1. When pattern compilation is invoked with an optimizationLevel of 2 or higher, the internal preprocessing functions firstPhasePreProcess() and secondPhasePreProcess() in src/index.ts perform unbounded pattern splitting and quadratic pairwise deduplication. An attacker supplying a crafted glob string containing repeated **/.. segments can induce a combinatorial explosion of pattern permutations followed by an O(M^2) comparison routine, blocking the Node.js event loop and exhausting process memory. TL;DR Unbounded glob pattern splitting on repeated '**/..' segments combined with quadratic pairwise deduplication at optimizationLevel >= 2 allows attackers to trigger severe CPU and memory exhaustion in minimatch versions prior to 11.0.1. --- ⚠️ Exploit Status: POC Technical Details - Vulnerability ID: CVE-2026-118764 / GHSA-g3hv-75wg-m249 - Weakness Class: CWE-405: Asymmetric Resource Consumption (Amplification) - Affected Component: minimatch (src/index.ts: firstPhasePreProcess & secondPhasePreProcess) - CVSS v4.0 Score: 6.9 Medium (GitHub CNA) / 8.7 High (OSV Availability Variant) - EPSS Score: 0.291% (21.74th Percentile) - Trigger Condition: Untrusted glob pattern compiled with optimizationLevel >= 2 - Impact: Node.js Event Loop Exhaustion & Out-of-Memory Denial of Service - Fixed Version: minimatch 11.0.1 Affected Systems - minimatch (npm package) - minimatch: < 11.0.1 (Fixed in: 11.0.1) Code Analysis Commit: 77a0f5c At optimization level 2, stop splitting **/.. portions once there are braceExpandMax patterns, and do not dedupe sets of more than 1000 patterns. diff --git a/src/index.ts b/src/index.ts index 402bba8f..7e7c2964 100644 --- a/src/index.ts +++ b/src/index.ts @@ -1150,6 +1153,8 @@ export class Minimatch { firstPhasePreProcess(globParts: string[][]) { + // each **/.. split doubles the patterns, so stop at the brace max + const { braceExpandMax = 100_000 } = this.options let didSomething = false do { didSomething = false @@ -1178,7 +1183,8 @@ export class Minimatch { p === '..' || !p2 || p2 === '.' || - p2 === '..' + p2 === '..' || + globParts.length >= braceExpandMax ) { continue } @@ -1242,6 +1248,9 @@ export class Minimatch { secondPhasePreProcess(globParts: string[][]): string[][] { + // each pattern is compared to each other one, so skip this for + // large sets. The patterns match the same paths either way. + if (globParts.length > 1000) return globParts for (let i = 0; i < globParts.length - 1; i++) { Exploit Details - GitHub: Official fix commit including test/preprocess-limits.ts demonstrating the 30-segment '**/../a/b' trigger payload. Mitigation Strategies - Upgrade minimatch to version 11.0.1 or newer to enforce pattern splitting and deduplication bounds. - Avoid configuring optimizationLevel >= 2 when compiling glob patterns derived from untrusted user input. - Configure a lower braceExpandMax option (e.g., 500 or 1000) to restrict maximum pattern permutations in memory. - Sanitize and length-limit user-supplied glob strings prior to compilation, rejecting patterns with repeated '**/..' traversal segments. Remediation Steps: 1. Run 'npm ls minimatch' or inspect package-lock.json, yarn.lock, or pnpm-lock.yaml to identify all installed versions of minimatch. 2. Upgrade direct dependencies by running 'npm install minimatch@^11.0.1' or updating package.json overrides/resolutions for transitive dependencies. 3. Audit application source code for calls to 'minimatch()' or 'new Minimatch()' that specify '{ optimizationLevel: 2 }' or pass unvalidated user options. 4. Deploy updated lockfiles and verify via unit tests that pattern compilation completes within expected time bounds. References - GitHub Security Advisory GHSA-g3hv-75wg-m249 - Official Fix Commit 77a0f5c4b80bdc641c3c930a1d9edf5ad2db57f8 - minimatch Release v11.0.1 - NVD Vulnerability Record for CVE-2026-118764 - CVE.org Authoritative Record for CVE-2026-118764 --- Read the full report for CVE-2026-118764 on our website for more details including interactive diagrams and full exploit analysis.
Building an In-Memory Semantic Cache for LLM Endpoints with Node.js and pgvector If you run an LLM-backed endpoint (a support bot, a docs assistant, a "chat with your data" feature), you've probably noticed something in your logs: people ask the same questions over and over, just worded differently. - "How do I reset my password?" - "I forgot my password, what do I do?" - "password reset steps" A traditional cache keyed on the exact prompt string treats these as three unrelated requests, so you pay for three LLM calls, three times the latency, and three times the tokens. A semantic cache fixes this. Instead of matching strings, it matches meaning: embed the incoming prompt, find the nearest previously answered prompt, and if it's close enough, return the stored answer without ever calling the LLM. In this post we'll build one in Node.js and TypeScript with a two-tier design: - L1: in-process memory. Microsecond lookups for exact repeats. - L2: Postgres + pgvector. Shared across instances, durable, with similarity search via an HNSW index. We'll also cover the part most tutorials skip: when a cache hit is actually a wrong answer, and how to design around it. --- How a semantic cache works Request ─▶ normalize ─▶ L1 exact lookup ──hit──▶ return │ miss ▼ embed(prompt) ◀── ~50-300 ms, small cost │ ▼ L2 nearest neighbor (pgvector) │ similarity ≥ threshold? ──yes──▶ return cached answer (+ promote to L1) │ no ▼ call the LLM │ ▼ store in L1 + L2 (embedding + answer) ─▶ return The economics work because embedding a prompt is much cheaper and faster than generating a completion. A miss costs you one extra embedding call plus a DB query. A hit saves a full generation. If your hit rate is decent, you come out well ahead on both cost and p50 latency. Real numbers depend entirely on your traffic, so measure them (we'll add metrics later) rather than trusting anyone's blog post, including this one. --- Step 1: Set up pgvector Enable the extension and create the cache table: CREATE EXTENSION IF NOT EXISTS vector; CREATE TABLE llm_cache ( id BIGSERIAL PRIMARY KEY, scope TEXT NOT NULL, -- tenant + model + prompt version (see below) prompt_hash TEXT NOT NULL, -- hash of normalized prompt, for exact match prompt TEXT NOT NULL, -- original text, useful for debugging and audits embedding vector(1536) NOT NULL, response JSONB NOT NULL, hit_count INTEGER NOT NULL DEFAULT 0, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), expires_at TIMESTAMPTZ NOT NULL ); -- Exact-match lookups CREATE UNIQUE INDEX llm_cache_scope_hash_idx ON llm_cache (scope, prompt_hash); -- Approximate nearest neighbor search using cosine distance CREATE INDEX llm_cache_embedding_idx ON llm_cache USING hnsw (embedding vector_cosine_ops); -- For cleanup jobs CREATE INDEX llm_cache_expires_idx ON llm_cache (expires_at); Notes: - vector(1536) must match your embedding model's output dimension. Check your model's docs, and keep it consistent. Mixing models in one column breaks similarity. - HNSW gives fast approximate search and doesn't require training data first (unlike IVFFlat). It needs pgvector 0.5.0 or newer. - <=> is pgvector's cosine distance operator. Cosine similarity is 1 - distance. The scope column is the most important design decision A cached answer is only valid for the context it was generated in. Put everything that changes the correct answer into scope: function buildScope(opts: { tenantId: string; model: string; promptVersion: string; // bump when you change your system prompt toolsetVersion?: string; }) { return [opts.tenantId, opts.model, opts.promptVersion, opts.toolsetVersion ?? '-'].join('|'); } Without this, tenant A's answer can be served to tenant B (a data leak), or a stale answer generated by an old system prompt keeps being served after you "fixed" the bot. Bumping promptVersion is your cheap, instant "invalidate everything" button. --- Step 2: The cache class Install dependencies: npm i pg pgvector lru-cache Here's the core implementation: // semantic-cache.ts import crypto from 'node:crypto'; import { Pool } from 'pg'; import pgvector from 'pgvector/pg'; import { LRUCache } from 'lru-cache'; export interface CachedResponse { text: string; model: string; usage?: { inputTokens: number; outputTokens: number }; } export interface CacheLookup { hit: boolean; tier?: 'l1' | 'l2'; similarity?: number; response?: CachedResponse; } export class SemanticCache { // L1: exact-match, in-process. Key = scope + prompt hash. private l1 = new LRUCache<string, CachedResponse>({ max: 5_000, ttl: 10 * 60 * 1000, // 10 minutes }); // Single-flight: if two identical requests arrive at once, only one calls the LLM private inflight = new Map<string, Promise<CachedResponse>>(); constructor( private pool: Pool, private embed: (text: string) => Promise<number[]>, private opts = { similarityThreshold: 0.92, ttlSeconds: 24 * 60 * 60, }, ) {} static normalize(prompt: string): string { return prompt.trim().toLowerCase().replace(/\s+/g, ' '); } private hash(text: string) { return crypto.createHash('sha256').update(text).digest('hex'); } async lookup(scope: string, prompt: string): Promise<CacheLookup & { embedding?: number[] }> { const normalized = SemanticCache.normalize(prompt); const promptHash = this.hash(normalized); const l1Key = `${scope}:${promptHash}`; // --- L1: exact match in memory --- const l1Hit = this.l1.get(l1Key); if (l1Hit) return { hit: true, tier: 'l1', similarity: 1, response: l1Hit }; // --- L2a: exact match in Postgres (no embedding needed) --- const exact = await this.pool.query( `SELECT response FROM llm_cache WHERE scope = $1 AND prompt_hash = $2 AND expires_at > now()`, [scope, promptHash], ); if (exact.rows[0]) { this.l1.set(l1Key, exact.rows[0].response); return { hit: true, tier: 'l2', similarity: 1, response: exact.rows[0].response }; } // --- L2b: semantic match via pgvector --- const embedding = await this.embed(normalized); const { rows } = await this.pool.query( `SELECT id, response, 1 - (embedding <=> $2) AS similarity FROM llm_cache WHERE scope = $1 AND expires_at > now() ORDER BY embedding <=> $2 LIMIT 1`, [scope, pgvector.toSql(embedding)], ); const best = rows[0]; if (best && best.similarity >= this.opts.similarityThreshold) { // Fire-and-forget stats update; don't block the response this.pool .query('UPDATE llm_cache SET hit_count = hit_count + 1 WHERE id = $1', [best.id]) .catch(() => {}); this.l1.set(l1Key, best.response); // promote so the exact phrasing is fast next time return { hit: true, tier: 'l2', similarity: best.similarity, response: best.response, embedding }; } return { hit: false, embedding }; } async store( scope: string, prompt: string, response: CachedResponse, embedding?: number[], ): Promise<void> { const normalized = SemanticCache.normalize(prompt); const promptHash = this.hash(normalized); this.l1.set(`${scope}:${promptHash}`, response); const vec = embedding ?? (await this.embed(normalized)); await this.pool.query( `INSERT INTO llm_cache (scope, prompt_hash, prompt, embedding, response, expires_at) VALUES ($1, $2, $3, $4, $5, now() + ($6 || ' seconds')::interval) ON CONFLICT (scope, prompt_hash) DO UPDATE SET response = EXCLUDED.response, embedding = EXCLUDED.embedding, expires_at = EXCLUDED.expires_at`, [scope, promptHash, prompt, pgvector.toSql(vec), response, String(this.opts.ttlSeconds)], ); } /** Wrap an LLM call with cache lookup, single-flight, and write-back. */ async getOrCompute( scope: string, prompt: string, compute: () => Promise<CachedResponse>, ): Promise<CacheLookup & { response: CachedResponse }> { const found = await this.lookup(scope, prompt); if (found.hit) return found as CacheLookup & { response: CachedResponse }; const key = `${scope}:${this.hash(SemanticCache.normalize(prompt))}`; let pending = this.inflight.get(key); if (!pending) { pending = compute() .then(async (res) => { // Don't let a cache write failure fail the user's request await this.store(scope, prompt, res, found.embedding).catch((e) => console.error('cache store failed', e), ); return res; }) .finally(() => this.inflight.delete(key)); this.inflight.set(key, pending); } return { hit: false, response: await pending }; } } A few things worth calling out: - Exact match before embedding. Many repeats are literally identical. Skipping the embedding call for those saves both time and money. - Single-flight. If 50 users ask the same new question at once (say, right after an outage), you want one LLM call, not 50. The inflight map handles that within a process. - The cache must never break the request. Failures on store are logged and swallowed. You should also wrap lookup so a database outage degrades to "just call the LLM" (see the Express example below). - Parameterized queries throughout. pgvector.toSql() serializes the array safely for the vector type. --- Step 3: Wire it into an endpoint // server.ts import express from 'express'; import { Pool } from 'pg'; import pgvector from 'pgvector/pg'; const pool = new Pool({ connectionString: process.env.DATABASE_URL }); pool.on('connect', (client) => pgvector.registerTypes(client)); const cache = new SemanticCache(pool, embedText); const app = express(); app.use(express.json()); app.post('/api/ask', async (req, res) => { const { question } = req.body; const tenantId = req.user.tenantId; // from your auth middleware const scope = buildScope({ tenantId, model: process.env.LLM_MODEL!, promptVersion: 'support-bot-v7', }); const started = performance.now(); try { const result = await cache.getOrCompute(scope, question, () => callLlm({ system: SYSTEM_PROMPT, user: question }), ); metrics.timing('ask.latency_ms', performance.now() - started, { cache: result.hit ? result.tier! : 'miss', }); metrics.increment('ask.cache', { result: result.hit ? 'hit' : 'miss' }); res.json({ answer: result.response.text, cached: result.hit }); } catch (err) { // If the cache layer itself blows up, fall back to the LLM directly console.error('cache path failed, calling LLM directly', err); const direct = await callLlm({ system: SYSTEM_PROMPT, user: question }); res.json({ answer: direct.text, cached: false }); } }); One caveat on that catch: it also catches errors from the LLM call itself, so you may double-call the LLM if the first call failed. In production, separate "cache infrastructure errors" from "compute errors" (for example by having lookup return a miss on DB failure internally). --- The hard part: choosing a similarity threshold The threshold is the knob that trades hit rate against correctness, and there's no universal right value. It depends on your embedding model, your domain, and how costly a wrong answer is. - Too low: you serve wrong answers confidently. - Too high: you rarely hit, and the cache is expensive overhead. Embedding similarity measures topical closeness, not intent equivalence. These pairs are all highly similar to an embedding model but need different answers: | Prompt A | Prompt B | Same answer OK? | |---|---|---| | "How do I cancel my subscription?" | "How do I not cancel my subscription?" | ❌ | | "What's the refund policy for annual plans?" | "What's the refund policy for monthly plans?" | ❌ | | "Is feature X available in Python?" | "Is feature X available in Go?" | ❌ | | "How do I reset my password?" | "I forgot my password" | ✅ | Calibrate with your own data Don't guess. Build a small labeled set from real logs: pairs of prompts marked "same intent" or "different intent". Then sweep thresholds: // calibrate.ts interface Pair { a: string; b: string; sameIntent: boolean } function cosine(a: number[], b: number[]) { let dot = 0, na = 0, nb = 0; for (let i = 0; i < a.length; i++) { dot += a[i] * b[i]; na += a[i] ** 2; nb += b[i] ** 2; } return dot / (Math.sqrt(na) * Math.sqrt(nb)); } async function sweep(pairs: Pair[]) { const scored = []; for (const p of pairs) { const [ea, eb] = await Promise.all([embedText(p.a), embedText(p.b)]); scored.push({ sim: cosine(ea, eb), sameIntent: p.sameIntent }); } for (let t = 0.80; t <= 0.99; t += 0.01) { const hits = scored.filter((s) => s.sim >= t); const falsePositives = hits.filter((s) => !s.sameIntent).length; const truePositives = hits.length - falsePositives; console.log( `threshold=${t.toFixed(2)} hits=${hits.length} ` + `correct=${truePositives} WRONG=${falsePositives}`, ); } } Choose the lowest threshold where WRONG is acceptably close to zero for your risk tolerance. For a customer-facing bot that quotes prices or policies, be very conservative. For a casual "explain this concept" endpoint, you can be looser. Defense in depth beyond the threshold 1. Segment by scope. Different products, plans, or languages shouldn't share cache entries. 2. Keep entities out of the fuzzy match. If prompts contain order IDs, names, or numbers, either extract them into the scope/key or skip semantic matching when they're present, because the nearest neighbor of "status of ORD-123456" is "status of ORD-654321". 3. Per-category thresholds. Stricter for transactional or factual queries, looser for general explanations. 4. Log near-misses. Record hits between 0.88 and 0.95 along with the matched prompt for periodic human review. It's the best source of calibration data you'll get. --- What you should not cache A cache hit means "the answer from last time is still correct". That's false for many requests: - Personalized answers ("what's my balance?"). Different users, different truths. - Time-sensitive answers ("what's the status of my order?", "what's today's date?"). - Responses that used tools with live data. Caching the final text freezes that data in place. - Non-deterministic or creative requests where the user expects variety ("give me another idea"). - Multi-turn context. If the answer depends on earlier messages, the last user message alone is not the cache key. Either only cache stateless first turns, or embed a summary of the conversation context. - Anything with sensitive data, unless you've thought through retention, access control, and your compliance obligations. Cached prompts and answers are stored data. Implement this as an explicit policy function rather than scattered ifs: function isCacheable(req: AskRequest): boolean { if (req.conversation.length > 1) return false; // multi-turn if (req.usedTools) return false; // live data involved if (req.user.requiresPersonalization) return false; if (/\b(today|now|current|latest|my)\b/i.test(req.question)) return false; // crude, tune it return true; } --- Performance and operations HNSW tuning At query time, hnsw.ef_search controls the recall/speed tradeoff (the default is 40; higher means better recall but slower): SET hnsw.ef_search = 100; For a cache you can usually tolerate slightly imperfect recall, since a missed neighbor just means one extra LLM call. Filtering with HNSW Our query filters by scope. With HNSW, filtering is applied after the index scan, so a highly selective filter can return fewer rows than you asked for (or none, even though matches exist). Options: - Use pgvector's iterative index scans (available in newer versions, via hnsw.iterative_scan) - Use partial indexes or table partitioning per large tenant - Keep scopes coarse enough that most scans find candidates Check the pgvector README for the options available in your installed version. Expiry and cleanup expires_at filtering in queries keeps stale entries from being served, but they still occupy space and index memory. Run a periodic cleanup: DELETE FROM llm_cache WHERE expires_at < now() - interval '1 day'; Run it in batches, or use partitioning by time if the table is large. Consider also evicting entries with zero hits after N days. They're not earning their keep. Embedding latency is on your critical path Every cache miss pays for an embedding call and the LLM call. Mitigations: - Skip semantic lookup for very short or very long prompts (tune the bounds) - Set a tight timeout on the embedding call. If it's slow, treat it as a miss and go straight to the LLM - Reuse the embedding computed during lookup when storing (we do this via the returned embedding) Streaming responses If you stream tokens to users, you can't cache until the stream completes. Buffer the full text as it streams, then store it afterwards. On a hit, you can either return the whole answer instantly or simulate streaming by chunking the cached text, which keeps your frontend code identical for both paths. A note on "in-memory" scaling The L1 cache is per-process. With many instances, each warms up independently, and invalidation (via promptVersion in the scope) works because new scopes simply never match old keys. If you need cross-instance L1 coherence, that's a sign to add Redis as a shared tier, but start without it. L1 + pgvector covers most workloads, and every extra tier is another thing to keep consistent. --- Metrics that tell you if it's working Instrument from day one: - Hit rate by tier (l1, l2, miss) and by scope or category - Latency by outcome (hit vs miss). Compare p50 and p95. - Embedding latency and error rate - Similarity distribution of hits. If most hits cluster right at your threshold, you're likely over-matching. - Estimated tokens saved, from the usage stored on each cached response - User feedback signals (thumbs down, re-asks) segmented by cached: true/false. If cached answers get disproportionately downvoted, raise the threshold. That last one is the real test. A cache with a great hit rate and worse user satisfaction is a regression. --- Putting it together ┌───────────────────────────────┐ Request ──▶ │ isCacheable? ── no ─────────┼──▶ LLM └──────────────┬────────────────┘ │ yes ┌──────────▼──────────┐ │ L1: in-process LRU │── hit ──▶ response (µs) └──────────┬──────────┘ │ miss ┌──────────▼──────────┐ │ L2: exact (Postgres)│── hit ──▶ response (ms) └──────────┬──────────┘ │ miss ┌──────────▼──────────┐ │ embed + pgvector kNN│── ≥ threshold ──▶ response └──────────┬──────────┘ │ miss ┌──────────▼──────────┐ │ single-flight → LLM │──▶ store L1 + L2 ──▶ response └─────────────────────┘ Key takeaways 1. A semantic cache matches meaning, not strings. Embed the prompt, find the nearest cached prompt, and reuse the answer above a similarity threshold. 2. Two tiers are enough. An in-process LRU for exact repeats and Postgres with pgvector (HNSW) for shared, durable semantic matches. 3. Put everything that affects correctness into the scope: tenant, model, prompt version. This prevents data leaks and gives you instant invalidation. 4. Calibrate the threshold with labeled data from your own traffic. Similar embeddings don't guarantee equivalent intent. 5. Be deliberate about what's cacheable. Personalized, time-sensitive, tool-backed, and multi-turn requests usually aren't. 6. Make the cache fail-open. It should only ever make your endpoint faster, never less available. 7. Measure user-facing quality, not just hit rate. Have you built a semantic cache, or tried one and abandoned it? I'd love to hear what threshold and embedding model worked for you in the comments. --- Further reading: the pgvector README (HNSW options, iterative scans, distance operators), pgvector-node, and lru-cache.