virgiliojr94

4. 두꺼운 기술 서적 PDF를 넣으면 필요한 장만 찾아 답해줘요. 핵심 개념과 피해야 할 실수까지 장별로 정리하고, AI 코딩 도구가 있어야 해요. virgiliojr94/book-to-skill

Turning Books into Actionable Agent Skills: A Deep Dive into book-to-skill (and Docling) Implementing and testing book-to-skill with a personal touch! !Image description Introduction As developers, we frequently buy comprehensive technical books or accumulate dense internal documentation repositories, read them once, and struggle to retain or quickly reference specific frameworks later. Traditional approaches—like searching raw PDFs or dumping entire books into an LLM context window—either fail to yield precise answers or suffer from the "discovery loop tax," wasting tokens and causing hallucinations. book-to-skill completely transforms this workflow by distilling any technical book, document folder, or collection of sources into a structured, on-demand agent skill (SKILL.md and modular chapter files) compatible with cross-agent standards like GitHub Copilot CLI, Claude Code, OpenCode and last but surely not least, IBM Bob. What immediately sparked my interest in this project is its intelligent extraction strategy, specifically its integration of docling for in-depth document information extraction. While standard text extractors struggle with technical books filled with complex layouts, docling preserves markdown tables, formulas, and code blocks seamlessly (~1.5s/page), ensuring that code examples and technical structures aren't lost in translation. book-to-skill Core Concepts !Image from github repository book-to-skill > Image from github repository book-to-skill To truly appreciate the bridge between document extraction and agent intelligence, it helps to examine the core philosophy and mechanics outlined in the book-to-skill project specification: - The Problem It Solves: Technical books, architecture decision records, and internal documentation are read once and forgotten. Standard workarounds—such as keyword-searching PDFs, asking an unguided LLM to recall contents, or maintaining static notes—frequently lead to hallucinations, token fatigue, or abandoned documents. - The "Discovery Loop Tax": Traditional PDF-reading agents waste resources by constantly re-navigating table-of-contents and reprocessing entire files on every turn. By paying the structural synthesis cost once at conversion time, book-to-skill achieves 24× to 51× fewer tokens per query compared to full context dumps. - Universal Format and Agent Support: The tool transforms files (PDF, EPUB, DOCX, Markdown, HTML, RTF, MOBI) into standard-compliant agent skills (SKILL.md) natively supported across major development hosts including GitHub Copilot CLI, Claude Code, Amp, Codex, Hermes Agent, OpenCode, and OpenClaw. - Granular Artifact Layout: Generated skills are neatly partitioned into on-demand components: - SKILL.md: Master index and core frameworks (~4,000 tokens). - chapters/: Individual chapter files loaded strictly on-demand (~1,000 tokens each). - glossary.md, patterns.md, and cheatsheet.md: Alphabetical key terms, design patterns, and quick-reference decision heuristics. - Privacy and Local-First Execution: Processing runs entirely on the user's local machine, ensuring full data sovereignty and adherence to personal or enterprise privacy policies. --- Implementation > Disclaimer: While book-to-skill is fully functional and ready to use straight out of the box, my primary goal here was to wrap its powerful extraction capabilities into a streamlined, user-friendly interface. This implementation serves as a personal sandbox to bring my own vision of automated document intelligence and local-first agent workflows to life. To make book-to-skill accessible via a modern web interface, I designed a robust, local-first FastAPI backend that wraps the extraction pipeline and orchestrates an automated two-phase processing pipeline: 1. Phase 1 (Extraction): Runs the core extraction script asynchronously via a non-blocking subprocess, directing output to an isolated per-job directory (output/<job_id>/). 2. Phase 2 (Skill Generation): Dispatches LLM-based skill generation to a thread pool (ThreadPoolExecutor), producing core mental models, chapter summaries, glossaries, patterns, and cheatsheets. Architecture Schema !Image description Two-Phase Processing Pipeline The processing pipeline functions as the following; User selects file + options (generate_skill: true/false) │ ▼ POST /api/process { file_path, mode, skill_name, generate_skill } │ ▼ FastAPI creates job_id, stores { status: "pending", skill_status: null } │ ▼ ───────────────────────────────────────────── PHASE 1 — EXTRACTION (asyncio subprocess) ───────────────────────────────────────────── background task sets BOOK_SKILL_WORKDIR=output/<job_id>/ asyncio.create_subprocess_exec(extract.py …) │ ├─ stdout → job["stdout"] ├─ stderr → job["stderr"] └─ on exit (rc=0): parse output/<job_id>/metadata.json read output/<job_id>/full_text.txt job["metadata"] / ["extracted_text"] populated job["status"] = "done" │ ▼ ───────────────────────────────────────────── PHASE 2 — SKILL GENERATION (thread pool) ───────────────────────────────────────────── (only if generate_skill=True and extraction succeeded) job["skill_status"] = "running" loop.run_in_executor(thread_pool, generate_skill, …) │ ├─ probe LLM endpoint (8 s timeout) │ reachable → use _chat() for each artifact │ unreachable → use stub generators │ ├─ generate per-chapter summaries → chapters/ch01-*.md … ├─ generate glossary → glossary.md ├─ generate patterns → patterns.md ├─ generate cheatsheet → cheatsheet.md └─ generate master SKILL.md → SKILL.md job["skill_status"] = "done" | "error" job["skill_result"] = { skill_name, skill_dir, files[], chapters, generated_at, llm_available, error } │ ▼ Frontend polls GET /api/jobs/{id} every 1.5 s status == "done" → populate Results tab, switch tab skill_status == "running" → keep polling + refresh skill files card live skill_status == "done" → render "Generated Skill Files" card │ ▼ GET /api/jobs/{id}/skill ├─ _resolve_skill_dir(): │ 1. skill_result["skill_dir"] if set and exists on disk (normal path) │ 2. scan output/<job_id>/skill/*/ for first subdir (server-restart fallback) ├─ auto-heal: skill_status "running" → "done" if .md files found on disk ├─ synthesise skill_result from disk when skill_result was null │ (llm_available = null → UI shows "🔍 recovered from disk") └─ return { skill_status, skill_result, files[] } GET /api/jobs/{id}/skill/download/… → serve individual .md files └─ also uses _resolve_skill_dir() — works without skill_result in memory --- Application Architecure and Code Excerpts !Image description !Image description Below is an excerpt from the FastAPI backend (backend/main.py) showcasing how extraction is executed asynchronously using asyncio.create_subprocess_exec while maintaining isolated per-job working directories via BOOK_SKILL_WORKDIR: """ Book-to-Skill Web Application Backend FastAPI server that wraps the book-to-skill library's extraction pipeline and triggers LLM-based skill generation after extraction completes. """ import asyncio import json import logging import os import sys import uuid from concurrent.futures import ThreadPoolExecutor from datetime import datetime, timezone from pathlib import Path from typing import List, Optional from fastapi import FastAPI, HTTPException, BackgroundTasks, UploadFile, File from fastapi.middleware.cors import CORSMiddleware from fastapi.responses import FileResponse, StreamingResponse from fastapi.staticfiles import StaticFiles from pydantic import BaseModel # ─────────────────────────────── Helpers (time) ────────────────────────────── def _now_iso() -> str: """Return current UTC time as an ISO-8601 string.""" return datetime.now(timezone.utc).isoformat() # ─────────────────────────────── Logging ───────────────────────────────────── logging.basicConfig( level=logging.INFO, format="%(asctime)s [%(levelname)s] %(name)s: %(message)s", ) logger = logging.getLogger("book_to_skill_app") # ─────────────────────────────── Config ────────────────────────────────────── BASE_DIR = Path(__file__).resolve().parent.parent INPUT_DIR = BASE_DIR / "input" OUTPUT_DIR = BASE_DIR / "output" SKILLS_DIR = Path(os.environ.get("SKILLS_DIR", Path.home() / ".agents" / "skills")) _thread_pool = ThreadPoolExecutor(max_workers=4) BOOK_TO_SKILL_REPO = BASE_DIR / "book_to_skill_repo" # File-size cap for uploads — configurable via UPLOAD_MAX_MB in .env (0 = unlimited) _upload_max_mb = int(os.environ.get("UPLOAD_MAX_MB", "100")) MAX_UPLOAD_BYTES = _upload_max_mb * 1024 * 1024 if _upload_max_mb > 0 else 0 # Supported upload extensions (mirrors what book-to-skill accepts) ALLOWED_UPLOAD_EXTENSIONS = { ".pdf", ".epub", ".docx", ".html", ".htm", ".rtf", ".mobi", ".azw", ".azw3", ".txt", ".md", ".rst", ".adoc", } # Ensure required directories exist INPUT_DIR.mkdir(exist_ok=True) OUTPUT_DIR.mkdir(exist_ok=True) # In-memory job store {job_id: JobResult} _jobs: dict[str, dict] = {} # ─────────────────────────────── Models ────────────────────────────────────── class ProcessRequest(BaseModel): file_path: str mode: str = "text" # "text" | "technical" skill_name: Optional[str] = None output_dir: Optional[str] = None generate_skill: bool = True # set False to skip LLM skill generation class DirectoryListRequest(BaseModel): directory: str = str(INPUT_DIR) class JobStatus(BaseModel): job_id: str status: str # "pending" | "running" | "done" | "error" file_path: str started_at: Optional[str] = None finished_at: Optional[str] = None stdout: str = "" stderr: str = "" extracted_text: Optional[str] = None metadata: Optional[dict] = None error: Optional[str] = None skill_status: Optional[str] = None # "pending" | "running" | "done" | "error" | None skill_result: Optional[dict] = None # SkillGenerationResult.to_dict() # ─────────────────────────────── App ───────────────────────────────────────── ... async def _run_extraction( job_id: str, file_path: Path, mode: str, skill_name: Optional[str], generate_skill: bool = True, ) -> None: _jobs[job_id]["status"] = "running" _jobs[job_id]["started_at"] = _now_iso() try: script = _resolve_script() except FileNotFoundError as exc: _jobs[job_id]["status"] = "error" _jobs[job_id]["error"] = str(exc) return job_workdir = OUTPUT_DIR / job_id job_workdir.mkdir(parents=True, exist_ok=True) cmd: List[str] = [sys.executable, str(script), str(file_path), "--mode", mode] proc = await asyncio.create_subprocess_exec( *cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, env={ **os.environ, "PYTHONIOENCODING": "utf-8", "BOOK_SKILL_WORKDIR": str(job_workdir), }, ) stdout_bytes, stderr_bytes = await asyncio.wait_for(proc.communicate(), timeout=300) # ... handles completion and triggers Phase 2 skill generation ... Furthermore, the skill generator (backend/skill_generator.py) includes a graceful fallback mechanism and model probing to ensure that even when an LLM endpoint (such as Ollama or llama.cpp) is offline, the application still outputs functional stub structures: """ Skill Generator =============== Reads extraction output (full_text.txt + metadata.json) produced by the book-to-skill extractor and calls an OpenAI-compatible LLM to generate a structured agent skill following the SKILL.md spec: output/<job_id>/skill/<skill_name>/ SKILL.md – master index + core frameworks (~4 000 tokens) glossary.md – alphabetical key-term definitions patterns.md – techniques & design patterns cheatsheet.md – decision rules & quick-reference tables chapters/ ch01-<slug>.md – per-chapter summary ... The generator degrades gracefully when no LLM is configured: it produces stub files so the application always has something to show. Environment variables (all optional — defaults work with Ollama): LLM_BASE_URL – OpenAI-compatible base URL (default: http://localhost:11434/v1) LLM_API_KEY – API key (default: "ollama" — Ollama ignores it) LLM_MODEL – Model name (default: llama3.2:latest). Must exactly match a model installed in Ollama (use `ollama list` to check). Legacy alias "llama3" was removed from Ollama; the current default is "llama3.2:latest". LLM_TIMEOUT – Per-request timeout in seconds (default: 120) """ from __future__ import annotations import json import logging import os import re import textwrap import time from dataclasses import dataclass, field from datetime import datetime, timezone from pathlib import Path from typing import List, Optional, Tuple logger = logging.getLogger("book_to_skill_app.skill_generator") # ─────────────────────────────── LLM config ─────────────────────────────────── # Values are read lazily on every call via _cfg() so that editing .env while # the server is running takes effect immediately — no restart required. from dataclasses import dataclass as _dataclass # local alias avoids shadowing @_dataclass(frozen=True) class _LLMConfig: base_url: str api_key: str model: str timeout: int def _cfg() -> _LLMConfig: """Return the current LLM configuration, re-reading .env on every call. Uses python-dotenv with override=True so values written to .env after server startup are immediately visible to the next request without a restart. The .env file is located relative to this file's project root. """ from dotenv import load_dotenv from pathlib import Path as _Path _env_path = _Path(__file__).resolve().parent.parent / ".env" load_dotenv(_env_path, override=True) return _LLMConfig( base_url = os.environ.get("LLM_BASE_URL", "http://localhost:11434/v1"), api_key = os.environ.get("LLM_API_KEY", "ollama"), model = os.environ.get("LLM_MODEL", "llama3.2:latest"), timeout = int(os.environ.get("LLM_TIMEOUT", "120")), ) # ─────────────────────────────── Result dataclass ───────────────────────────── ... def _llm_available() -> bool: """Quick probe: returns True when the LLM endpoint answers a tiny request.""" try: models_req = urllib.request.Request( f"{LLM_BASE_URL.rstrip('/')}/models", headers={"Authorization": f"Bearer {LLM_API_KEY}"}, method="GET", ) with urllib.request.urlopen(models_req, timeout=8): pass return True except Exception as exc: logger.debug("LLM server not reachable: %s", exc) return False Application's Configurations The application can provide thresholds for documents to be processed, as well as using LLMs, through an environment configuration. !Image description # ─── Book-to-Skill Web App — Environment Configuration ─────────────────────── # Copy this file to .env and fill in your values. # Host and port the FastAPI server listens on APP_HOST=0.0.0.0 APP_PORT=8080 # Upload size limit in MB (default: 100). Set to 0 for unlimited. # UPLOAD_MAX_MB=100 # Override where generated skills are stored (default: ~/.agents/skills) # SKILLS_DIR=/path/to/custom/skills # Override the path to the book_to_skill_repo clone # BOOK_TO_SKILL_REPO=/path/to/book-to-skill # Set to "development" to enable FastAPI auto-reload ENV=production # ─── LLM / Skill generation ─────────────────────────────────────────────────── # OpenAI-compatible endpoint for skill generation. # Default points to local Ollama. Change LLM_BASE_URL for llama.cpp: # LLM_BASE_URL=http://localhost:9931/v1 # Or any OpenAI-compatible service (e.g. OpenAI, Groq, Mistral). LLM_BASE_URL=http://localhost:11434/v1 LLM_API_KEY=ollama # LLM_MODEL must exactly match a model installed in Ollama (run `ollama list`). # "llama3" is a legacy alias removed in newer Ollama releases — use "llama3.2:latest". LLM_MODEL=llama3.2:latest # Per-LLM-request timeout in seconds (default: 120) # LLM_TIMEOUT=120 API Accessibility The original code could be called through command line, so the provided application provides also APIs to be call for instance throgh curl commands. !Image description --- Processing and Test Outputs !Image description When the backend processes an input document through its two-phase pipeline—combining Docling or format-specific parsers with LLM-powered generation—it builds a structured output hierarchy. Below is an example of a generated chapter summary illustrating how technical sections and core ideas are captured: # Chapter 30 — Chapter 15: Direct Connect Public Virtual Interface (p. 303) > **Note:** This file was originally generated as a stub because the LLM detection > mechanism returned `False` even though Ollama was running. The root cause has > been fixed (see summary below); this file has been updated to reflect that Ollama > **is** detected and available. --- ## LLM Detection Status | Field | Value | |-------|-------| | `llm_available` | **`true`** | | LLM endpoint | `http://localhost:11434/v1` | | Model detected | `llama3.2:latest` | | Detection method | Two-step probe (GET /models → POST /chat/completions) | --- ## Chapter Summary **Chapter 15: Direct Connect — Public Virtual Interface** This chapter covers the AWS Direct Connect Public Virtual Interface (VIF), which provides dedicated private connectivity from on-premises infrastructure to AWS public services (S3, DynamoDB, CloudFront, etc.) without traversing the public internet. ### Key Concepts - **Public VIF** — a logical interface over a Direct Connect link that advertises AWS public IP prefixes via BGP; traffic to AWS public services exits through the Direct Connect circuit rather than over NAT/internet. - **BGP peering** — each VIF requires a BGP session; the customer advertises their public IP prefixes and AWS advertises all AWS public service CIDRs. - **Route filtering** — AWS enforces strict prefix filters; only globally routable (non-RFC 1918) customer prefixes are accepted. - **Multi-region reach** — a single Public VIF can reach AWS public endpoints in **all AWS regions** (not just the Direct Connect location region). - **Hosted vs. dedicated** — Public VIFs can be provisioned on both dedicated (1 Gbps / 10 Gbps) and hosted (sub-1 Gbps) Direct Connect connections. ### Design Patterns | Pattern | When to use | |---------|-------------| | Single Public VIF | Simplest; adequate for non-critical workloads | | Redundant Public VIFs (two DX locations) | HA + failover for production | | Public VIF + Site-to-Site VPN fallback | Cost-effective HA when second DX is too expensive | | LAG (Link Aggregation Group) | Increase bandwidth + redundancy on a single DX location | ### Common Exam Traps 1. **Public VIF ≠ internet access** — traffic stays on the AWS backbone; it does not give clients internet egress. 2. **Public VIF ≠ Private VIF** — Private VIF reaches VPCs; Public VIF reaches AWS public service endpoints. 3. **BGP AS path** — Direct Connect uses BGP; you must have a public ASN (or use the 64512–65534 private ASN range in specific hosted scenarios). 4. **Prefix advertisement** — customers must advertise at least one public IP block; you cannot advertise `0.0.0.0/0` on a Public VIF. Pipeline Verification and LLM Probing !Image description To ensure robust generation, the system features a two-step probe (_llm_available()) that verifies the endpoint via GET /models and checks the model via POST /chat/completions. For instance, verifying technical documentation like AWS Direct Connect chapters confirms correct model response and artifact generation: - LLM Detection Status: Verified active (true) with llama3.2:latest connected via http://localhost:11434/v1. - Test Suite Results: Execution of the pytest suite confirms pipeline integrity across modular extractors and fallback handlers: 43 passed, 1 warning in 64.44s[cite: 13] --- Conclusion Through this implementation, we successfully realized a complete, local-first web UI and orchestration layer for book-to-skill. Key achievements include: - Advanced Document Extraction: Leveraging Docling to parse technical PDFs accurately without losing code formatting, tables, or structural hierarchies. - Asynchronous & Non-Blocking Architecture: Utilizing FastAPI, background tasks, and thread pools to handle heavy parsing and LLM inference tasks smoothly. - Resilient State Management: Implementing disk-scan fallbacks and auto-healing job statuses so generated skills remain accessible even across server restarts. - Actionable Agent Integration: Translating static book pages into structured, on-demand agent skills (SKILL.md, glossary.md, patterns.md, cheatsheet.md, and chapter files) that drastically cut down token overhead during development workflows. Thanks for reading 🤹 Links - Book-to-Skill repository: virgiliojr94/book-to-skill - Sponsor of the repository: sponsors/virgiliojr94 - Code repository for this post: aairom/book-to-skill-test

[pdf책 내용을 ai로 활용할 수 있는 skill로 만들기] 1. 설치 `npx skills add virgiliojr94/book-to-skill` 또는 Claude Code라면 `git clone virgiliojr94/book-to-skill ~/.claude/skills/book-to-skill` 한 줄로 끝.[](<virgiliojr94/book-to-skill>) 2. 변환 에이전트에서 `/book-to-skill ./내책.pdf 스킬이름` 입력하면 프레임워크·챕터별 파일·판단 기준을 담은 스킬로 자동 정리.[](<github.com/virgiliojr94/book-to-skill/blob/master…>) 3. 사용 방법 이후 `/스킬이름 키워드` 또는 `/스킬이름 ch05`만 치면 해당 챕터만 읽어 실제 책 내용으로 답하고 토큰은 24~51배 절약.[](<github.com/virgiliojr94/book-to-skill/blob/master…>)

@dannypostma Here an AI skill to transform any books/docs into AI skills for your agent: virgiliojr94/book-to-skill

Bro, holy shit... turn any technical book PDF into a Claude Code skill. book-to-skill compiles books for your coding AI: Extracts rules, code & anti-patterns Uses 24x to 51x fewer tokens than PDFs Supports PDF, EPUB, DOCX & docs Repo: virgiliojr94/book-to-skill t.co

PDF-книга в скилл для агента #shorts book-to-skill: техническая книга как скилл для Claude Code Купил книгу, прочитал, а через месяц не помнишь, что было в седьмой главе. Поиск по PDF выдаёт страницы, а не ответы. Агент без контекста галлюцинирует. book-to-skill прогоняет PDF через конвертер и собирает структурированный скилл: фреймворки, правила решений, антипаттерны и отдельные файлы по главам. Дальше агент подгружает нужную главу по запросу и отвечает по реальному тексту. ⚡ Главные фичи: • 24x-51x меньше токенов, чем заливать книгу в контекст целиком • Единый формат SKILL.md для Copilot CLI, Amp, Claude Code, Codex, OpenCode • Работает не только с книгой: папка, glob, набор источников 🛠 Быстрый запуск: git clone virgiliojr94/book-to-skill ~/.claude/skills/book-to-skill 💡 Где пригодится: держишь под рукой книгу по системному дизайну и спрашиваешь агента про паттерны репликации прямо в редакторе, без переключения на PDF. 🔗 Официальный репозиторий #софт #opensource 👉 Telegram: @the_sintez #shorts #нейросети #ai #софт #технологии

技術書を読んでも仕事に活かせていない人へ。 無料OSS「book-to-skill」が話題。 手元の技術書（PDF等）をClaude Code等のSkill（AIへの手順書）に変換。 章ごとに構造化し必要な部分だけ読み込むためトークンを大幅節約。 積読消化に最適。保存推奨。 virgiliojr94/book-to-skill