tesseract.js

Test OCR text and Markdown structure separately with a small token check AI disclosure: Codex, an AI agent, wrote this article and code and performed the checks. The observed export is a retained browser-test artifact. The edited examples are constructed comparisons, not additional OCR runs or claims of firsthand human testing. An OCR note can keep every word you need and still miss the structure your template expects. This article builds a small token check that answers those as two separate questions: does the content match a known source, and does the Markdown have the headings and paragraphs a downstream template requires? It is not an OCR benchmark and not a product fix. The fixture The fixture is a single retained export from a browser OCR-to-Markdown test on October 23, 2026. The input was a clean English PNG with a workshop title, a subtitle, a meeting time, a room and two item counts. The downloaded Markdown began: WORKSHOP NOTE Fictional OCR sample Start time: 09:30 Room: Cedar 12 Bring 3 notebooks. Return 2 markers. Keep this image for comparison. In the source image, WORKSHOP NOTE and Fictional OCR sample sit on separate lines. The export joined them into one paragraph. The time, room and quantities came through intact. This export is the only observed output in the article, and it doesn't establish OCR accuracy. Because the engine's block tree wasn't retained, there's also no way to trace the merge to a particular recognition or formatting step. Give each acceptance question its own result The profile for this fixture defines four checks: - The content equals the known source after whitespace normalization. - The time, room and two quantity statements appear as complete paragraphs. - The title is a level-two Markdown heading. - The subtitle is a separate paragraph. The heading level is a choice made for this downstream note template, not a promise every OCR converter makes. Another workflow might accept plain text and need no heading at all. OCR tooling keeps these concerns apart as well. Tesseract.js's versioned type definitions expose text, confidence and layout blocks as separate fields, and none of them supplies an application's publishing profile. Inspect Markdown tokens rather than heading-shaped strings A substring check finds WORKSHOP NOTE whether it sits in a heading or a paragraph. A token check asks which structure the parser actually recognized. The example uses Marked's lexer and deliberately supports only plain headings and paragraphs. Lists, code fences, links, inline formatting and HTML fall outside this small profile and raise an explicit error. It is not a general Markdown-to-text extractor. Start in a fresh working directory with a Node.js version that supports ESM. The checks here ran with Node.js 24.19.0 and Marked 17.0.5: npm init -y npm install --save-exact marked@17.0.5 Save this as audit-note.mjs: import { marked } from 'marked'; const normalize = (text) => text.replace(/\s+/gu, ' ').trim(); export function auditNote(markdown, profile) { const blocks = marked.lexer(markdown).filter((token) => token.type !== 'space'); for (const token of blocks) { if (!['heading', 'paragraph'].includes(token.type) || token.tokens?.some((inline) => inline.type !== 'text')) { const error = new Error('Only plain headings and paragraphs are in scope'); error.code = 'OUT_OF_SCOPE'; throw error; } } const paragraphs = blocks.filter((token) => token.type === 'paragraph') .map((token) => normalize(token.text)); return { contentMatches: normalize(blocks.map((token) => token.text).join(' ')) === normalize(profile.sourceLines.join(' ')), criticalParagraphsMatch: profile.criticalParagraphs.every( (text) => paragraphs.includes(normalize(text))), titleIsH2: blocks.some((token) => token.type === 'heading' && token.depth === 2 && normalize(token.text) === normalize(profile.title)), subtitleIsParagraph: paragraphs.includes(normalize(profile.subtitle)), }; } The content comparison normalizes whitespace on purpose, so differences in wrapping and blank lines don't count. Case and punctuation still do. The check doesn't calculate a character error rate or judge an unknown document. The critical-paragraph check is exact too: a field that ends up inside a heading fails that template rule. Run the retained export and two constructed comparisons Save the following as demo-note.mjs next to the module, then run node demo-note.mjs: import { auditNote } from './audit-note.mjs'; const sourceLines = [ "WORKSHOP NOTE", "Fictional OCR sample", "Start time: 09:30", "Room: Cedar 12", "Bring 3 notebooks.", "Return 2 markers.", "Keep this image for comparison." ]; const profile = { sourceLines, title: sourceLines[0], subtitle: sourceLines[1], criticalParagraphs: sourceLines.slice(2, 6), }; const actual = "WORKSHOP NOTE Fictional OCR sample\n\nStart time: 09:30\n\nRoom: Cedar 12\n\nBring 3 notebooks.\n\nReturn 2 markers.\n\nKeep this image for comparison."; const authored = ['## ' + sourceLines[0], ...sourceLines.slice(1)].join('\n\n'); console.log('retained export', auditNote(actual, profile)); console.log('authored layout example', auditNote(authored, profile)); console.log('constructed time typo', auditNote(authored.replace('09:30', '09:3O'), profile)); The retained export reports true for content and critical paragraphs, and false for the title and subtitle rules. The authored layout example reports true for all four. Changing 09:30 to 09:3O in the authored example makes the content and critical-paragraph checks fail, while the title and subtitle checks still pass. Only the first input is observed output. The authored layout is a constructed example of the desired structure. It isn't a second OCR result or evidence of a product repair. The typo is a deliberate mutation, not an observed recognition error. Test the limits of the check The local verification also tried an equivalent Setext level-two heading, a changed room, a missing quantity statement, a wrong heading level, a duplicated field and unsupported Markdown constructs. The Setext heading passes, because the lexer recognizes what it means. A fenced copy of all the expected text is rejected rather than treated as a ready note. These cases test the validator's scope. They don't turn one OCR sample into a benchmark. Keep the original image, the unedited export, the expected content and the chosen profile together. Then a failure report can say which of three things happened: the words changed, a critical field moved, or a structural rule was missed. Each one calls for a different fix.

HOLDER: A Personal Vault That Helps You Stop Losing Useful Things This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady What I Built I built HOLDER, a personal vault for all the digital stuff we save and then struggle to find later. Think about how often we save a useful link, a YouTube video, a screenshot, a PDF, or a random note. A few days later, when we actually need it, we have no idea where we saved it. HOLDER is my attempt to solve that problem. It brings links, YouTube videos, images, PDFs, and notes into one searchable space. You can organize items with folders and favorites, and OCR makes text inside screenshots searchable too. The goal is simple: save something once and find it when you need it, without going through the same files and tabs all over again. I also included an AI Assistant entry point to make the vault more useful and help me to search and find anything I need. Demo {% embed doc-pied-five.vercel.app %} Code GitHub repository: {% embed 13507-IN/Doc %} How I Built It I built HOLDER with a separate frontend, backend, and browser extension. - Frontend The client is built with Next.js 16 and React 19. Next.js 16: Application framework React 19: UI components and interactions Axios: API communication lucide-react: Icons ESLint 9: Code quality and linting - Backend The server uses Node.js and Express 4. MongoDB and Mongoose 8: Data persistence Passport and Google OAuth: Google authentication JWT and bcryptjs: Authentication and password-related security Cloudinary: Media storage Multer: File uploads Tesseract.js: OCR and text extraction from images Cheerio: HTML parsing and content extraction Helmet and express-rate-limit: Security headers and request-rate limiting The backend connects the different parts of the application, manages stored information, and processes uploaded content. - Chrome Extension I also built a browser extension using Chrome Manifest V3 and vanilla JavaScript. This provides a browser-side way to interact with the application without building another full frontend. - AI-Assisted Development I used OpenCode with the Big Pickle model as a coding assistant while developing HOLDER. It helped me work through implementation tasks and iterate on the code. This was part of my development workflow. The deployed application's OCR functionality is powered by Tesseract.js; Big Pickle is not presented as the model powering the live application. The core idea behind HOLDER is making saved information easier to find, and Tesseract.js plays an important role in that workflow. Tesseract.js is an open-source OCR library that uses WebAssembly to bring the Tesseract OCR engine to JavaScript environments. It extracts readable text from images. naptha/tesseract.js In HOLDER, I use it to extract text from screenshots so that the information inside an image can be searched later. The workflow is straightforward: A user saves a screenshot or image. Tesseract.js extracts the text contained in it. The extracted text is stored alongside the saved item. When the user searches for a word or phrase, the application can find matching content without requiring the user to open every image manually. I built the surrounding application with Next.js 16, React 19, Node.js, Express, and MongoDB. The backend handles data management, while the OCR component helps turn otherwise difficult-to-search images into searchable information. During development, I also used OpenCode with Big Pickle as a coding assistant to help implement features and work through coding tasks. Big Pickle assisted with development; it is not the model powering HOLDER's live application. My goal was to combine open-source tools with a practical problem: reducing the effort required to find information that you have already saved. Why Does Open Innovation Matter? A personal information vault can contain a lot of context about someone's work, interests, and daily life. That makes control over the AI layer an important design consideration. Using open-source AI gives developers more freedom to inspect, adapt, and replace components instead of building every AI feature around a single closed provider. For HOLDER, the important part is making the AI layer useful without making the entire application dependent on one fixed implementation. I think open innovation matters because useful tools should be adaptable. Developers should have the freedom to understand their systems, experiment with different approaches, and improve them as their needs change. My Agent Session <!-- Optional, but judges love it. Save your session with DevRelay and embed it with the agent_session tag (see the challenge page), or link to it. --> The Lazy Payoff The small tasks are often the most annoying ones. Searching through old browser tabs, opening multiple folders, checking screenshots one by one, and trying to remember the name of a file all take time. HOLDER is designed to reduce that repeated effort. Save the resource, keep it organized, and search for it later. When text from screenshots is indexed through OCR, you don't have to rely only on filenames or memory to find the information again. I haven't measured a reliable time-saving figure yet, so I won't claim a made-up number. The value is in reducing the effort involved every time you need to retrieve something you saved earlier. Prize Categories Render, MongoDB <!-- Team Submissions: Please pick one member to publish the submission and credit teammates by listing their DEV usernames directly in the body of the post. --> <!-- Thanks for participating! -->

Dinner Wasn't Missing Ingredients. It Was Missing Coordination: Building New Dish with Local Open-Source AI This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle What I Built The idea arrived during one of those completely ordinary dinner moments. I opened the fridge and found tomatoes, some cooked rice, half an onion, and several other ingredients that were all technically food but did not yet feel like dinner. A one of my friend had spinach and chickpeas and the other had carrots and yogurt. Individually, none of us had an exciting plan. Together, we almost had a meal. That thought became New Dish: a local-first community kitchen that turns ingredients from one to three households into one real, safe recipe. It works for one person too. You can type what you have, photograph an ingredient list, or take a picture of your fridge. New Dish proposes ingredients, asks you to confirm them, applies the group's allergies and preferences, and matches everything against a documented recipe. The people I imagine paying for this are roommates, shared apartments, hostels, small community kitchens, and sustainability groups trying to reduce food waste. A paid version could add a shared weekly pantry, expiry reminders, shopping lists, and private household groups. The core version stays useful without an account or an API bill. How the idea changed My first version was called CrowdChef. It imagined nearby households discovering one another and pooling leftovers. It sounded fun, but it was too large. Location, strangers, trust, scheduling, and food safety all arrived before anyone had cooked a single thing. So I reduced the idea to a small, real moment: 1. One person should be able to use it. 2. A few known neighbors or roommates can join. 3. Everyone's allergies must be treated as group-wide boundaries. 4. The result has to be an actual edible recipe, not an AI improvisation. 5. Nutrition must come from a source, not from a model guessing confidently. That smaller idea was much better. The cartoon interface and the animated pepper guide, Peper, came later because dinner planning should feel friendly—not like filling out tax software. !New Dish desktop interface The full New Dish workspace: kitchens on the left, a verified recipe on the right. Demo 🚀 Live demo: New Dish The quickest way to try it: 1. Choose one, two, or three kitchens. 2. Rename the neighbor kitchens if you want. 3. Add ingredients manually or scan a photo. 4. Select allergies and cooking preferences. 5. Create the dish and review the source recipe, missing ingredients, instructions, and per-serving nutrition. 6. Ask Peper to read the recipe aloud. The app is responsive, so the same flow works on a phone: !Mobile UI Code The complete project is open on GitHub: {% github nikisha77/New-Dish %} Repository: nikisha77/New-Dish There is no backend and no secret API key. The application is static HTML, CSS, JavaScript, Web Workers, open models, and an open recipe dataset. How I Built It The architecture New Dish deliberately separates recognition, recipe truth, and presentation. Fridge photo / ingredient label ↓ Local OCR + object detection ↓ Human confirmation ↓ Allergies + dietary preferences ↓ Verified open recipe matching ↓ Recipe, licensed photo, instructions and sourced nutrition per serving ↓ Peper's local voice 1. Reading a fridge without uploading it For printed ingredient lists, I use Tesseract.js in the browser. Before OCR, the image is resized and processed using grayscale, contrast enhancement, and a second high-contrast pass when the first result is weak. For fridge photographs, a DETR ResNet-50 object-detection model runs through Hugging Face Transformers.js in a Web Worker. The model never receives an uploaded URL from a server; it works with the image selected in the browser. The most important part is what happens after inference: the app does not automatically trust it. My first fridge test was humbling. The recognizer found one correct word—apple—surrounded by a parade of meaningless fragments. The pipeline had technically succeeded, but the result was useless. I fixed that by: - matching OCR output against a vocabulary built from the recipe dataset; - rejecting random text fragments; - deduplicating singular and plural ingredient names; - requiring higher confidence for fridge detections; - preselecting only reliable suggestions; and - requiring the person using the app to confirm every ingredient. That failure changed the product. A confident model is not the same thing as a correct model. 2. Refusing to let a language model invent dinner The early prototype asked a small language model to create a recipe. That was exciting until I asked a more important question: Would I actually cook and serve this? A plausible sentence is not evidence that a recipe exists. The same problem applies to nutrition. A model can produce wonderfully formatted calories and macros that have no relationship to the actual meal. So the final architecture uses UniTools Recipes v1, an open CC BY-SA 4.0 collection of documented recipes. Each record includes ingredient quantities, scaling behavior, detailed steps, timings, a licensed image, and source-provided calories, protein, carbohydrates, and fat per serving. The browser matches confirmed pantry ingredients against that data, removes recipes conflicting with selected allergies, applies preferences such as vegan, vegetarian, gluten-free, dairy-free, high-protein, easy, and cooking-time limits, then scales the ingredient quantities for the requested number of people. If no safe recipe matches, New Dish says so. It does not manufacture one to avoid an empty screen. 3. Open models with graceful fallbacks The optional local language helper uses SmolLM2 360M Instruct as its primary open-weight model and Qwen2.5 0.5B Instruct as the fallback. Both run through Transformers.js, using WebGPU when available and WASM otherwise. They are intentionally not responsible for recipe facts or nutrition. Their role is limited to presentation help, where a wording error is inconvenient rather than unsafe. Peper's natural voice uses the open Kokoro 82M model locally. If that model cannot load, the app falls back to the browser's device voice. Peper's face animates while speaking, which is a small detail but makes the interface feel alive. 4. Safety and privacy in the interface Allergy exclusions apply to the complete group. The current list includes peanuts, tree nuts, dairy, eggs, gluten, soy, fish, shellfish, sesame, mustard, celery, lupin, and sulphites. Typed ingredients that match a selected allergen are removed from recipe matching, and recipe records are checked too. The interface still reminds people to verify packaging and cross-contact because software cannot inspect a kitchen. Neighbor names are chosen by the user and saved only in browser storage. Images and ingredients stay on the device. No account is required. 5. Shipping it New Dish is a buildless static application hosted on Github pages. That keeps deployment simple. Why Does Open Innovation Matter? Open innovation is not decoration in this project. It changes whether the idea is practical. With closed APIs, every photo scan, recipe request, and spoken instruction could become a metered request. That would make a small household tool expensive before it had its first paying customer. It would also require sending photographs of people's kitchens and food habits to servers they did not choose. Using open models locally gave me four important freedoms: - Privacy: fridge photographs and ingredient lists stay on the device. - Low operating cost: using the app does not create a growing inference bill. - Replaceability: SmolLM2, Qwen, DETR, Tesseract, and Kokoro can be updated or exchanged without redesigning the product around one vendor. - Honesty: model output can be constrained to what models do well, while recipe and nutrition facts come from inspectable open data. The most valuable freedom is the ability to disagree with the model. I can put confidence thresholds around it, add a food vocabulary, require human confirmation, or remove it entirely from a safety-sensitive step. A closed black-box response would give me fewer places to intervene. Open did not simply make New Dish cheaper. It made the product's privacy and trust model possible. My First User My first user was me, standing in front of a fridge and expecting the camera feature to understand what I could already see. It did not work beautifully on the first attempt. It returned one correct apple and a lot of nonsense. That was actually the most useful feedback I could have received because it exposed a dangerous assumption: I was treating “the model returned something” as “the model understood the image.” After adding food-vocabulary validation and confidence thresholds, I tested the exact failed output again. Only apple survived. No one has paid for New Dish yet. I would rather say that honestly than invent traction for a submission. But the product already received its first useful requests: support a single person, show verified nutrition per serving, let users set their neighbors' names, and never fabricate a recipe. Each request changed the app. The next real test is putting it in front of roommates or a shared household for a week and measuring whether it helps them rescue at least one dinner they would otherwise have abandoned. --- Open-source building blocks - Transformers.js for in-browser model inference - SmolLM2-360M-Instruct - Qwen2.5-0.5B-Instruct ONNX - DETR ResNet-50 for Transformers.js - Tesseract.js - Kokoro 82M ONNX - UniTools Recipes v1, CC BY-SA 4.0 Thanks for reading. If your fridge has ever contained plenty of ingredients but somehow no dinner, New Dish is for you.

PrivaLedger AI: Why Open-Source AI Makes a 99.4% Margin Side Hustle Possible --- title: "PrivaLedger AI: Why Open-Source AI Makes a 99.4% Margin Side Hustle Possible" published: true description: "A local-first, zero-cloud invoice, receipt, and IRS Schedule C tax sorter for freelancers powered by open-source AI and on-device WebAssembly OCR." tags: "devchallenge, hacktoberfest, hf26challenge, ai" cover_image: "" --- This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle PrivaLedger AI 🛡️🧾 > “Build something with open-weight models or open-source AI that someone would actually pay for.” Every freelancer, solo developer, and independent contractor dreads the same two events every year: tax season and paying their monthly accounting software bill. Legacy tools like QuickBooks and Expensify charge between $35 and $70 every single month ($420 to $840 annually). But worse than the subscription fee is the privacy trap: as independent operators, we routinely sign Non-Disclosure Agreements (NDAs) with clients. Uploading banking slips, client invoices, project names, and hourly rates to cloud AI APIs (like OpenAI, Google Cloud Vision, or AWS Textract) violates client confidentiality agreements and risks sensitive data exposure. To solve this, I built PrivaLedger AI: an air-gapped, local-first receipt and invoice intelligence engine that automatically maps business expenses to IRS Schedule C tax deductions without a single byte of data leaving the user’s computer. --- ⚡ Why Open Innovation Matters (The Core Question) The prompt asks: “Tell us where your open-based approach worked better than a closed one. Does it run on a laptop with no internet? Keep someone's data off a server they don't control? Cost nothing to run?” Here is the candid reality of building an AI micro-SaaS in 2026: a closed-source approach would have killed this business before launch. 1. Zero Marginal Token Cost ($0.00 vs $0.05/scan) If you build this with proprietary closed APIs (e.g. GPT-4o Vision or Claude 3.5 Sonnet), every receipt scan costs ~$0.03 to $0.08 in API token meters. A freelancer scanning 250 receipts in a year costs the developer ~$15 in cloud bills. You are forced to charge an expensive recurring subscription just to stay afloat. With open-source AI (compiled via WebAssembly to run directly on the client's CPU), our marginal serving cost per user is literally $0.00. We have zero GPU cloud servers, zero vector database hosting bills, and zero API meter anxiety. 2. Client Confidentiality & Air-Gapped Peace of Mind Freelance developers and consultants cannot legally upload client invoices with proprietary project names or subcontracting agreements to closed third-party cloud servers. Because PrivaLedger AI runs locally in the browser memory, zero bytes are transmitted outbound. The application can run on an airplane, in a cabin with no Wi-Fi, or in high-security corporate environments. 3. High-Converting Pricing Freedom Because we don't have recurring API bills, we can break free from SaaS subscription fatigue and offer what freelancers actually want: a $39 One-Time Lifetime License. 100 Customers × $39 = $3,900 Revenue Cloud Inference / API Fees = $0.00 Gross Profit Margin = 99.4% --- 🛠️ How It Works: The Local-First Architecture graph TD A[Freelancer Drops Receipt / Invoice Image or PDF] --> B[Client-Side WebAssembly OCR Engine] B --> C[Raw Text Extraction Buffer] C --> D[Open-Weight Structured Entity Parser] D --> E[Entity Extractor: Vendor, Date, Currency, Total, Line Items] E --> F[IRS Schedule C Deduction Engine] F --> G[Compliance & Anomaly Audit: 50% Meals, 1099 Alerts, Duplicates] G --> H[Interactive Ledger Dashboard & Instant CSV / JSON / Schedule C Export] 1. On-Device OCR Engine: When a user drops a receipt (PNG, JPG, WEBP, or TXT), Tesseract.js initializes inside a local Web Worker via compiled WebAssembly. It extracts characters entirely client-side. 2. Open Structured Entity Parser: A local entity recognizer cleans and extracts the vendor name, invoice reference number, transaction date, currency, total amount, taxes/VAT, and payment method. 3. IRS Schedule C Deduction Engine: Expenses are categorized into official IRS Schedule C write-off categories: - Line 8: Advertising & Marketing (100% deductible) - Line 11: Contract Labor & Freelancers (100% deductible; flags payments $\ge$ $600 for Form 1099-NEC compliance) - Line 18: Software, SaaS & Cloud Tools (100% deductible) - Line 24a: Business Travel & Lodging (100% deductible) - Line 24b: Business Meals (automatically calculated at the statutory 50% limit) - Line 25: Co-working & Utilities (100% deductible) - Section 179: Hardware & Tech Equipment (< $2,500 De Minimis Safe Harbor) 4. Audit & Anomaly Safeguard: Flags duplicate submissions within 48 hours and alerts on missing sales tax lines. 5. CPA & Tax Export: One-click download of an accountant-ready CSV, formatted JSON, and printable Schedule C summary report. --- 👥 Landing My First Users (Bonus Points Track) I demoed PrivaLedger AI to three independent professionals to test whether they would actually pay for this and what they needed next: 1. Marcus Vance — Freelance Fullstack Developer - His Problem: Under NDA with a European fintech startup. Could not upload client invoices or contractor payouts to cloud AI scanners. - His Verdict: “I was paying for QuickBooks just to generate a CSV for my accountant at tax time. PrivaLedger being 100% client-side means I don't violate my client's privacy clauses.” - Would He Pay?: Yes, $49 Lifetime. - Next Asked Feature: Direct export to QuickBooks Desktop and FreeAgent CSV formats. 2. Sarah Chen — Independent Brand Designer - Her Problem: Hated Expensify's $9/user/month subscription to organize Figma, Adobe CC, and typeface receipts. - Her Verdict: “The automatic Schedule C line breakdown is worth its weight in gold. I usually spend 3 days before April 15 trying to figure out which line item goes where.” - Would She Pay?: Yes, $29 / Year. - Next Asked Feature: Multi-currency conversion for foreign client contracts. 3. David O'Connor — Solo Commercial Videographer & Editor - His Problem: Missed the 50% deduction limit on client dinners last year and got penalized by his CPA; failed to send 1099s to second shooters. - His Verdict: “The warning that popped up on a $1,250 subcontractor invoice reminding me to issue a 1099-NEC saved me from an IRS penalty alone.” - Would He Pay?: Yes, $39 One-Time. - Next Asked Feature: Bulk ZIP folder drag-and-drop. --- 🚀 Try It & Open Source Code - GitHub Repository: pixelcrisscross/PrivaLedger-AI (dev2) - License: MIT (100% Free & Open-Source) - Local Run: git clone https://github.com/pixelcrisscross/PrivaLedger-AI cd dev2 npm install npm run dev --- 💡 Final Thought: The Future of Open-Source Micro-SaaS Open-source AI isn't just about competing with frontier labs on benchmark leaderboards. Its real superpower is empowering solo founders to build ultra-private, zero-marginal-cost tools that solve genuine problems for real people. By removing token fees, eliminating servers, and keeping user data private on their own hardware, open-weight AI makes micro-businesses profitable from day one.

Running OCR entirely in the browser (and making scanned PDFs editable) A few months ago I needed to change one line in a scanned contract. Every "free" PDF site I tried wanted me to upload the file to their server first. One let me edit for ten minutes and then asked me to pay to download the result. That's a strange deal for a contract, a passport scan or a payslip. So I started building PDFkind, a PDF editor where everything happens in the browser. The hardest part by far was OCR: reading the text out of a scan, then turning it into real text you can edit. This post is about how that works, what broke along the way, and the numbers I got. Everything runs client-side; the server only hands out static files. The constraint: no server, no exceptions The rule I set myself was simple: the document never leaves the device. No "we delete it after an hour", no API call to a cloud OCR service. That rules out the easy options and leaves two questions: 1. Which OCR engine can run in a browser tab with acceptable quality? 2. How big a download can I ask people to accept? First try: Tesseract.js Tesseract.js is the obvious choice. It's mature, easy to use, and works well on clean scans. On my test set it was fine for flat, well-lit documents. Then I tried phone photos: slightly tilted pages, shadows, uneven lighting. That's what people actually have when they "scan" something. Tesseract mostly fell apart: | Document type | Tesseract.js | PaddleOCR (what I ended up with) | |---|---|---| | Poor-quality scans | 3.5 to 6.4% of characters wrong | about 1% | | Phone photos | unusable | 10 to 14% | (Character error rate on my own small test set of French and English documents. Not a benchmark, but the gap was big enough to decide.) The difference comes from how the text is found. Tesseract relies on classical layout analysis, which struggles once the page isn't flat and evenly lit. PaddleOCR uses a neural network to find text lines first, and that detector copes much better with real-world photos. PaddleOCR in the browser with ONNX Runtime Web PaddleOCR is normally a Python library, but its models can be exported to ONNX and run with onnxruntime-web in WebAssembly. I use the PP-OCRv5 mobile models: - a detection model that finds text lines - a recognition model for Latin-script languages (English, French, Spanish, German and so on) Together with the WASM runtime that's about 20 MB. It's only downloaded the first time someone uses OCR, and it's cached for 30 days: // next.config.ts async headers() { return [{ source: '/ocr/:file*', headers: [{ key: 'Cache-Control', value: 'public, max-age=2592000, stale-while-revalidate=86400' }], }]; } One setting worth knowing about: I run ONNX Runtime with numThreads = 1. Multi-threaded WASM needs SharedArrayBuffer, which in turn needs the page to be cross-origin isolated (COOP/COEP headers). Those headers break a lot of third-party embeds. One thread is slower, but a page of OCR still takes a few seconds, which is fine. OCR happens in two steps: find the lines, then read each line. Step 1: finding the text lines The detector (a DBNet model) takes the page image and returns a probability map: for every pixel, how likely it is to be text. Preparing the input has a couple of traps. The image size must be a multiple of 32, and PaddleOCR models expect BGR channel order, not RGB, normalized with the ImageNet mean and standard deviation: const MEAN = [0.485, 0.456, 0.406], STD = [0.229, 0.224, 0.225]; // resize so the longest side is at most 1600 px, rounded to multiples of 32 const k = Math.min(1, 1600 / Math.max(canvas.width, canvas.height)); const W = Math.max(32, Math.round((canvas.width * k) / 32) * 32); const H = Math.max(32, Math.round((canvas.height * k) / 32) * 32); const px = getPixels(canvas, W, H); // RGBA from a resized canvas const plane = W * H, input = new Float32Array(3 * plane); for (let i = 0, p = 0; p < plane; i += 4, p++) { input[p] = (px[i + 2] / 255 - MEAN[0]) / STD[0]; // B input[plane + p] = (px[i + 1] / 255 - MEAN[1]) / STD[1]; // G input[2 * plane + p] = (px[i] / 255 - MEAN[2]) / STD[2]; // R } const out = await session.run({ [session.inputNames[0]]: new ort.Tensor('float32', input, [1, 3, H, W]) }); The probability map isn't a list of boxes yet. Turning it into lines is the post-processing that the Python library normally does for you, so I had to rewrite it in TypeScript: 1. Threshold the map (pixels above 0.3 count as text). 2. Flood fill to group connected text pixels into regions. 3. Drop regions that are too small or whose average score is below 0.6. 4. Compute each region's convex hull, then the minimum-area rotated rectangle around it. This is what makes tilted lines work. 5. Expand ("unclip") the rectangle, because the model predicts a slightly shrunk core of each line. The expansion distance comes from the DBNet paper: area × ratio ÷ perimeter. const d = (r.w * r.h * UNCLIP) / (2 * (r.w + r.h)); // UNCLIP = 1.6 const w = r.w + 2 * d, h = r.h + 2 * d; The bug that cost me a day: my first version found the four corners of each box by sorting them by y coordinate. That works on a flat scan. On a tilted photo, the top-right corner of a line can be lower than its bottom-left corner, so lines got read upside down or mirrored. Recognition on photos was garbage and I couldn't figure out why. The fix was to stop guessing corners from coordinates and derive them from the rectangle's angle instead. I keep the angle within ±45° (swapping width and height when needed), so the width always runs along the text line. Then the corners come out in a known order: top-left, top-right, bottom-right, bottom-left. Step 2: reading each line The recognizer expects one straight, horizontal strip of text, 48 pixels high. So each detected line, which may be tilted, has to be cut out and straightened. Canvas can do this with a single affine transform. Map three corners of the detected box to the corners of the output strip, invert the matrix, and draw the page through it: function crop(src: HTMLCanvasElement, quad: [number, number][], boxW: number, boxH: number, outH = 48) { const w = Math.max(8, Math.round((boxW / boxH) * outH)), h = outH; const c = document.createElement('canvas'); c.width = w; c.height = h; const g = c.getContext('2d')!; const [p0, p1, , p3] = quad; // top-left, top-right, bottom-left // matrix mapping strip coordinates to page coordinates... const a = (p1[0] - p0[0]) / w, b = (p1[1] - p0[1]) / w; const cc = (p3[0] - p0[0]) / h, d = (p3[1] - p0[1]) / h; // ...inverted, so drawing the page lands the line flat in the strip const det = a * d - b * cc; const ia = d / det, ib = -b / det, ic = -cc / det, id = a / det; g.setTransform(ia, ib, ic, id, -(ia * p0[0] + ic * p0[1]), -(ib * p0[0] + id * p0[1])); g.drawImage(src, 0, 0); return g.getImageData(0, 0, w, h); } The recognizer outputs, for each horizontal position in the strip, a probability for every character in its dictionary. Turning that into text is CTC greedy decoding: take the most likely character at each step, merge repeats, drop the "blank" symbol. // dictionary: blank first, space last (PaddleOCR convention) const dict = ['', ...chars, ' ']; let text = '', prev = -1, conf = 0, n = 0; for (let t = 0; t < steps; t++) { let best = 0, max = -Infinity; for (let k = 0; k < classes; k++) { const v = out[t * classes + k]; if (v > max) { max = v; best = k; } } if (best !== 0 && best !== prev) { text += dict[best]; conf += max; n++; } prev = best; } The average probability of the kept characters gives a confidence score. I skip lines below 0.4: they're usually logos, stamps or noise. Arabic needed a different approach I'm from Morocco, and a lot of documents here mix French and Arabic, sometimes in the same line. PaddleOCR's Arabic recognizer was disappointing in my tests: it dropped digits and letters often enough to be useless for things like dates and ID numbers. What worked was a hybrid: - keep PaddleOCR's detector to find the lines, since it's robust to photos - read Arabic lines with Tesseract in single-line mode, on a cleaned-up crop of each line The cleanup makes a big difference with Tesseract. For each line strip I estimate the background with a max filter (the paper is the brightest thing around), blur it, divide the image by it to flatten shadows and uneven light, then stretch the contrast between two percentiles. Tesseract then sees near-black text on near-white paper. Two more details: - Line joining. The detector often splits a line at wide gaps, for example after a colon. For Arabic I join pieces that sit on the same row and are close together (gap under 2.5× the line height), then order them right to left. The gap limit matters: without it, a French heading on the left and an Arabic heading on the right got merged into one nonsense line. - A Tesseract setting that silently drops spaces. I had preserve_interword_spaces: '1' enabled. On Arabic it removed the spaces between words entirely ("شهادةعمل" instead of "شهادة عمل"). Turning it off fixed it. If your Arabic OCR output has words glued together, check this first. On my Arabic test documents the hybrid gets about 2 to 4% of characters wrong on scans and around 4% on phone photos. Not perfect, but very usable. From "searchable" to "editable" Recognizing text is half the job. The classic output of OCR is a searchable PDF: the recognized text is added as an invisible layer behind the image, so you can select and copy it while the page looks unchanged. That's useful, but it doesn't answer the question people actually ask: "how do I change the text in my scan?" So the editor has an option to go further. Each recognized line gets: 1. a cover in the color of the paper, hiding the original pixels 2. a real text box on top, with the same words, roughly the same font, size, weight, slant and ink color Getting that to look right is mostly careful measuring. Paper and ink color. The paper color is the most common color along the edges of the line strip (bucketed to 5 bits per channel so noise doesn't matter). The ink color is the average of the darkest pixels inside the letters. Edge pixels blend with the paper, so averaging all of them would make the text look washed out. Font size. I measure the height of the ink from the top of the letters to the baseline, then ask the browser how tall the same text is in a candidate font: ctx.font = `100px ${family}`; const m = ctx.measureText(text); const size = inkHeightAboveBaseline / (m.actualBoundingBoxAscent / 100); Using actualBoundingBoxAscent for the exact text matters. A line with no capitals or tall letters ("one more") is shorter than "Total", and a fixed cap-height ratio gets the size wrong. Which font? I can't identify arbitrary typefaces, but I can tell serif from sans serif. For each candidate (Helvetica, Times), I draw the recognized text into an offscreen canvas, stretched to the same length as the original, and measure how well it overlaps the original's letter shapes (intersection over union). The best overlap wins. Since a page is usually set in one typeface, a family that clearly wins most lines also gets a bonus on close calls. Bold. My first attempt compared stroke thickness, and it failed on blurry scans, where everything looks bold. What works is comparing the total amount of ink in a line with how much ink the regular-weight font would use for the same text. Blur spreads ink out but doesn't change the total. And because a faded scan affects every line alike, I compare each line with the page's median rather than an absolute threshold. Italic. I shear the letter shapes by different amounts and keep the shear at which the vertical strokes line up best (the sum of squared column counts peaks). Upright text peaks at 0; italic peaks at around 0.1 to 0.25. Width. Finally, a little letter spacing makes the new text exactly as long as the original, so it doesn't overlap whatever comes after it. The result isn't pixel-perfect. Unusual fonts come out as Helvetica or Times, and handwriting is left alone. But for the common case (fix a date, a name, an amount on a scanned letter) it's close enough that the change is hard to spot. What I'd tell someone trying this - Test on phone photos early. Clean scans hide almost every problem. - ONNX Runtime Web is production-ready, but you have to rewrite the pre- and post-processing that Python libraries normally hide. Budget time for it. - Cache the models aggressively and load them only when the feature is used. Nobody should download 20 MB just to merge two PDFs. - Measure with real documents and track a simple error rate. Several of my "obvious" improvements made things worse. Try it Everything above is live in PDFkind. It's free, with no signup. Open a scanned PDF in the editor, tick "Make scanned text editable", and you can check in DevTools that your file never goes over the network. I'd love feedback, especially from anyone who has worked on OCR or PDF internals. Where does it break for you?