Mentions
Conversations linking to GitHub, newest first. Times are UTC.
Mentions by source
X
Bluesky
Hacker News
V2EX
Qiita
DEV
Mastodon
Latest posts
Expenso an offline-first Android App, No Internet Hey everyone! I've been working on Expenso, a personal finance and expense management app for Android, and I'd love to get some real-world feedback from people here. One thing that bothers me about finance apps is how much personal financial information we end up sharing with online services. I wanted to build something that takes a different approach. Expenso is designed to keep your financial data on your device. There is no internet permission, no cloud dependency, no analytics tracking, and no telemetry. Your transactions and financial calculations stay local. Here's what you can do with it: - 💰 Track income, expenses, and transfers. - 📊 Manage budgets with spending limits and alerts. - 📈 Visualize cash flow, savings rates, and spending trends. - ⏰ Set reminders for bills and recurring payments. - 📄 Generate detailed PDF financial statements. - 🔐 Protect locally stored data with encrypted database storage and biometric authentication. - 💾 Back up and restore your data whenever you need it. A few technical details for fellow developers: It's built natively for Android using Java 17, Room, SQLCipher, Android Keystore, and Material Design 3. The idea is simple: personal finance management without handing over your financial data to another service. If you're an Android user, I'd really appreciate it if you could give Expenso a try: 📱 Google Play: play.google.com 📚 Project documentation: maurihimanshu/expenso-docs I'm particularly interested in hearing: 1. Does the interface and overall experience feel intuitive? 2. Are there any features missing that you'd expect from a personal finance app? 3. Would you actually prefer an offline-only finance app over one with cloud synchronization? I'm building this independently, so honest criticism is very welcome. Even a small piece of feedback could help me improve the app. Thanks for checking it out! 🙌
I built a Tower Defense MMORPG with a city of Claude agents. I'm a developer who has been building games since I was 15. I'm now 33 and working a full time consultancy and operating my own business as well as having a family. Without Claude making an MMORPG by myself would never be possible. So a few months ago when I got time over working for my clients and family I've spent that extra time on my own project Avaryn. Using Gas City I've set up a workflow that is essentially a games studio that I operate through. I've built a full UI featuring a queue of work being done that I can rate if it goes into the game or not. After the work gets landed I get to rate it one ticket at a time. The beautiful thing about gas city is the beads ticketing system that the AI agents use to follow the work, it's never lost and followed up on like an internal Jira board just for the agents. preview.redd.it Over these last few months I've been able to build a truly massive game for one person with a custom game engine, infrastructure, website, stripe integration and more which is completely enabled by Claude. Opus 5.5 was a true game changer for my workflows. Since July 12th when the first commit landed I've been able to sculpt a world, build storylines and iterate on the game I've always dreamed of building. An MMORPG that truly rewards strategy game players. After building with a single Claude session for a while I switched over to using agents from a recommendation of a close friend of mine. Gas City is great but it uses a ton of tokens. When setting it up make sure you set: - Auto compaction to reasonable values (opus 5.5, sonnet 5.5 to 250K and Haiku 5.5 to 125k). - In the prompts tell Opus and Sonnet to use Haiku 5.5 subagents where it makes sense. Also make sure to remove unneccesary roles out of your lineup. I've had one too many managers and It's cost a ton of tokens and the work got worse and worse over time. So I've wrapped gas city with my own webapp essentially enabling me to work in hyperspeed. There are still rough edges to the project but I feel a lot of it is due to the sheer scope of the task I've taken on. What's there today is still beyond anything I've been able to even imagine before and I love working on it now, the only limiting factor for me is really time. Time is what Claude just gives me more of, I can just tell it what to do and I don't have to be an expert 3d artist, 2d artist, designer and animator all at once. I've used a mix of Tripo3D and Opus 5.5 generated 3d (Tripo3D can be a lot cheaper if you cancel and get the 80% discount on the second month as well as the 50% discount on the first). I have spent a little too much on the Elevenlabs API so far but I'm hoping to get into the Elevenlabs startup program soon. 😵 As for the API spend for Claude I'm at around~ 75K$ per month on agents but that's heavily subsidized by the subscriptions obviously. So make sure to use subscriptions people, it saves so so so much on the Claude usage and nothing else comes close to what claude can produce for the same amount of spend, not even close. For someone that wants to build a similar workflow here are some takeaways from the months I've spent building Avaryn: If you're one step behind me - When starting out using Gas City make sure to include the ideas you have of how to work in the agent prompts so they never forget it. - Before you run agents in parallel, give each one its own git worktree and one task. Land through a queue that runs the tests first, or they break each other's work. - Measure your usage from the transcripts. My first heavy usage week got costly due to work nobody asked for: status reports, idle loops and investigations. - Use small helpers. Haiku with a short context is about 40x cheaper per token than Opus. - Judge art and UI from screenshots. I will be answering followup questions if anyone has any of those!
👋 Friends! ⏰Just 12 hours left to vote for your favorite developer tool at DevHunt. 87 amazing devtools are competing for the "Tool of The Week" title 🥇 Finalists : → @ArticosHQ → RedditAPIs by github.com/redditapis → Taskless by @thecodedrift → @Omnichar → optscale.ai Kompressor Your vote is important, so don't miss out! → devhunt.org
@amo_holiday Ravbug/MCSoundsExtractor 遅いけど置いておきますね…
@Isaac_Dev0 @X @Isaac_Dev0 happy to connect and looking forward for your feedback on C-Kapsalis/devils-analyst. Star if you appreciate the work!
Garmin G5 Improvements for MSFS 2024 I have added the G5 mod for MSFS 2024 to several aircraft. All credit goes to the original creator of the G5 mod (johnygomez/msfs-garmin-g5-ifr). I merely integrated it into a few additional aircraft. I have only tested it in MSFS 2024, so I cannot guarantee it works perfectly, but based on my testing, I haven't found any issues that would hinder its use. The G5 mod has been added to the following aircraft: - Cessna 152 - Cessna 172SP Classic - Diamond DA40 TDI - Cessna 185F Skywagon - Grumman HU-16 Albatross - Piper PA-28-236 Dakota - Cessna T210N - Cessna 182RG II - A36 Bonanza Professional - A36TC Bonanza Professional - B36TP Bonanza Professional - Commander 114 - Commander 114TC - B60 Duke - B60 Grand Duke - B60T Turbine Duke - Caravan Professional - Caravan Professional (Cargo Pod) - Cessna 414AW Chancellor - PA-28R Arrow III - PA-28R Turbo Arrow III - PA-28R Turbo Arrow IV - PA-28-161 Warrior II - PA-38 Tomahawk - Aeroprakt A32 Vixxen - F406 Caravan II - Black Square - Baron Professional - bagolu c172m You must already own the aircraft shown in the list. This mod simply adds the G5 to those aircraft. modsfire.com
not sure how many would be interested in this but if you want a claude-code inspired theme for omp (oh-my-pi by @can1357 ) Then here you go: crxssrazr93/claude-code-theme-omp Enjoy. t.co
MobSF v4.6.4 ya analiza .dex sueltos, justo lo que dejan packers y cargas dinámicas de código. Suma visor de tráfico con mitmproxy, JADX 1.5.6 y arregla frida-server en Android 10/11. Lo bueno empieza cuando el código no está donde parece 👀 github.com/MobSF/Mobile-Security-Framework-MobSF/…
@ariaghora github.com/arhen/agents/blob/main/skills/self-hea… Dah pakai 2 bulan ini, wonderfull. Ban yak plugin/exts yang update sendiri :)
@brucexu_eth 如果要接近Codexbar的体验,可以用OpenUsage robinebers/openusage
Custom lists script Hello everyone, I was getting tired of scrolling though hundreds and hundreds of movies and wanted easier access to lists of movies with certain criteria without manually creating lists. I Clauded for myself this script that can: - Create a playlist based on movie length (I wanted to have a view of short movies, 100 min and below in my case) - Create a playlist per nationality (currently uses audio language as reference which isn't ideal but it's possible to plug in a TMDB API key for more accurate info - Create Director collections (I go though films mainly by filmography and didn't find any view satisfactory in that regard) lbnsq182/infuse-lists This script then reruns automatically based on how you set it: infuse-lists schedule --day friday --time 23:30 # weekly, any weekday at HH:MM infuse-lists schedule --daily --time 05:00 # every day infuse-lists schedule --run-now # trigger the installed job once (test) infuse-lists unschedule # remove it Be sure to select a network share that's common on your Mac + other devices and the custom playlists/collections will show up on every devices. You can Pin to homescreen any playlist/collection for easier one click access! (This parsing is to my knowledge dependent on Infuse version and is subject to change so use it an your own risk. Make sure you run the latest version of Infuse everywhere. If anything breaks on your Mac you can always re-install Infuse. ) (I'm not sure this post is breaking any rule enforced by Firecore, if so, I will remove it)
スマートフォンを頭脳にした自律走行車を作っている。ソフトウェアは公開済み、車体は配線の途中 スマートフォンを頭脳にした自律走行車を、個人で作っています。市販のトイラジコン(ジョーゼン ダートマックス 1/16 トヨタ86 ドリフトドライブ)に Galaxy Note 10+ を載せ、部屋の地図を作りながら自律走行させる計画です。 これまで地図化の仕組みをシミュレーションで測定した記事と、車体を分解して配線を特定した記事を書きました。この記事は全体の話です。何をやっていて、現状がどうなっていて、最終的に何を目指しているのかを書きます。 結論を先に書きます。 頭脳はスマートフォン。自己位置はスマホの ARCore、距離は ToF センサーから取り、地図と経路計画は自作のコードで処理する ソフトウェア側は完成し、GitHub で公開済み。シミュレーションでは、カバレッジ 96.9%、既知セルの正解率 95.6% までいった 車体側は、分解と配線の特定が終わり、駆動モーターの線を基板から外したところ。モーター自体が生きているかの確認が次の関門 目指しているのは、スマホを頭脳として機械に差し込む形の実証。この86は、その1号機 構成 車体はトイラジコンそのものです。受信機基板を取り外し、代わりにマイコン(ESP32)とモータードライバを入れて、モーターをスマホから回せるようにします。 スマホとマイコンは USB ケーブルでつなぎます。無線にしないのは、手元の ESP32 基板に技適マークが無く、日本で電波を出す使い方ができないためです。有線でも成立するので、そのまま有線で進めています。 スマホ側のアプリは Flutter で書きました。地図化と経路計画のコードは、シミュレーションと実機で同じものを使います。 土台のシャーシ全体。中央の基板を取り外して、マイコン側に置き換える。 なぜスマホを頭脳にするのか 理由は、自己位置の推定と距離の測定が、完成品として手に入るからです。 地図化の精度は、地図の作り方より自己位置の精度で決まります。これはシミュレーションで測って分かりました。自己位置にずれが積み重なる条件では、カバレッジが 95.0% と数字上は埋まるのに、正解率は 84.0% まで落ち、到達に約9倍の移動がかかりました。 その自己位置を部品から自作すると、ちょうど崩れた部分を作り直すことになります。スマホの ARCore はカメラと IMU(加速度と角速度のセンサー)を組み合わせて自己位置を出していて、ここが完成品として使えます。難易度が最も高い部分を買って済ませ、自分のコードは地図化と判断の部分に集中する、という分け方です。 もう一つの理由は単純で、Galaxy Note 10+ が手元にあったことです。機種は当面この1台に固定して開発しています。 現状 ソフトウェア側は終わっています。地図化、経路計画、未探索の境目を次の目的地にする探索、安全に止まるための決め方まで実装し、アプリとして GitHub で公開しました。 シミュレーションで生成された地図。白が通れる場所、黒と灰色が壁と障害物、赤がロボットの位置。 車体側は途中です。分解して、基板の太い線7本の行き先は全部特定できました。前のモーターが操舵用で、後ろが駆動用です。駆動モーターの線を基板から外すところまでは進みました。 止まっているのは、その先です。 駆動モーターが生きているか、まだ確定していない。電池1本(1.5V)では弱くて判定できなかった ハンダ作業で基板側を焦がした。以後の配線はハンダを前提にしない形に変えた(ネジ端子で線を留めるモータードライバに替える。部品の購入はこれから) スマホを車体に固定する台が、まだ無い 配線の進め方については、大学でデジタルファブリケーションを専門とする先生に、実物を持ち込んで相談する予定です。 目指しているもの まず目指すのは、自宅の部屋で、地図を作りながら自律走行させることです。走るたびに走行の記録(映像と操作のログ)を貯めて、後から見返して改善できる形にします。走る、記録する、改善する、という循環を作るのが先です。 その先で証明したいのは、スマホを頭脳として機械に差し込む形が成り立つかどうかです。頭脳側(自己位置、距離、判断)をスマホに寄せれば、車体側はモーターと簡単な制御基板だけで済みます。土台の機械を取り替えても、頭脳は同じものが使えるはずだ、という仮説です。 86を土台に選んだ理由は、手元にあったことと、目立つことです。実証の1号機は、見た目で目立つ方が伝わります。 残っていること 駆動モーターの動作確認。電池ボックスに電池6本を入れて当てるのが次の一手 ネジ端子のモータードライバの購入と、残りの配線のつなぎ直し スマホを固定する台の製作 実機での距離の較正(1m の実測で合わせる) 初走行と、実機での地図化の測定 製作の記録とコードは GitHub に置いています。 fumi3333/phone-robot
OpenAIに障害発生か、、、 Codexがクソ遅い、というのをChatGPTに調べさせる、という地獄w codex#58858
I made a self-hosted Excalidraw for teams that deploys on Coolify in 3 steps I run my own instance on Coolify, so I made the deploy as short as I could: 1. New resource: Public Repository, https://github.com/maciekzygmunt/excaliself, branch main 2. Build Pack: Docker Compose, location /docker-compose.coolify.yaml 3. Deploy Coolify generates the Postgres password and the auth secret, and passes the service domain to the app as BASE_URL. Postgres stays on the internal network, and drawings live in a volume, so they survive redeploys. What you get: the official Excalidraw canvas inside a small app with accounts, folders, a shared team space, search inside drawings, and live collaboration with guest links. Free, MIT. Two Coolify things I ran into, in case they save someone time: - Auto-deploy fires on push, before CI has published the new image, so you deploy the old one. If you want every push live, build on the server instead (there's a separate compose file for that). - PR previews get the production env values by default. I run previews on an in-memory database so they can never touch the real one. Docs: excaliself.com Feedback welcome.
Github - github.com/mohdshamii
Ai made desktop environment Told claude to make this de mostly as a test of how much it could do (mostly made for fedora). Here is a link if you would like to test it (cache-byte/Dusk-desktop-environment). Wallpapers don't come with the de. Note that I didn't write a single line of code, all made with claude (kinda disappointing I know 😢🤧).
I am building Python library for reading electrochemical measurement files, need example files to add support I am using AI-assist tool to write non-plain text electrochemical measurement files (e.g. Metrohm .nox, biologic .mpr, gamry .dta, etc). It is frustrating sometime to always use there software to export as csv or xlsx so I vibecoded this python library. If your file is not supported, open an issue on github and attached example file (can be garbage data), to attached it, rename the file like this, examplle.mpr.txt. kevinsmia1939/PyElectrochemReader To install the library: pip install pyelectrochemreader Here is the simplest way to use the library. from pyelectrochemreader import read import matplotlib.pyplot as plt data = read("example/data/CV_C01.mpr")[0] plt.plot(data["voltage"], data["current"]) plt.xlabel("Voltage (V)") plt.ylabel("Current (A)") plt.show()
Discworld Noir: running on Windows, Linux & Steam Deck (unofficial ScummVM fork) I've been working on getting Discworld Noir running in ScummVM. The fork has been tested on Windows, Linux and Steam Deck, and I'd like more people to try it. You need your own copy of the English Windows version, all three CDs. No game files are included. The repository has setup instructions, source code and the Linux download: Alexbeav/discworld-noir-scummvm A portable Linux build is available to download, tested on Steam Deck and an Arch Linux (Omarchy) machine. The repository also includes the source patch; the public download described here is the Linux build. This is an unofficial fork of ScummVM's Tinsel engine. It adds the missing 3D actor movement and fixes problems with the notebook, conversations, movie audio, music, lighting and saves. Automated playthroughs using normal game inputs reached the ending during development. Those runs included changes between builds, so I can't claim a complete playthrough of the current release on one unchanged build. My manual playthrough is still in progress, and I'd like more people to try it. There are still bugs: Lewton looks too small on the Wharf, some lighting and character rendering are off, and walking can take odd routes or refuse a spot. Save manually; autosave was disabled during testing. Use the game's options for volume, as ScummVM's volume sliders don't work here. For transparency, I'm a DevSecOps engineer rather than a game programmer. AI agents wrote most of the code; I directed the work and tested the changes. This is an independent hobby project, not an official ScummVM release. ScummVM does not accept AI-written contributions, so I'm not submitting this patch upstream - but if anyone wants to base hand-written code off my results and get Discworld Noir officially supported, that's great in my book! If you still have the discs and fancy revisiting Ankh-Morpork, I'd appreciate bug reports - open an issue on the repository. Please include where it happened, what you did, and a save from just before the problem if possible.
I built a self-hosted home for Excalidraw drawings: one docker compose, app + Postgres, amd64 and arm64 I wanted my Excalidraw drawings on my own server instead of in browser storage, so I built Excaliself. It's the official Excalidraw canvas inside a small app: accounts with invite links, folders, a shared team space, search inside drawings, and live collaboration with guest links. The server side: - One docker compose file with two containers: the app and Postgres 17 - Images for amd64 and arm64, so an ARM mini PC or board works too - Drawings and images are stored in Postgres, so a backup is one pg_dump - Nothing else to run: no Redis, no S3, no separate collab server It's free and MIT licensed. Repo: maciekzygmunt/excaliself Docs: excaliself.com Happy to answer questions about the setup.
@exampIe_Iife そうなの。一応ソースは公開しているから実験してみたかったらぜひ。 mouseos/UniXNU-a04br3
Discworld Noir: running on Windows, Linux & Steam Deck (unofficial ScummVM fork) I've been working on getting Discworld Noir running in ScummVM. The fork has been tested on Windows, Linux and Steam Deck, and I'd like more people to try it. You need your own copy of the English Windows version, all three CDs. No game files are included. The repository has setup instructions, source code and the Linux download: Alexbeav/discworld-noir-scummvm A portable Linux build is available to download, tested on Steam Deck and an Arch Linux (Omarchy) machine. The repository also includes the source patch; the public download described here is the Linux build. This is an unofficial fork of ScummVM's Tinsel engine. It adds the missing 3D actor movement and fixes problems with the notebook, conversations, movie audio, music, lighting and saves. Automated playthroughs using normal game inputs reached the ending during development. Those runs included changes between builds, so I can't claim a complete playthrough of the current release on one unchanged build. My manual playthrough is still in progress, and I'd like more people to try it. There are still bugs: Lewton looks too small on the Wharf, some lighting and character rendering are off, and walking can take odd routes or refuse a spot. Save manually; autosave was disabled during testing. Use the game's options for volume, as ScummVM's volume sliders don't work here. For transparency, I'm a DevSecOps engineer rather than a game programmer. AI agents wrote most of the code; I directed the work and tested the changes. This is an independent hobby project, not an official ScummVM release. ScummVM does not accept AI-written contributions, so I'm not submitting this patch upstream - but if anyone wants to base hand-written code off my results and get Discworld Noir officially supported, that's great in my book! If you still have the discs and fancy revisiting Ankh-Morpork, I'd appreciate bug reports - open an issue on the repository. Please include where it happened, what you did, and a save from just before the problem if possible.
给 Claude Code、Codex 等 agent 提供一个 skill,把一条教学论断生成为可交互的 HTML 教学图,让学习者按「预测 → 动手改 → 解释」的循环自己操作验证。 ywEdAi/teachable-figures t.co
@wtogami Have a look at that Cedriceuh/freetoken-rdna3
`shutdown -h now` destroyed 13 of our 48 Ceph OSDs. Here is how we got the data back --- title: "shutdown -h now destroyed 13 of our 48 Ceph OSDs. Here is how we got the data back" published: true description: A planned power-down killed the RocksDB index on 13 OSDs. Every supported repair tool refused to run, because each one opens the database it is supposed to repair. This is what worked. tags: ceph, storage, incidenthandling, devops --- We powered a site down on purpose. Electricians needed the room dead for an hour to upgrade a panel and fit a UPS. We had a runbook, we followed it, every application database shut down cleanly and reported exit code 0. Then we turned the site back on and 13 of our 48 Ceph OSDs refused to start. Not one disk had failed. Every byte of object data was intact. But each of those 13 OSDs had a destroyed RocksDB index, and without it BlueStore will not mount the object store at all. At the worst point 103 placement groups were offline, our shared filesystem was degraded, and two production object stores had data we could not reach. This is the story of how we got all of it back except a single 4MB object, and why every supported repair tool in Ceph was useless to us. First, what Ceph actually is (skip this if you run it) If you already operate Ceph, jump to the next section. If you do not, none of the rest will land without this. Ceph is distributed storage. You give it a pile of disks spread across a pile of machines, and it presents them as one system that speaks three protocols at once: block devices for VMs, a POSIX filesystem you can mount, and an S3 compatible object store. Same cluster, same disks, three front doors. The pieces you need for this story: - OSD (Object Storage Daemon). One process per disk. A host that owns 21 disks runs 21 OSD processes. This is the unit of storage and the unit of failure, and how many of them sit on one host is the whole root cause. - Placement group (PG). A pool is sliced into thousands of PGs, and each PG is replicated onto a set of OSDs. Clients do not track objects, they compute a PG with a hash and then compute its OSDs with an algorithm called CRUSH. No lookup table, no metadata server in the data path. It is genuinely elegant. - Failure domain. You tell CRUSH what a "thing that can die" is: a disk, a host, a rack. CRUSH then guarantees replicas land in different ones. Pick this wrong and your replication is decorative. - MON (Monitor). A small quorum of daemons holding the maps that describe the cluster. They do not touch your data, but nothing works without them. - BlueStore. Modern OSDs do not use a filesystem. The OSD writes objects to the raw block device itself, which is faster and avoids a whole category of double-write problems. And that last one is where this incident lives. BlueStore writes your object data straight to the disk, but it has to remember where everything went: which object is at which offset, its checksums, its attributes, the PG bookkeeping. That index lives in an embedded RocksDB on the same device. The analogy that made this click for our non-storage colleagues: the disk is a book with no page numbers and no table of contents, and RocksDB is the index. The text is all still there. But with no index, you cannot find chapter 4, and BlueStore will not even open the book to let you try. It refuses to mount at all. A visual representation may help, Ceph works more or less like this: !Image description What Ceph is genuinely good at - Hardware death. Pull a disk, lose a host, and it re-replicates itself back to full redundancy with nobody paged. This works so well it is easy to forget it is happening. - Scaling sideways. Add disks, add hosts, capacity and throughput go up. No forklift migration. - One system, three protocols. Not running separate SAN, NAS and object stacks is worth real money and real headcount. - No proprietary controller. It is open source, so when it breaks in an interesting way you can read the code that is failing. That mattered enormously here. What Ceph is bad at, and you should hear this before you deploy it - It is complex, and the complexity is not optional. CRUSH rules, pool sizing, failure domains, PG counts. You will get one of these wrong at least once. - It assumes failures are independent. Its entire design premise is that disks and hosts die one at a time, at random. A single operator action that hits every node at once is the one thing the model does not cover, and that is exactly what we did. - Small clusters are fragile in a way big ones are not. With thousands of OSDs, losing 13 is a Tuesday. With 48, losing 13 takes pools offline. - The recovery tooling assumes the metadata is healthy. This is the sharp edge of the whole article. Ceph is excellent at recovering from a dead disk. It is much weaker at recovering from an intact disk with a broken index, because almost every tool that would fix the index needs to open the index first. None of this is a reason to avoid Ceph. We would deploy it again. But "self-healing" describes hardware failure, not operator failure, and we learned the difference the expensive way. Our setup, and why it is shaped like this Nothing exotic, but the shape explains the blast radius. consumers (VMs, apps, S3 clients) | storage network | +-----------+-----------+-----------+-----------+-----------+ | | | | | | sto-01 sto-02 sto-03 sto-04 mon x3 | | | | (maps only, +-----+-----+ +-----+-----+ no data) | | +-------v--------+ +-------v--------+ | ENCLOSURE A | | ENCLOSURE B | | 24 NVMe | | 24 NVMe | | dual-ported | | dual-ported | +----------------+ +----------------+ 48 OSDs total, unevenly split within each host pair real split was 3 / 21 / 5 / 19 across the four hosts CRUSH failure domain = ENCLOSURE (not host) All pools: size=2, min_size=2, one copy in each enclosure Four storage hosts, two external NVMe enclosures, 48 OSDs. The drives are dual-ported, meaning each physical disk is wired to two hosts at once, so either host of a pair can take over the OSDs if its partner dies. That is a nice property and it is also why the topology is not the obvious one. The OSDs are not spread evenly. The real split across the four hosts was 3, 21, 5 and 19. Within each pair, one machine had quietly ended up owning almost every drive in its enclosure while its partner held a handful. Nobody designed that, it accumulated. It is also the single fact that decided how bad this incident got. Two decisions here matter for what follows: The failure domain is the enclosure, not the host. This looks wrong until you remember the disks do not live in the hosts. If CRUSH thought "host" was the boundary, it could happily place both replicas of a PG on two OSDs that are physically in the same enclosure, served by two different hosts. Lose that enclosure and you lose both copies. Setting the domain to the chassis forces one replica into each enclosure, so any pool survives an entire enclosure going away. size=2, min_size=2. Two copies, and both must be present for writes to be accepted. This is a deliberate trade: it refuses to take writes it cannot fully replicate, rather than accepting them onto one copy and hoping. On a two-enclosure layout it is the honest setting. Now the arithmetic that hurt. A PG only goes offline when both of its copies are unavailable, and since the copies are always in different enclosures, that needs one failed OSD in A and one failed OSD in B, both holding the same PG. Our 13 dead OSDs were split across both enclosures. Every PG that happened to have landed on one from each side went dark. That produced 88 offline PGs from 13 failures, and later 103 as more state came in. And the number that actually caused the incident: 21 OSD processes on a single host. Ask systemd to stop 21 of anything at once, against a single default timeout, while each one is trying to flush its own database. Some of them will not make it. Had those same 48 OSDs been spread 12 / 12 / 12 / 12, every host would have had twice the timeout budget per daemon. The scene Here is what we knew an hour after power came back: - 35 of 48 OSDs up. The other 13 crash-looping across all four storage hosts - Every failing OSD reported a variation of the same error: rocksdb: verify_sharding unable to list column families: Corruption - The disks themselves were healthy. No I/O errors in dmesg, no SMART complaints - 88 placement groups already offline, by the arithmetic above. That number grew to 103 as more state came in - Our cluster had been HEALTH_OK with 48/48 OSDs for three weeks before this The last point matters. This was not creeping hardware rot. Something we did during the shutdown broke them all at once. Step 1: The cluster lied to us about its own state (ceph osd set nodown) The first ceph -s after power-on looked fine: 48 osds: 48 up (since 86s), 48 in (since 3w) It was wrong. Our shutdown procedure sets the standard quiesce flags before halting, and nodown was still set. nodown prevents OSDs being marked down, so the monitor keeps reporting them as healthy regardless of reality. We released the flags and got the truth: 48 osds: 35 up (since 7m), 48 in (since 3w) Lesson: if your power-on checklist validates the OSD count before releasing the quiesce flags, that check passes falsely. Release the flags first, or cross-check with systemctl list-units 'ceph-*osd*' --state=failed on each host. We spent ten minutes acting on a number that was fiction, and the only reason it was not longer is that an OSD count is easy to sanity-check from the hosts themselves. Step 2: What actually broke (bluefs-export) BlueStore keeps its metadata in an embedded RocksDB, which lives on a small filesystem called BlueFS on the same block device. RocksDB continuously rewrites two small files: - CURRENT: a 16 byte pointer containing the current MANIFEST's filename - MANIFEST-NNNNNN: the log of version edits describing which SST files hold what fsck would not open the database. But bluefs-export does not need RocksDB at all, it just reads BlueFS and dumps the files to a directory: cephadm shell --name osd.0 --mount /var/tmp/osd0-bluefs:/export -- \ ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-0 bluefs-export --out-dir /export That worked on every single failing OSD. Suddenly we had the whole database as ordinary files we could inspect without risk. This one command is the foundation of everything that follows. What it showed on the first OSD: 25 SST files present and intact, MANIFEST present, and CURRENT containing 16 bytes of binary garbage where it should have contained the string MANIFEST-000147\n. Here is the giveaway. A healthy MANIFEST starts with a CRC and a known string. Ours did not: healthy manifest: 56 f9 b8 f8 1c 00 01 01 1a 6c 65 76 65 6c 64 62 >V........leveldb< our MANIFEST: 00 00 00 00 00 00 68 07 00 00 00 00 00 00 00 00 >......h.........< another OSD: 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 >aaaaaaaaaaaaaaaa< That run of 0x61 is the letter a repeated. It is a disk test fill pattern left over from whenever those drives were last burned in. That is what unflushed blocks read back as. So BlueFS had journaled the file metadata for those writes, recording size, mtime and allocated extents, but the file contents were never flushed to the device. On restart RocksDB read the recorded size and got whatever was on those blocks beforehand. Every damaged file carried an mtime inside our shutdown window. We did this to ourselves. The root cause, in one sentence shutdown -h now on a host running 21 OSD containers asks systemd to stop them all in parallel against its default timeout, and SIGKILLs any that do not finish in time. Our runbook was meticulous about stopping applications gracefully, giving every database a generous -t and verifying exit code 0. Not one application database needed crash recovery afterwards. The storage layer got none of that courtesy: just shutdown -h now and hope. Step 3: Every repair tool refuses to run Here is the part that makes this failure mode nasty. Ceph ships tools that could fix this. None of them will run. bluefs-import, the supported way to write a file back into BlueFS: inferring bluefs devices from bluestore path rocksdb: verify_sharding unable to list column families: Corruption: CURRENT file does not end with newline error preparing db environment: (5) Input/output error It opens the RocksDB environment before writing. Which is the thing that is broken. ceph-kvstore-tool bluestore-kv ... destructive-repair: mounts BlueFS fine, then opens RocksDB before repairing. Same wall. RocksDB's best_efforts_recovery option, which exists precisely to ignore a corrupt CURRENT and MANIFEST and rebuild from the SST files, is a DB::Open option. Ceph calls rocksdb::DB::ListColumnFamilies() before DB::Open(), and ListColumnFamilies reads the MANIFEST directly. The option never gets a chance to apply. Every road is circular: to repair the index, the tool must first open the index. Step 4: Writing to the raw block device (bluefs-log-dump) If no tool will write the file, write it yourself. BlueFS records every file's inode and physical extents on the block device, and you can dump that log: ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-0 bluefs-log-dump 0x1107f: op_dir_link db/CURRENT to 137 0x10cd4: op_file_update file(ino 137 size 0x10 ... extents [1:0x568000~10000]) db/CURRENT is inode 137, size 0x10 (16 bytes), living at device offset 0x568000. Before writing anything, prove the location and save a rollback: # read those bytes raw and byte-compare against the exported copy dd if=$DEV bs=1 skip=$((0x568000)) count=16 status=none > /tmp/raw16.bin cmp /tmp/raw16.bin /var/tmp/osd0-bluefs/db/CURRENT && echo "location confirmed" # save the whole enclosing 4K block as an exact rollback dd if=$DEV bs=4096 skip=$((0x568000/4096)) count=1 of=/var/tmp/block.orig status=none Then a read-modify-write of that single block, replacing only the first 16 bytes with MANIFEST-000147\n. The error changed: before: Corruption: CURRENT file does not end with newline after: Corruption: checksum mismatch in file db/MANIFEST-000147 Progress. The fix worked and exposed the next layer. Gotcha that will bite you: some OSDs record their extents in op_file_update_inc (incremental) records rather than op_file_update. If you only read the latter you get a stale offset with no extents and you will write to the wrong place. Our byte-comparison guard caught exactly this on one OSD and aborted before writing. Never skip the comparison step. Step 5: Rebuild the index on a copy, not on the patient The MANIFEST could not be repaired in place. We wrote a small parser for RocksDB's log format (7 byte header: masked CRC32C, length, type) and ran it over the file: file size: 11592 bytes valid records: 0 FIRST BAD RECORD at offset 0 -> CRC mismatch type=104 len=0 Type 104 is 0x68, not a valid record type. Stale bytes from offset 0. There was nothing to truncate back to, which is the difference between "interrupted mid-write" and "never written at all". If a file had been torn mid-write we would have seen N valid records and then damage, and we could have truncated to the last good offset. Zero valid records means the contents never reached the disk. That distinction is worth being able to measure, so here is the parser. Pure stdlib, because it has to run on a storage host or inside a cephadm container where you cannot install anything. {% details rocksdb-log-check.py (full script) %} #!/usr/bin/env python3 """ rocksdb-log-check.py - validate a RocksDB MANIFEST / WAL record by record. Answers one question: how much of this file is still readable, and where exactly does it stop being readable? python3 rocksdb-log-check.py db/MANIFEST-000147 THE FORMAT (rocksdb/db/log_format.h, log_reader.cc) A MANIFEST is a log file: a sequence of 32 KiB blocks, each holding one or more records. +--------+--------+------+-----------------------+ | CRC32C | length | type | payload | | 4 LE | 2 LE | 1 | <length> bytes | +--------+--------+------+-----------------------+ * The CRC is CRC32C (Castagnoli, poly 0x82F63B78), NOT zlib's CRC32, and it is stored *masked* (see unmask below). * The CRC covers the type byte plus the payload. Not the header. * Records never straddle a block boundary. If fewer than 7 bytes remain in a block, the writer zero-fills them and starts the next block. * A run of zeroes is normal padding, not damage. RocksDB's own reader tolerates it, which is also why zero-padding a rebuilt manifest back into place works. READING THE OUTPUT valid records: 0, bad record at offset 0 The file never held a valid record. Nothing to salvage, nothing to truncate back to. This is the signature of unflushed data: the metadata was journaled but the contents never reached the device, so you are reading whatever was on those blocks beforehand. valid records: N, bad record partway in Torn mid-write. Truncating to the last good offset usually yields a loadable manifest. A type byte outside 1..8 means you are almost certainly looking at stale disk bytes rather than a damaged record. """ import struct import sys BLOCK_SIZE = 32768 # rocksdb::log::kBlockSize HEADER_SIZE = 7 # 4 byte crc + 2 byte length + 1 byte type MASK_DELTA = 0xA282EAD8 # rocksdb::crc32c::kMaskDelta POLY = 0x82F63B78 # CRC32C (Castagnoli), reversed representation RECORD_TYPES = { 0: "kZeroType", 1: "kFullType", 2: "kFirstType", 3: "kMiddleType", 4: "kLastType", 5: "kRecyclableFullType", 6: "kRecyclableFirstType", 7: "kRecyclableMiddleType", 8: "kRecyclableLastType", } # Python's zlib only ships CRC32, so build a CRC32C table by hand. _TABLE = [] for _i in range(256): _c = _i for _ in range(8): _c = (_c >> 1) ^ (POLY if _c & 1 else 0) _TABLE.append(_c) def crc32c(data, crc=0): crc ^= 0xFFFFFFFF for byte in data: crc = _TABLE[(crc ^ byte) & 0xFF] ^ (crc >> 8) return crc ^ 0xFFFFFFFF def unmask(masked): """Inverse of rocksdb::crc32c::Mask: subtract the delta, rotate right 17.""" rot = (masked - MASK_DELTA) & 0xFFFFFFFF return ((rot >> 17) | (rot << 15)) & 0xFFFFFFFF def check(path): with open(path, "rb") as handle: data = handle.read() print(" file: %s" % path) print(" file size: %d bytes" % len(data)) pos = 0 records = 0 last_good = 0 failure = None while pos + HEADER_SIZE <= len(data): # Records never straddle a block boundary. A short tail is padding. offset_in_block = pos % BLOCK_SIZE if BLOCK_SIZE - offset_in_block < HEADER_SIZE: pos += BLOCK_SIZE - offset_in_block continue header = data[pos:pos + HEADER_SIZE] masked_crc = struct.unpack("<I", header[0:4])[0] length = struct.unpack("<H", header[4:6])[0] rec_type = header[6] # An all-zero header is preallocated space, not corruption. if masked_crc == 0 and length == 0 and rec_type == 0: print(" [zero region begins at offset %d]" % pos) break payload = data[pos + HEADER_SIZE:pos + HEADER_SIZE + length] if len(payload) < length: failure = (pos, "truncated payload: want %d, have %d" % (length, len(payload))) break # The CRC covers the type byte and the payload, not the header. expected = unmask(masked_crc) actual = crc32c(bytes([rec_type]) + payload) if expected != actual: failure = (pos, "CRC mismatch type=%d (%s) len=%d want=%08x got=%08x" % (rec_type, RECORD_TYPES.get(rec_type, "INVALID"), length, expected, actual)) break records += 1 pos += HEADER_SIZE + length last_good = pos print(" valid records: %d" % records) print(" last good byte offset: %d" % last_good) if failure: print(" FIRST BAD RECORD at offset %d -> %s" % failure) else: print(" no bad record found before EOF or zero padding") print(" trailing bytes after last good: %d" % (len(data) - last_good)) return 0 if records and not failure else 1 if __name__ == "__main__": if len(sys.argv) != 2: sys.exit(__doc__.strip().splitlines()[0]) sys.exit(check(sys.argv[1])) {% enddetails %} Run it against a healthy manifest from any working OSD first, so you know what a good result looks like on your own cluster before you trust it on a broken one. But RocksDB has a repairer that rebuilds the MANIFEST by scanning the SST files. It will not run against a live broken OSD. It runs perfectly against a directory: cp -a /var/tmp/osd0-bluefs /var/tmp/osd0-test # work on a copy cephadm shell --name osd.0 --mount /var/tmp/osd0-test:/t -- \ ceph-kvstore-tool rocksdb /t/db destructive-repair The results were much better than we feared: | OSD | SST files before | after | quarantined | |---|---|---|---| | A | 28 | 28 | MANIFEST only | | B | 33 | 33 | MANIFEST only | | C | 24 | 24 | MANIFEST only | | D | 25 | 23 | MANIFEST + 2 SSTs | | E | 37 | 31 | MANIFEST + 6 SSTs | On most OSDs it retained every single SST. The object metadata was fine all along. Only the index describing it was gone. Now we had a valid rebuilt MANIFEST sitting in a directory, and the same problem as before: how do we get it back in? Same answer. Locate the original MANIFEST's extents, write the rebuilt one into them, zero-padded. RocksDB's log reader explicitly tolerates zero-filled regions, because preallocation produces them. Step 6: The assert that sent us to the source code The database opened. Then: src/kv/RocksDBStore.cc: 1221: FAILED ceph_assert(recreate_mode) No message, no explanation. So we read the source, and it explained everything: static const char* sharding_def_file = "sharding/def"; static const char* sharding_recreate = "sharding/recreate_columns"; static const char* resharding_column_lock = "reshardingXcommencingXlocked"; ... bool recreate_mode = status.ok() && sharding_recreate_text == "1"; ... if (missing_cfs.size() > 0 && /* lock not among missing */) { ceph_assert(recreate_mode); for (...) db->CreateColumnFamily(missing_cfs[i].options, ...); } ... if (existing_cfs.size() != rocksdb_cfs.size()) { derr << "extra columns in rocksdb..."; return -EIO; } Three facts fell out, and they define the whole problem space: 1. Missing column families: Ceph will create them for you, but only in recreate_mode, which is set by a BlueFS file sharding/recreate_columns containing "1". Which only bluefs-import can create. Which needs the database open. Circular again. 2. Extra column families: hard -EIO, no workaround at all. 3. The assert is skipped entirely if a magic column family named reshardingXcommencingXlocked is among the missing ones. Our rebuilt database had fewer column families than sharding/def declared, because RocksDB's repairer only creates families it finds SSTs for, and some of ours were empty. So the fix was to make sharding/def match reality. If you take one thing from this article, take this: when an assert has no message, read the source. RocksDBStore.cc lines 1025 to 1240 turned an impossible problem into a mechanical one. Step 7: The 130 byte file that cost us an attempt sharding/def is a plain text file describing the column family layout: m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32 It is a fixed size BlueFS file, so any replacement must be exactly the same byte length. Drop the m(3) entry and you are 5 bytes short. The obvious move is to pad with trailing spaces. Do not pad with spaces. The parser splits on spaces in a while (!text_def.empty()) loop, so trailing spaces create phantom column families with empty names, which keeps missing_cfs non-zero and fires the same assert. That cost us a full attempt, a byte-for-byte revert, and a second pass at working out why an apparently correct definition still would not parse. Pad by expanding a numeric option value with leading zeros instead. Same value, no trailing space: p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=0000032 P=min_write_buffer_number_to_merge=32 The database opened, and fsck started reporting ordinary BlueStore findings instead of refusing to mount. On several OSDs it simply returned fsck success. One more flag was needed for the write path: rocksdb: Corruption: force_consistency_checks: VersionBuilder: L0 file #409 with seqno 12429585 15050034 vs. file #501 with seqno 14519025 14953111 That is an artefact of the repairer reconstructing the manifest with overlapping sequence numbers. Setting force_consistency_checks=false in bluestore_rocksdb_options unblocked it. That is a config change, not a data change, and it is reversible. The first OSD came up. The shared filesystem went healthy immediately. Step 8: Stop repairing OSDs you do not need (osdmaptool) At this point we had a proven procedure and 12 more broken OSDs. Getting the first one back had taken half an hour of careful, irreversible work, and every one after it still meant raw-device writes on hardware holding live data. Before grinding through all of them, we asked a better question: which of these OSDs does anyone actually still need? First, get the list of what is genuinely broken, not what is merely unhappy: ceph pg dump pgs_brief | awk '$2 !~ /active\+clean/ {print $1, $2}' 4.3ff down 4.3a3 down 4.266 undersized+peered 9.2 undersized+peered 10.8 down ... Two different states, and the difference is everything: - undersized+peered means one replica is gone. The other is alive and holds a complete copy. The PG is not serving writes because min_size=2, but no data is at risk. - down means both replicas are on failed OSDs. Nothing is serving it and nothing can, until one of those two OSDs comes back. Now ask CRUSH where each of those PGs is supposed to live: ceph osd getmap -o osdmap.bin osdmaptool osdmap.bin --mark-up-in --test-map-pg 4.3ff osdmaptool: osdmap file 'osdmap.bin' parsed '4.3ff' -> 4.3ff 4.3ff raw ([17,43], p17) up ([17,43], p17) acting ([17,43], p17) --mark-up-in is the flag that makes this useful. It tells osdmaptool to compute placement as if every OSD in the map were up and in. Without it you get the current degraded reality, which just repeats what you already know: the PG has nowhere to live. With it you get the intended placement, which is the question you actually care about: if I fix these OSDs, where does this PG go? Reading the output 4.3ff wants two OSDs, osd.17 and osd.43. Our 13 dead OSDs were 0, 1, 5, 6, 7, 8, 9, 10, 13, 15, 16, 17, 21. So for this PG: - osd.17 is dead - osd.43 is alive One of the two survives, and because it is a full replica it holds every object in that PG. This PG needs no repair work at all. We can purge osd.17 outright and Ceph will re-replicate the PG from osd.43 onto any healthy OSD in the right enclosure. The hours of manifest surgery would buy us nothing here. Compare 10.8, which came back acting ([5,32]). Both osd.5 and osd.32... osd.5 is in our dead list. osd.32 is not, but 10.8 was reported down rather than undersized, which told us the map was stale and osd.32 had also dropped. When both members of the pair are dead, that is a PG we must go and rescue, and it is the only kind that justifies the expensive procedure. So every stranded PG falls into exactly one of three buckets: | Dead replicas | PG state | What it means | Action | |---|---|---|---| | 0 of 2 | active+clean | untouched | nothing | | 1 of 2 | undersized+peered | a complete copy survives | purge the dead OSD, let it re-replicate | | 2 of 2 | down | no live copy anywhere | must repair at least one of the pair | Running it over everything Loop the whole inactive list and bucket it: ceph osd getmap -o osdmap.bin DEAD="0 1 5 6 7 8 9 10 13 15 16 17 21" ceph pg dump pgs_brief | awk '$2 !~ /active\+clean/ {print $1}' | while read pg; do acting=$(osdmaptool osdmap.bin --mark-up-in --test-map-pg "$pg" 2>/dev/null \ | grep -oE 'acting \(\[[0-9,]+\]' | grep -oE '[0-9]+(,[0-9]+)*') n=0 for osd in ${acting//,/ }; do case " $DEAD " in *" $osd "*) n=$((n+1));; esac done echo "$pg acting=[$acting] dead_replicas=$n" done | sort -k3 -r The tally across all 88 stranded PGs: dead_replicas=2 23 PGs <- the real work dead_replicas=1 65 PGs <- purge and re-replicate, no repair needed 65 of the 88 offline PGs did not need anything repaired. They were offline purely because min_size=2 refuses writes at one copy, not because the data was gone. Purging the dead OSD and letting Ceph re-replicate from the survivor fixes them, and that is a normal, boring, supported operation. That left 23 PGs that genuinely had no live copy. Taking the union of the OSDs appearing in those 23 both-dead pairs gave us a critical set of 5 OSDs, not 13. The other 8 could be purged immediately. One more thing the output told us, and it is the payoff for getting the failure domain right back in the setup section. Because CRUSH is constrained to put one replica in each enclosure, a both-dead pair is always one OSD in enclosure A and one in enclosure B. So for each of those 23 PGs we had a free choice: whichever of the two OSDs looked more repairable was the one to work on. We never had to fix a specific OSD, only one of a pair. Two of the five turned out to be unrecoverable, and it cost us nothing, because their partners carried the same PGs. Lesson: in a recovery, the cheapest work is the work you prove you do not have to do. Ten minutes with osdmaptool --mark-up-in turned 13 expensive repairs into 5, and told us exactly which 8 OSDs we could stop worrying about. Do this before you start repairing, not after. Step 9: Get the data out, do not put the OSD back For the OSDs too damaged to return to service, there is a better move than nursing them into production. Extract the placement groups and import them onto healthy hardware: # from the damaged OSD ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-9 --op export \ --pgid 9.2 --file /x/9.2.export \ --bluestore-rocksdb-options="...,force_consistency_checks=false" # onto a healthy one (stop it first) ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-43 --op import --file /x/9.2.export This is what Ceph maintainers recommend for this situation, and it is much safer than running a hand-patched OSD in production. We recovered 22 placement groups this way, including a 13.6GB one, moving them between hosts over the storage network in seconds. Two tricks that mattered here: A single unreadable object can block a whole PG export. Ours aborted with read failure: (5) Input/output error on one object. Remove that object and the other 25 export fine: ceph-objectstore-tool --data-path ... --pgid 4.3a3 \ '["4.3a3",{"oid":"...","key":"","snapid":-2,"hash":...,"pool":4,"namespace":""}]' remove ceph osd lost and ceph osd purge do not touch the disk. They edit the OSD map only. We verified this empirically by exporting a complete 84MB placement group from a fully purged OSD, using --no-mon-config to skip the auth lookup that the purge removed. That property is what made the whole recovery safe: nothing was destroyed to reach a healthy cluster. The one object we lost Final tally: one object out of 159,001. It was a 4MB block that was physically unreadable on its disk, a genuine hard I/O error, in a placement group whose other replica was on another failed OSD. We identified exactly what it was by decoding the filesystem backtrace stored in the first object of the same file: rados -p cephfs.cephfs.data getxattr 10000103179.00000000 parent > parent.bin ceph-dencoder type inode_backtrace_t import parent.bin decode dump_json That gave us the full path. It turned out to be a fragment of a nightly backup image from five days earlier, superseded by three newer complete backups. 516 of that file's 517 objects survived. Being able to answer "exactly which file, exactly which bytes" is worth the effort. "We lost one object" is a shrug. "We lost bytes 16MB to 20MB of a superseded backup taken on the 13th, and here are the three newer copies" is an answer you can take to a customer. And then, before declaring victory: ceph pg deep-scrub <pgid> # for every recovered PG inconsistent_pgs: 0 scrub_errors: 0 pgs_by_state: [('active+clean', 1265)] "Import successful" is not the same as data integrity. Scrub is the proof. What we got wrong (and what we fixed) 1. We never stopped the OSDs. The whole incident traces to this. The runbook stopped applications carefully and left up to 21 OSD containers per host to systemd's default timeout. Now: systemctl stop ceph-osd.target, then a loop that waits and confirms every OSD container is gone before shutdown is allowed to run. 2. We trusted ceph -s with nodown set. It reported 48 of 48 OSDs up while 13 were crash-looping. Release the flags before validating anything, and never let a post-power-on checklist assert on a number the flags can fake. 3. Our runbook's storage topology was wrong. It said one host held zero OSDs. It held three, and they were load-bearing. A document nobody re-verifies drifts into fiction, and you find out during an incident. 4. sudo docker compose with ~ in a volume path. During restart, ~/models/huggingface expanded to /root, and Docker silently created an empty directory instead of failing. Cost us a crash-looping service until we noticed the bind was wrong. Check docker inspect mounts, do not assume. 5. We twice concluded the data was unrecoverable. Both times we were wrong, and both times someone pushed to keep going. The second time, that was the difference between losing 18 placement groups and losing one object. The takeaway Ceph is genuinely good at surviving hardware failure. It is much less good at surviving you, and the tooling assumes a working database in places where you most need it not to. Three things are worth internalising before you need them: Work on copies. bluefs-export gives you the entire database as files while the OSD refuses to mount. Everything destructive we did, we did on a copy first, then applied. We reverted three separate attempts byte-for-byte and lost nothing by trying them. Verify before you write, always. Every raw device write was preceded by a byte comparison proving the offset, and by saving the block we were about to overwrite. That discipline caught a wrong offset that would have corrupted an OSD we still needed. Read the source. The assert that blocked us cold had no message. Two hundred lines of RocksDBStore.cc turned it into a checklist. If you run infrastructure on open source, the source is part of your documentation. And go check your shutdown procedure right now. If it stops your applications carefully and then calls shutdown -h now on a host running database daemons, you have the same bug we had. It has not bitten you yet because your daemons have always finished in time. One day one of them will not. The tools, and where to get them Everything we wrote during this recovery is published. The first two are dangerous in different ways, so read the header comments before running either. | Tool | What it does | |---|---| | rocksdb-log-check.py | Validates a RocksDB MANIFEST or WAL record by record. Tells you whether a file was torn mid-write (truncate and recover) or never written at all (nothing to salvage). Pure stdlib, so it runs inside a cephadm container. | | bluestore-manifest-transplant.sh | The Step 4 to Step 7 procedure, packaged: locate CURRENT, MANIFEST and sharding/def on the raw device, byte-verify every offset, save a rollback, rebuild sharding/def to match the recovered column families, write. Dry run by default, and it aborts if any offset fails verification. | | ceph-safe-shutdown.sh | The fix. Stops ceph-osd.target and then waits and confirms every OSD is actually gone, exiting non-zero if any are still running when the budget expires. Chain it: ./ceph-safe-shutdown.sh && shutdown -h now. This is the script whose absence caused the entire incident. | If you only take one of them, take the last one. Further reading - Ceph: BlueStore config · ceph-bluestore-tool · ceph-objectstore-tool · CRUSH maps · Placement groups - RocksDB Repairer, which is what destructive-repair invokes - RocksDB log format, if you want to understand the parser - Ceph source: src/kv/RocksDBStore.cc, the sharding verification and column family handling. This is the file that turned an impossible problem into a mechanical one --- From the first failed OSD to a scrub-verified clean cluster took about three and a half hours. The single most valuable command in the whole incident was bluefs-export, and we found it in the man page.
Dónde cuesta más y menos comprar un apartamento de 3 o más habitaciones en Itagüí: ranking de barrios En Itagüí, un apartamento de 3 o más habitaciones cuesta 560 millones de pesos en la mediana en Suramerica, el barrio más caro, y 298 millones de pesos en San… DOI: 10.5281/zenodo.23525470
Kanzo update: one kanban board shared across all your machines and agents, still a 4.5 MB binary using 4.5 MB of RAM A week ago I shared kanzo here, a small kanban board I built that you and your coding agents share: a board in your browser plus an MCP server in one program. You tell Claude Code "take the next task from kanzo" and it moves the card to doing and comments as it works. The thing I kept missing was using one board from more than one computer. Git sync worked, but only when you switched machines. So now one machine runs kanzo share on, the others run kanzo join <address>, or you click the share icon on the board and paste the address, and everyone's agents work from the same tasks, live. On the home network it just works, and from outside I use Tailscale. A Pi at home keeps the board, so my laptops don't need to stay on. How Claude helped: I built kanzo with Claude Code. For this feature I set the constraints (no new MCP tool, nothing extra for agents to read, no setup) and Claude Code wrote a good part of the code, then I reviewed and cut it down. It was also useful for measuring, so I could check sharing didn't make it slower or bigger. And the tasks for building it were tracked in kanzo itself, with Claude Code picking them up. Performance is the same as before: it starts in 2 ms, a 3,000-task board loads in 17 ms, and sharing adds nothing for the agents. If you turn on Backlog in Settings, agents leave those tasks alone until you move them to todo, which is where I park ideas. It's free and open source (AGPL). To try it on macOS/Linux: curl -LsSf github.com/emanuele-em/kanzo/releases/latest/down… | sh then run kanzo, and add it to Claude Code with claude mcp add kanzo --scope user -- kanzo mcp. No login on the shared board though, so keep it on your own network or VPN. emanuele-em/kanzo
Meltype fork紹介② @appipinopi さんのMeltype-Flashです! appipinopi/Meltype-Flash
Turn any android phone into Desktop class experience including full linux environment without root, integration with windows & mac. Access linux directly from PC on your phone! dotnetdreamer/open-android-dex t.co
@Biggest If you have 64GB+ RAM you can have a look at that Cedriceuh/freetoken-rdna3
搞了个marketing的skill 依旧免费 卧槽,我特么跟钱有仇 aronhy/marketing-skill-hub t.co
Multi-user KDE Plasma over RDP on a headless server KRdp is great for sharing the session of a machine with a monitor, but it doesn't start a separate Plasma desktop for each account on a server nobody sits at. And Plasma can't start on a headless machine, because KWin wants a monitor. What I ended up doing: each account gets a headless sway with a virtual output the size of the RDP client's window. The sway config runs startplasma-wayland. KWin sees it's inside another Wayland compositor and runs as a fullscreen window of it, the way it does when nested for development. Sway does the capture (H.264) and the input, and Plasma never needs a real monitor. It works for login, lock screen, logout, reconnect, and 30 minutes idle. An idle Plasma session takes about 750 MB. What I haven't tested: how many busy Plasma users a server holds, and AMD or NVIDIA on Arch. The write-up has the config and the small things that broke (clipboard, keyboard layout, what "Log out" does in a nested setup): dev.to It's part of Waydesk, a free multi-user RDP desktop server for Linux (AGPL-3.0, my project): zhoorta/waydesk I'd like to hear from anyone who runs Plasma for several users on a server: what breaks, and what you'd want from it.
@Kantorcodes @Kantorcodes Public repo: agentsearchhq-coder/agentsearch-mcp Ownership already verified on HOL for io.github.agentsearchhq-coder/agentsearch-mcp (Owner verified Nov 2). Happy to help with any catalog update.
I made a translucent icon pack generator for Nothing OS 5.0 It's free and open source. You can download and check it out from here: linuxct/nothing-icongen In contrast with other icon packs, this one uses Nothing's icon generation pipeline from its launcher and original icon pack (the one with solid background), but changes the icon's background to be translucent in the same way as the widgets. To achieve this, it rebuilds the icon pack on the device on the fly - once everything is correctly set up, the icon pack will be kept up to date with your installed apps automatically! Installing or updating apps rebuilds your icon pack on the fly, and changing the device's dark/light theme also updates the icon pack accordingly without rebuilding it (I know that for other icon packs, this was not always updated instantly, but this one does, with some tricks). --- Notice: Follow the README's instructions carefully. The app cannot be installed directly on-device! --- Edit: ever since this was posted I pushed some updates to IconGen, which include: - Complete rewrite of the icon generation pipeline to not be dependant on specific Nothing launcher versions - Fixes for BitmapDrawable icon types - Added an appfilter so that the custom icons only show in the launcher, not in the notification shade or settings - Implemented an optional update checker (it only queries GitHub, nothing else) If you had downloaded an earlier version, make sure you update to 1.0.0.4 or newer to take advantage of these updates!
Compared tree ensembles in Module 6 of #mlzoomcamp 🌳 Decision tree → random forest → XGBoost, then tuned depth, forest size, and learning rate to lower validation RMSE. Code: github.com/ePaddyy/machine_learning_zoomcamp/blob… @DataTalksClub @Al_Grigor
Vimの設定を Luaで書きたくない。っていうか Vim9 script が好きやねん。 マルチコアCPUだったら簡単に並列処理可能になったのあまり知られていない気がする。(`:h --stdio-channel`) github.com/vim/vim/commit/673be365fd1f94caa60fabd…
复制一段话,中间顺手复制了链接,那段话就被顶掉了,系统只认最后一次。 TieZ 跨平台剪贴板管理器,历史按时间排列,可打标签搜索收藏,本地存储,Win/macOS/Linux。 jimuzhe/TieZ
Claude kept re-reading my whole codebase and burning my tokens. So I gave Claude a memory with Graphify. Now it checks its memory, not every file. I save more than 60% TOKENS just with this skill Here is the link: Graphify-Labs/graphify t.co
[ethresear.ch] [zk-s[nt]arks] Vocdoni Protocol: Enabling Decentralized Voting for the Masses with ZK Technology NemoUnus November 2 Following your suggestion, I have opened the design question as an issue on davinci-zkvm before writing any code: Design discussion: an anonymous census option (hidden leaf + per-process nullifier) · Issue #6 · vocdoni/davinci-zkvm · GitHub It covers an anonymous census option (hidden leaf + per-process nullifier) and asks whether you would accept it as a new census origin. Corrections to my reading of the current code are very welcome. — Nemo Unus Visit Topic to respond. You are receiving this because you enabled mailing list mode. Design discussion: an anonymous census option (hidden leaf + per-process nullifier) · Issue #6 · vocdoni/davinci-zkvm · GitHub: davinci-zkvm#6
@iamlukethedev My bad, I missed the follow-up: PR #254 documents a real Grok export/import run. The 'untested' note I quoted above is stale. hermes-official-plugins#254
9/ Notes, practice workbook and images: CosmasMwirigi/100-days-of-excel #Excel #Statistics #DataAnalytics #LearningInPublic t.co
I made a small Expo plugin for native Android widgets in Kotlin I needed home screen widgets on Android for an Expo app and wanted to write them the normal Android way, with Kotlin and RemoteViews, without keeping the android folder in git. So I packaged what I had: rsoaresdev/expo-android-widgets You keep your widget sources in a folder, the config plugin copies them in on every prebuild and adds the receivers to the manifest. From JS you call setWidgetData() and reloadAllWidgets(), and the widget reads the data with one line of Kotlin. There’s an example widget you can copy. It’s tested on SDK 58, on an emulator and in CI with a real Gradle build. If you’d rather write widgets in JSX, react-native-android-widget is great. This one is for when you want plain Android widgets. Feedback welcome, especially from anyone on older SDKs.
👋 Welcome to r/LowLevelBasic - Introduce Yourself and Read First! This is our new home for all things related to Low-Level Basic. We're excited to have you join us! Low-Level Basic : 1. Readable & Approachable: Clear Sub, Function, Module, Class, Select Case, With, and Print statements without bracket clutter. 2. True Bare-Metal Control: Explicit pointer types (Pointer To T), MMIO-safe registers (Volatile Pointer To T), inline assembly (Asm), and byte-accurate struct alignment (<Packed>, <Align(N)>). 3. Rust-Grade Safety Intent: Bounds-checked arrays and null-pointer safety guards by default; explicit Unsafe context blocks required for raw memory addresses and physical pointer manipulation. 4. Freestanding & Self-Contained: Zero mandatory host C library runtime dependencies. Perfect for microkernels, OS kernels, bootloaders, firmware, and high-performance standalone utilities. 5. Modern Language Power: First-class generic templates (Structure Box(Of T), Function Min(Of T)), pattern matching (Match ... Case), lambda expressions (=>), deterministic scoped cleanup (Using ... End Using), interfaces & vtables, and algebraic Result(T, E) error types. What to Post Post anything that you think the community would find interesting, helpful, or inspiring. Feel free to share your thoughts, photos, or questions about LLB. Community Vibe We're all about being friendly, constructive, and inclusive. Let's build a space where everyone feels comfortable sharing and connecting. How to Get Started Read the documentations on the repo: karunci/LLB
Silk.NET Sea of Grass Hello! I have long been obsessed with the idea of sailing on a sea of grass. A few years ago I tried building it in Unity3D and UE, but couldn't get it efficient enough for gameplay. Last year, I started from scratch with Silk.NET and wrote the grass engine core manually, later Claude helped me optimize it further. As a hobby project, I've made a small game around the technology, where you play as an acorn sailing a vast sea of grass searching for the 'perfect soil' to root in. I will release it as a free download soon. I wanted to share the grass engine separately and asked Claude to extract it from my main project and put it in a new repo: kinomikun/SilkGrass This code represents my design, but it is not my original code. If you're inspired, please use this in any way you like. And I'm interested if you can think of ways to make it cheaper or prettier. Feel free to ask me anything about the project. Thanks! Edit: removed the unintentional silk.net hyperlink.
Slang Is a Dial: Training a 125M Language Model From Scratch on Free GPUs > TL;DR: I built a language model from scratch, end to end, on free GPUs: a 125.9M-parameter model that writes in a register you choose, with slang, emoji, abbreviation, elongation and formality as five dials. The idea is simple and testable: measure the register of every training document with code, write it as five control tokens in front of the text, and the model learns p(text | register). On 200 held-out requests the generated register lands within 0.146 of the request on average, against 0.646 for a shuffled-request baseline. Behind that number sit a dated slang lexicon, a deterministic register annotator, a data pipeline and tokenizer, a verified Llama-style model, a lease-based training relay that turns flaky free sessions into one continuous run, a regression-tested training stack, and a Llama-format export that loads in three lines of transformers. Five live sketches below let you poke each idea. > > Model: Ravikishan/bussin-125m · Code: Ravikisha/Bussin The numbers at a glance | | | |---|---| | Model | 125.9M parameters, Llama architecture, 49,152-token vocabulary, 1,024 context | | Training | About 7.8 billion tokens on free Kaggle T4 pairs, $0 spent | | Register control | Register error 0.146 against 0.646 for a shuffled-request baseline (200 held-out requests) | | One dial at a time | slang_3 measured 2.88, emoji_2 measured 2.00, abbrev_3 measured 2.40 | | Verification | Six behavioural test suites; export matches the original to 3.6e-05 in logits | | Try it | Ravikishan/bussin-125m on the Hub, code on GitHub | --- Table of Contents 1. [The idea: cringe is register mismatch](#the-idea-cringe-is-register-mismatch) 2. [What I built](#what-i-built) 3. [Register as an input](#register-as-an-input) 4. [Measuring register with code](#measuring-register-with-code) 5. [A dated lexicon that knows what is current](#a-dated-lexicon-that-knows-what-is-current) 6. [The data pipeline and tokenizer](#the-data-pipeline-and-tokenizer) 7. [The model](#the-model) 8. [Training systems](#training-systems) 9. [The relay: continuous training on free compute](#the-relay-continuous-training-on-free-compute) 10. [Engineering rigor](#engineering-rigor) 11. [Results](#results) 12. [Talking to it](#talking-to-it) 13. [Shipping it](#shipping-it) 14. [Skills this project demonstrates](#skills-this-project-demonstrates) 15. [What is next](#what-is-next) --- The idea: cringe is register mismatch Ask a general-purpose model to "talk like Gen Z" and you get "Dear Sir, no cap, I hope this email finds you bussin." It knows the words. It has no model of when they fit. That led to the central observation of this project: nobody cringes at a teenager saying "that's so cooked" to a friend. They cringe when a bank's Twitter account says it. So cringe is not slang. It is register mismatch: the words do not fit the setting. If that is right, the average-of-all-text model is the wrong tool, because averaging the register of a contract and a Twitch chat produces exactly that blend. What you want is a model that treats register as something you can set, and a way to measure whether it did what you asked. Both turned out to be buildable with no human labelling at all. What I built This is a full-stack project, and every layer is code in one repository, with tests: | layer | what I built | |---|---| | Research | A 17-section specification: the thesis, the data tiers, the compute plan, the evaluation design | | Lexicon | A dated slang dictionary with current / aging / dead status and emerging-term discovery | | Annotator | A deterministic five-dimension register scorer that also grades model output | | Data | Mining, cleaning, language routing, PII scrubbing, stage-ordered uint16 shards | | Tokenizer | A 49,152-token byte-level BPE with the register control tokens built in | | Model | A Llama-style 125.9M decoder with a verification suite | | Training | WSD schedule, curriculum, multi-GPU trainer, exact token accounting, hashed checkpoints | | Systems | A lease-based relay and a stateless orchestrator that run training across free sessions | | Evaluation | Register Calibration Error, a paired likelihood test, single-dimension sweeps | | Release | A Llama-format export with a logit-parity check, a model card, a repo with figures | !From idea to release, built end to end Register as an input Instead of p(text), learn p(text | register): {% katex %}\mathcal{L}(\theta) = -\sum_{t} \log p_\theta\left(x_t \mid x_{\lt t},\, r\right){% endkatex %} where {% katex inline %}r{% endkatex %} is a short vector describing the register of the whole document. In practice {% katex inline %}r{% endkatex %} is five small integers, written as control tokens in front of every training document: <|endoftext|><|reg|><|slang_3|><|emoji_1|><|abbrev_2|><|elong_0|><|formal_0|><|/reg|> ngl that movie was bussin At inference you write the prefix yourself and the model continues in that register. Slang stops being a prompt trick and becomes a dial. !Five dials, one prefix: slang, emoji, abbreviation, elongation and formality | dimension | levels | what it measures | |---|---|---| | slang | 0 to 4 | weighted lexicon hits per 100 tokens | | emoji | 0 to 3 | emoji and chat emotes per 100 tokens | | abbrev | 0 to 3 | internet abbreviations (ngl, tbh, fr) per 100 tokens | | elong | 0 to 2 | stretched words (sooooo) and keysmashes | | formal | 0 to 3 | a 0 to 1 formality score from five signals | The register tokens are real vocabulary: 34 special tokens with fixed ids, including end-of-text, padding, register open and close, and every dimension level, so slang level 3 is always token 7. Drag the knobs to build a prefix and see exactly what the model is given, with the real token ids. Set a single dial to its sweep level to see what the trained model produced for it. !p5 visualisation A frame from the interactive sketch in the original. The whole pipeline, end to end: !Mine text, measure its register, prefix each document, train, then dial !mermaid diagram Diagram from the original. The last arrow is the part I like most. Because the register is measured by code and not labelled by people, the same annotator that labels the training corpus can score what the model writes. That makes "cringe" a number, which I call Register Calibration Error (RCE): {% katex %}\mathrm{RCE} = \frac{1}{N}\sum_{n=1}^{N} \frac{1}{5}\sum_{d \in \text{dims}} \left| r^{\text{requested}}_{n,d} - r^{\text{generated}}_{n,d} \right|{% endkatex %} It rises when the model under-uses slang and when it over-uses it, so it cannot be gamed by suppressing slang. Measuring register with code The annotator is deterministic code in bussin/data/register.py: no LLM calls, no human labelling, and cheap enough to run over a whole corpus (it is budgeted at about 30,000 documents per second per core). The thresholds that turn raw rates into levels: | dimension | cut points | unit | |---|---|---| | slang | 0.5, 2, 5, 10 | weighted hits / 100 tokens | | emoji | 0.5, 2, 6 | emoji / 100 tokens | | abbrev | 2, 6, 14 | abbreviations / 100 tokens | | elong | 1, 4 | elongation events / 100 tokens | | formal | 0.30, 0.55, 0.78 | score from 0 to 1 | The design decisions are where the thinking is: - Calibrated buckets. The cut points are chosen so every level holds a meaningful slice of the corpus. A short chat message is about 10 tokens, so a rate per 100 tokens would put a single slang word in the top bucket. Dividing by at least 30 tokens treats a short message as weak evidence, which gives the model a smooth gradient of levels to learn from instead of two extremes. - Ambiguity handling. "Living" and "cap" are ordinary words and slang. Ambiguous words count at 30% weight unless the document also contains an unambiguous slang term, which anchors them at full weight. That trades a little recall for far fewer false positives on ordinary prose. - One signal, one dimension. Emotes (KEKW) and abbreviations (ngl) feed their own dimensions and are excluded from the slang count, even though dictionaries list them too. - Formality from five signals. Terminal punctuation, capitalised sentence starts, mean word length, formal function words (however, therefore) and the absence of chat markers, averaged with equal weight. Here is a simplified port you can poke. It uses a small lexicon, but the thresholds, the 30-token floor, the anchoring rule and the five-signal formality score are the real ones. Click to change the message, and toggle the floor to see what it does to the one-word example. !p5 visualisation A frame from the interactive sketch in the original — watch it run. A dated lexicon that knows what is current Slang needs a lexicon, and a lexicon needs a source. I built mine from the largest public Urban Dictionary dump plus curated lists. Every term carries a first-seen date and a status (current, aging or dead) that is refined from corpus evidence, and the dead terms double as a test set for outdated slang. Then I stress-tested it against 19 contemporary terms and learned something useful about the structure of the problem: the newest definition in that dump is dated 2023-11-09, so a dated dictionary can say what is dead but structurally cannot say what is current. Lexicographers lag usage by years and the dumps lag the lexicographers. So the lexicon has three layers: - Dated sources for the long tail and for status. - CURRENT_SLANG: a curated set of unambiguous recent coinages (bussin, skibidi, fanum tax, mewing, cheugy) that doubles as the anchor set for the ambiguity rule above. - DRIFTED_SENSES: ordinary words whose meaning moved, with a documented sense for each: cooked ("ruined, or doing extremely well"), ate ("performed excellently"), cap, bet, mid, no crumbs, hits different. A frequency filter normally removes these, and it is right to in general, so each exemption needs a written reason. - Emerging-term discovery: a corpus pass that ranks words by recent-versus-baseline frequency against an English vocabulary, so new slang can enter the lexicon from the data itself. The build turns 152,941 dictionary definitions into 55,499 terms plus 1,322 curated entries. The data pipeline and tokenizer The pipeline is a sequence of numbered stages, from building the lexicon through mining, tokenizer training, tokenization and sharding. A few choices worth calling out: - Language routing, not language dropping. Statistical language ID is unreliable on short chat, so the router combines py3langid with marker lists and routes English and Hinglish into their own pools, dropping other languages. - PII scrubbing on the body. A mandatory pass over the text itself, not just the schema, replaces handles, emails, phone numbers and similar patterns with placeholder tokens. - Author columns dropped at ingest, before anything is written. - Elongation capped but remembered. sooo and soooooo carry different register information, so the cleaner caps the run length and records the original length for the annotator. - Stage-ordered shards. Tokens are written as uint16 shards tagged with a curriculum stage, so training order is a property of the data, not of a flag. - A tokenizer gate. The 49,152-token byte-level BPE had to pass a round-trip check before any training: exact round trip (1.0000) and 3.37 bytes per token on Gen-Z text. The model A small Llama-style decoder. The point of choosing something standard is that every result is about the data and the objective, not the network. | | | |---|---| | Parameters | 125,851,392 | | Layers / width | 14 / 768 | | Attention | 12 query heads, 4 KV heads (grouped-query), head dim 64 | | FFN | SwiGLU, hidden 2048 | | Positions | RoPE, theta 10,000 | | Norm | RMSNorm (computed in fp32) | | Embeddings | tied; vocabulary 49,152 (byte-level BPE) | | Context | 1,024 | Embeddings are {% katex inline %}49152 \times 768 = 37{,}748{,}736{% endkatex %} parameters, about 30% of the model, which is why tying them pays off at this size. Three implementation details: - Block-diagonal document mask. Training packs several short documents into one window. The mask is causal and limited to the same document, so nothing attends across a boundary. - fp16 plus a gradient scaler. The free T4 GPUs have no bf16, so activations run in fp16 while logits and loss are computed in fp32, where a 49k-wide logit row does not saturate. - z-loss. A small penalty keeps the log-partition from drifting: {% katex %}\mathcal{L} = \mathrm{CE} + 10^{-4}\,\overline{\left(\log \sum_j e^{z_j}\right)^{2}}{% endkatex %} The model ships with its own verification suite: parameter counts match the analytic formula exactly at all four planned sizes (125M, 400M, 1B and a smoke model); untrained cross-entropy lands at 10.95 against {% katex inline %}\ln(49152) \approx 10.80{% endkatex %}; and tests assert causality, document isolation, RoPE against an independent reference implementation, and grouped-query head grouping. The same code scales: the repository carries configurations for 400M (396,418,176 parameters) and 1B (1,255,245,824), sized so that the 400M model reaches 75.7 tokens per parameter on the same free-compute plan. Training systems A schedule built for an unknown budget. The usual cosine schedule needs the total step count before step one. A free quota is genuinely unknown (Kaggle's own wording is "30 hours or sometimes higher depending on demand"), so I used Warmup-Stable-Decay: warm up, hold the peak for as long as the quota lasts, then decay whenever you decide to finish. $$ \eta(s) = \begin{cases} \eta_{\max}\,\dfrac{s+1}{W} & s \lt W \[4pt] \eta_{\max} & W \le s \lt S \[4pt] \eta_{\min} + (\eta_{\max}-\eta_{\min})\left(1-\sqrt{\dfrac{s-S}{D}}\right) & s \ge S \end{cases} $$ The decay phase doubles as the Gen-Z anneal: the last steps run at a low learning rate over the most contemporary data, so register gets shaped without overwriting the general-English competence learned earlier. A stable-phase checkpoint stays valid, so a different ending never costs the whole run. Move your mouse across the plot to change when the quota ends: > A frame from the interactive sketch in the original — it does not render here. Curriculum as data order. Shards are read in stage order (general, then a ramp, then the Gen-Z anneal), with shuffling inside a stage and never across one, so the curriculum is guaranteed by the loader. Multi-GPU without the tax. The model is wrapped in DataParallel through a thin module that returns only scalar losses, which avoids gathering roughly 0.8 GB of logits on every micro-step. Measured throughput on two T4s is about 14,000 tokens per second, at a global batch of 262,144 tokens. Exact token accounting. The trainer counts the tokens the loader actually served and refuses to run if that differs from what the plan assumes, so the "tokens seen" figure in every checkpoint is real, not an estimate. Checkpoints you can trust. Each checkpoint (about 1.66 GB with optimizer state) holds weights, optimizer, scheduler, gradient scaler, RNG state and the data cursor, with a SHA-256 manifest verified on load. Local copies are removed after a verified push so a long session never fills the disk, and old checkpoints are pruned within a single run's namespace only. The relay: continuous training on free compute Free sessions end after at most 12 hours, or crash, or hit a quota. So training is a relay: each session resumes exactly where the last one stopped, with no server in the middle. - Checkpoints go to the Hugging Face Hub. - A single file, RUN_STATE.json, is the lease: who holds the run and until when. - The lease is claimed by compare-and-swap on the Hub's commit SHA: read the file and its revision, write with that revision as the parent, and if anyone committed first the Hub answers 412 Precondition Failed. That rejection is the CAS. - The holder heartbeats every 10 minutes, and each heartbeat re-reads the state to confirm it still owns the lease. - A stateless orchestrator reads state and quota on a schedule and starts a session only when the lease is free or expired. !Kaggle sessions hand over through checkpoints and a lease on the Hugging Face Hub !mermaid diagram Diagram from the original. This sketch simulates the protocol with illustrative timings (the real heartbeat is 10 minutes and checkpoints land about every 50). Watch a session end, the lease lapse and the next session win the claim, sometimes against a duplicate that loses the race. Click to end the current session. > A frame from the interactive sketch in the original — it does not render here. Around that core sits a set of operational mechanisms: - Divergence watchdog. It compares the median loss of the last 50 steps with the previous 50; a jump above 1.5x triggers a rollback to the last good checkpoint at a reduced learning rate. It resets at stage boundaries and treats a gradient-scaler overflow as a skipped step. - Stall detector. Three consecutive sessions ending on the same step halt the orchestrator instead of spending quota. - Quota ledger. A ledger tracks GPU-hour use per platform, reconciles it and closes orphaned sessions. - Platform adapters. Dispatch adapters for Kaggle GPU and TPU, Colab and Lightning share one interface, with Kaggle GPU run end to end. - History compaction. The checkpoint repository is squashed automatically when its commit history grows, only when no live lease exists. - A status dashboard rendered to a static page after every tick. - Bit-exact resume. An interrupted run reproduces an uninterrupted one with 0.0 relative divergence, which a test verifies. Over the project the relay moved training across many sessions on free Kaggle T4 pairs: about 110 GPU-hours and 6.74 billion tokens in the pretraining phase, then 1.07 billion tokens (4,067 steps) in the register-curriculum anneal. Engineering rigor A model trained for days on free hardware cannot be debugged by re-running it, so the project leans on tests that assert on behaviour, not just on code paths: | test | what it checks | |---|---| | model_sanity_test | init loss near {% katex inline %}\ln V{% endkatex %}, causality, document isolation, RoPE vs an independent reference, GQA grouping | | objective_test | trains a tiny model through the real shard writer, loader and loss, then asserts on what it learned | | accounting_test | 13 checks: token accounting, simulated two-device runs, curriculum order, prune scope | | lease_test | 18 checks, including the two-worker race and heartbeat ownership | | relay_e2e_test | 16 checks across a full two-session hand-off, including no leftover local checkpoints | | resume_test | interrupted vs uninterrupted run: 0.0 relative divergence | The most valuable of these is objective_test. During an audit of the training objective I found a loss-alignment defect (a duplicated label shift) that the usual signals do not reveal, because a healthy-looking loss curve cannot tell you which target the model is learning. I fixed it at the loss, wrote objective_test so the property is checked by what the model learns (it fails on the pre-fix loss and passes on the fix), and continued training under the corrected objective. Treating the loss contract as something to test, not assume, is a habit I would carry to any training stack. Results Two complementary tests. 1. A likelihood test. Score the same held-out text under its true register prefix, a shuffled one and none, paired by document. At step 3,314 a shuffled prefix cost +0.054 nats on average (95% CI +0.047 to +0.062), and the true prefix beat the shuffled one in 80.7% of documents: the model genuinely conditions on the register. 2. A generation test (RCE). I took 200 requests from the measured registers of real held-out documents, stratified so every slang level is requested. The model wrote 48 tokens from the prefix alone and the annotator measured the output. The shuffled-request baseline scores the same generations against another document's request, which is what ignoring the prefix looks like. Click the sketch to flip between the two pairings: > A frame from the interactive sketch in the original — it does not render here. !Register error by dimension: model vs shuffled baseline vs real same-document text The model improves on the shuffled baseline on every dimension, with every 95% interval excluding zero, and its output lands on the requested register more tightly than a real continuation of the same document does (0.146 against 0.466). The cleanest demonstration is to turn one dial at a time, everything else at zero, 50 samples each: !Requested versus measured level when one dimension is set high Requesting slang_3 measured 2.88, emoji_2 measured exactly 2.00, abbrev_3 measured 2.40, elong_2 measured 1.80 and formal_3 measured 2.62. At level 0, every dimension measured between 0.00 and 0.02. The register is a control you can actually reach for. A sample from the prefix slang_3 emoji_1 abbrev_2 (temperature 0.8, one word masked): > I actually thought the same, I was a lil disappointed as I was expecting some cool s*** for this movie, but then i actually liked it, I hope the director can find something funny to make a Casual, hedged, slangy, in the register that was asked for. Talking to it bussin-125m is a base language model, not an assistant. It has no instruction tuning, so it does not answer questions: it continues text in the register you ask for. That makes the register prefix the whole interface, and the cleanest way to see it is to hold the message fixed and turn the dials. Same message, same sampling settings (outputs are cut at 60 tokens), different registers. "thoughts on the new iphone" | register | dials | the model continues | |---|---|---| | formal | slang 0, emoji 0, abbrev 0, formal 3 | I don't know much about phones, so I've been wondering how to connect my phone to the phone with a SIM. I was thinking of buying the new one, however I'm not quite sure how to connect to the new phone. Do you think the SIM connection is the problem? | | chill | slang 1, abbrev 1 | also if you know anyone who uses airpods, pls share them | | casual | slang 3, emoji 1, abbrev 2 | i don't know what the phone is worth rn but it's a nice looking phone that i wanna use for a couple of hours, but i dont know if it's worth anything at all to have a phone that's mid to not worth \$100 also, i feel like i would prefer it | | hype | slang 4, emoji 2, abbrev 3, elong 1 | i don't know what the phone is worth rn but if u wanna get a new phone lmk i wanna get a new iPhone idk if i should get the iphone 12 or the 12 pro i wanna keep my 16 pro btw 🙏 | "is pineapple on pizza ok or nah" (the chill setting returned only a bare link for this one, so it is left out) | register | dials | the model continues | |---|---|---| | formal | slang 0, emoji 0, abbrev 0, formal 3 | I've been meaning to ask if you're okay with this particular pizza and if I should try my best to make it better? I'm going to eat it. | | casual | slang 3, emoji 1, abbrev 2 | for me pineapple is literally the best pizza because its easy to get in and out of, and its not even half as good as the other two i would say pineapple on pizza is actually worse imo, u get the same amount of cooked pizza than pizza on pizza | | hype | slang 4, emoji 2, abbrev 3, elong 1 | for a reference i don't eat pineapple because i love eating pineapple but if u like pineapple on pizza, sure, u can eat pineapple on pizza bro so i'm not actually gonna say pineapple on pizza is bad so yeah, i love pineapple on pizza bro 😭😭😭 | Read down the columns: formal gives capitalised, fully punctuated sentences, while casual and hype give lowercase text, rn, lmk, imo, u, and emoji at the high setting. Nothing about the message changed. That is the dial. I also ran 50 everyday prompts through it: 20 casual, 15 neutral and 15 formal, each with a matching register prefix, same settings throughout. A representative selection (outputs are cut at 70 tokens, and ... marks where I shortened one further): | prompt | register | the model continues | |---|---|---| | i have an exam tomorrow and havent studied | casual | i dont wanna fail, i wanna get into clg (pls do tell me if it actually works) can u pls tell me how to study? i wanna study for 12 hrs everyday edit: how many days will be left? | | can you recommend a good show to binge | casual | i kinda like this kinda vibe tho like if u guys want to start u can try it like naruto naruto is a good show but if u want more anime, pls give me recommendations if u have any suggestions pls give ... | | what's the best game right now | casual | edit: actually the best one rn is smash bros but i'm gonna go with smash bros ultimate edit 2: i'm playing sonic 3 rn and i like the character so much that i'm kinda thinking of buying it, it's ... | | should i cut my own hair | casual | i don't think my hair is actually that good tho, i have thick, wavy and straight hair and i've had a really bad time with frizz i want to be able to grow a nice hair idk ... | | best way to save money as a student | chill | http try to avoid eating out tho | | Explain the difference between weather and climate. | formal | Climate is a term used in studies that look at the climate of a particular region or area, especially in relation to weather patterns and the overall climate of a particular region or area. | | What are the benefits of regular exercise? | formal | What kind of exercise is recommended to lose weight? How long does it usually take for your body to adjust? Also, what are the risks associated with increasing your activity level? | | How should I prepare for a job interview? | formal | My interview went on for 5 hours. I don't want to waste any more time, so I was wondering how I can prepare accordingly. | | What should I consider when buying a laptop? | formal | Parts list: GPU: Intel® NVIDIA GeForce RTX 5080 16 GB Video Card CPU: AMD Ryzen 7 9800X3D 3.7 GHz 8-Core Processor AIO: Thermalright Frozen Notte 360 ... | The pattern across all 50: the model holds a register for a whole passage and stays near the topic for a sentence or two, and it behaves like a forum or chat post on the subject rather than an answer to it. That is exactly what a base model trained on community text should do, and it is the foundation an instruction-tuning stage would sit on. Like any base model trained on raw community text, it can produce rough content, so the weights are released for research use. Shipping it The model is a standard Llama architecture, so publishing it for everyone is a matter of renaming tensors, and the export proves the rename is correct: embed.weight -> model.embed_tokens.weight blocks.N.attn.wq / wk / wv -> model.layers.N.self_attn.q_proj / k_proj / v_proj blocks.N.attn.wo -> model.layers.N.self_attn.o_proj blocks.N.ffn.w_gate / up / down -> model.layers.N.mlp.gate_proj / up_proj / down_proj blocks.N.attn_norm / ffn_norm -> input_layernorm / post_attention_layernorm norm.weight -> model.norm.weight (lm_head is tied, so it is dropped) The export script loads both the original model and the converted LlamaForCausalLM and compares logits on random token ids: max difference 3.6e-05 and 100% argmax agreement. It refuses to report success otherwise. The result is on the Hub with a model card, a tokenizer and a generation config, and anyone can use it: from transformers import AutoModelForCausalLM, AutoTokenizer repo = "Ravikishan/bussin-125m" tok = AutoTokenizer.from_pretrained(repo) model = AutoModelForCausalLM.from_pretrained(repo).eval() # levels: slang 0-4, emoji 0-3, abbrev 0-3, elong 0-2, formal 0-3 prompt = "<|endoftext|><|reg|><|slang_3|><|emoji_1|><|abbrev_2|><|elong_0|><|formal_0|><|/reg|>" ids = tok(prompt, return_tensors="pt", add_special_tokens=False) out = model.generate(**ids, max_new_tokens=60, do_sample=True, temperature=0.8, top_k=50, top_p=0.95, pad_token_id=1) print(tok.decode(out[0][ids["input_ids"].shape[1]:])) The repository carries the specification, the configs, the tests, the evaluation scripts and a figure generator that draws every chart in the README from the measured results, so the charts cannot drift from the numbers. Skills this project demonstrates - ML research and problem framing. Turning a fuzzy cultural idea ("cringe") into a conditional-modelling hypothesis and a metric (RCE) that is cheap, deterministic and hard to game. - Data engineering. A dated lexicon, language routing, body-level PII scrubbing, stage-ordered sharding and a tokenizer gate. - Model engineering. A Llama-style decoder from first principles: RoPE, grouped-query attention, SwiGLU, document masking, fp32 logits, z-loss, with an analytic parameter-count check. - Training systems. WSD scheduling, multi-GPU data parallelism without the logit-gather tax, exact token accounting, hash-verified checkpoints and bit-exact resume. - Distributed-systems thinking. A lease implemented as compare-and-swap on a commit SHA, heartbeats with ownership checks, stall detection and a stateless orchestrator. - Evaluation design. Paired likelihood tests, shuffled-request baselines, single-dimension sweeps and bootstrap intervals. - Testing discipline. Behavioural tests that fail on the old code, including a test of what the model learns. - Release engineering. A tensor-renaming export with a logit-parity gate, a model card, a reproducible figure pipeline and a public repository. - Resourcefulness. About 110 GPU-hours of training on free tiers, with no paid compute. What is next The natural next layer is instruction tuning, so the register dial sits on top of an assistant that answers questions. The scaffolding is built for a bigger run: the 400M and 1B configurations are written and sized, the lease relay is platform-agnostic, and the next steps are a Gen-Z validation set so the loss curve can see the anneal, human ratings to complement the annotator, a content-safety filter ahead of a larger corpus, and a from-scratch run of the audited training stack at 400M. Everything is open: - Model: huggingface.co - Code and write-up: Ravikisha/Bussin - Raw results: docs/results/rce_final-4067.txt in the repo If you take one idea from this: make the thing you care about measurable, then make it an input. Once register was a number, it could be labelled by code, conditioned on, and evaluated, all without a single human annotation.
Discworld Noir: running on Windows, Linux & Steam Deck (unofficial ScummVM fork) I've been working on getting Discworld Noir running in ScummVM. The fork has been tested on Windows, Linux and Steam Deck, and I'd like more people to try it. You need your own copy of the English Windows version, all three CDs. No game files are included. The repository has setup instructions, source code and the Linux download: Alexbeav/discworld-noir-scummvm A portable Linux build is available to download, tested on Steam Deck and an Arch Linux (Omarchy) machine. The repository also includes the source patch; the public download described here is the Linux build. This is an unofficial fork of ScummVM's Tinsel engine. It adds the missing 3D actor movement and fixes problems with the notebook, conversations, movie audio, music, lighting and saves. Automated playthroughs using normal game inputs reached the ending during development. Those runs included changes between builds, so I can't claim a complete playthrough of the current release on one unchanged build. My manual playthrough is still in progress, and I'd like more people to try it. There are still bugs: Lewton looks too small on the Wharf, some lighting and character rendering are off, and walking can take odd routes or refuse a spot. Save manually; autosave was disabled during testing. Use the game's options for volume, as ScummVM's volume sliders don't work here. For transparency, I'm a DevSecOps engineer rather than a game programmer. AI agents wrote most of the code; I directed the work and tested the changes. This is an independent hobby project, not an official ScummVM release. ScummVM does not accept AI-written contributions, so I'm not submitting this patch upstream - but if anyone wants to base hand-written code off my results and get Discworld Noir officially supported, that's great in my book! If you still have the discs and fancy revisiting Ankh-Morpork, I'd appreciate bug reports - open an issue on the repository. Please include where it happened, what you did, and a save from just before the problem if possible.
We just released a desktop app for our open-source AI agent (Mac & Windows) Hi everyone, I'm Apollon, one of the founders of a small AI lab in Athens. Today we released the Alysis desktop app for Mac and Windows. The agent underneath it is open source (Apache-2.0) and has been available as a CLI for a while. Now you can use it without the terminal. What it does: - Works with any model: OpenAI, Anthropic, Gemini, DeepSeek, or local models through Ollama and LM Studio. You can switch per task. - Handles the non-code stuff too: spreadsheets, Word docs, slides, PDFs, research - Runs commands in a sandbox, so it can't touch things you didn't allow It's free with your own API key or a local model. If you'd rather not deal with keys, we also have very low-cost plans starting at $1/month. We're a small team, so we know there will be rough edges. I'd really like to hear what breaks and what's missing. GitHub: AlysisAi/AlysisCode Download: alysiscode.com
not everything is perfect, i'm having a bit of trouble with the licenses to make rumb propietary, the engine (Syrakon/tramuntana) is a fork from servo also it has integrated gpui from zed.