HackerNews

API de Hacker News: cómo armar tu propio lector Hacker Times apareció hace pocos días en Hacker News con un formato poco común: presenta las historias del sitio como la portada de un periódico, con ediciones por hora y enlaces directos a la fuente original. Detrás de ese diseño no hay ningún acuerdo especial con Y Combinator ni acceso privilegiado a datos: hay una API de Hacker News pública, gratuita y sin autenticación que cualquiera puede consultar desde su propia terminal. Construir algo similar no exige pedir permiso ni token de acceso. Alcanza con entender los endpoints disponibles, cómo se estructuran los datos y qué límites prácticos tiene esa interfaz cuando la usás en serio. TL;DR - La API de Hacker News vive en hacker-news.firebaseio.com/v0/ y no pide API key ni registro.- topstories.json, newstories.json y beststories.json devuelven arrays de IDs, nunca el contenido completo.- Cada item.json trae un objeto con by, score, time, title, type y url, o null si fue borrado.- Algolia replica el mismo contenido en hn.algolia.com/api con búsqueda de texto completo.- Un lector personalizado funcional entra en menos de 30 líneas de JavaScript con fetch nativo de Node.js. ¿Qué es la API de Hacker News? La API de Hacker News es la interfaz pública de solo lectura que expone en JSON cada historia, comentario y usuario del sitio de Y Combinator. Corre sobre Firebase Realtime Database, no requiere autenticación ni clave, y permite construir lectores o bots con los mismos datos que usa el sitio original. El proyecto es open source y lo mantiene el propio equipo de Hacker News en GitHub. La documentación completa, con cada endpoint y el esquema de cada tipo de dato (story, comment, job, poll, pollopt), vive en el repositorio HackerNews/API. A diferencia de muchas APIs comerciales, no hay cuotas pagadas ni planes. Es un recurso que la comunidad mantiene para la comunidad, y por eso un script que dispare miles de peticiones por segundo termina perjudicando a todos los demás clientes que comparten el mismo backend. El endpoint topstories.json devuelve hasta 500 IDs, nunca el contenido completo de cada historia. Por qué importa Antes de esta API, enterarse de qué pasaba en Hacker News significaba abrir el sitio y actualizar la pestaña a mano. Hoy existen lectores alternativos, bots de Slack, extensiones de navegador y boletines automatizados que corren sobre los mismos endpoints, sin pedir permiso a nadie. Hacker Times es un ejemplo reciente: toma las historias del día y las presenta como una edición de periódico, con portada propia y actualizaciones por hora. El proyecto se publicó como Show HN y, como cualquier otro cliente, solo necesita hacer peticiones HTTP a la misma interfaz pública que vas a usar en este artículo. También importa como caso de estudio de diseño de API: expone datos crudos sin capas de permisos, delega la búsqueda de texto a un tercero (Algolia) en lugar de construirla él mismo, y deja la presentación completamente en manos del desarrollador. Esa apertura tiene un costo, eso sí: sin llaves ni cuotas, la única barrera contra el abuso es la buena fe de quien consume los datos, así que cachear y espaciar peticiones queda en manos de cada cliente, no del servidor. Cómo funciona por dentro La API corre sobre Firebase Realtime Database, el mismo servicio de Google que usan miles de apps móviles para sincronizar datos. Eso explica por qué cada URL termina en .json: no es una convención REST elegida a propósito, es literalmente cómo Firebase expone cualquier nodo de su árbol de datos. Los endpoints se dividen en dos grupos. El primero son listas de IDs: topstories, newstories, beststories, askstories, showstories y jobstories. Cada una devuelve un array plano de números enteros, nunca el contenido de la historia. El segundo grupo son items individuales, en /v0/item/{id}.json, que trae el objeto completo de una historia, comentario, encuesta, opción de encuesta o posición de trabajo, según el valor de su campo type. Cada item tiene un set de campos fijo según su tipo. Una story trae by (usuario), score, time (timestamp Unix), title, url y kids (los IDs de sus comentarios de primer nivel). Un comment trae parent en lugar de url, y su propio kids con las respuestas anidadas. Si pedís un id borrado o marcado como muerto por moderación, los campos deleted o dead vienen en true. Hay un detalle que rompe expectativas la primera vez: pedir /v0/item/999999999.json, un id que nunca existió, no devuelve un error 404. Devuelve null con código 200. El código tiene que filtrar nulls antes de leer story.title, o el script revienta con un TypeError. curl https://hacker-news.firebaseio.com/v0/maxitem.json Ese comando confirma que la API de Hacker News está viva: devuelve un entero que crece con cada historia o comentario nuevo en el sitio, por ejemplo 45102873. Si ese número no cambia en varios minutos de actividad normal, el problema está en tu red, no en Firebase. Flujo de un lector básico flowchart TD A["Tu script"] --> B["GET /v0/topstories.json"] B --> C["Array de hasta 500 IDs"] C --> D["GET /v0/item/ID.json (por cada ID)"] D --> E["Objeto story: title, score, by, url"] E --> F["Tu lector o filtro"] Ejemplos prácticos El ejemplo más corto posible es pedir una sola historia ya publicada, para ver la forma exacta de la respuesta: curl https://hacker-news.firebaseio.com/v0/item/8863.json Esa es, literalmente, la respuesta documentada en el repositorio oficial para el id 8863: la historia con la que Drew Houston presentó Dropbox en Hacker News en 2007. { "by": "dhouston", "id": 8863, "score": 111, "time": 1175714200, "title": "My YC app: Dropbox - Throw away your USB drive", "type": "story", "url": "http://www.getdropbox.com/u/2/screencast.html" } El formato no cambió en casi veinte años. El segundo paso es traer una lista de portada completa. Con fetch nativo de Node.js, incluido desde la versión 18 sin instalar nada: const res = await fetch("https://hacker-news.firebaseio.com/v0/topstories.json"); const ids = await res.json(); console.log(ids.slice(0, 5)); Eso imprime un array de cinco enteros, por ejemplo [41234567, 41234321, 41233890, 41233654, 41233112]. Los valores cambian todo el tiempo porque la lista se reordena cuando una historia sube o baja en el ranking, pero la forma, un array plano de números, es siempre la misma. El tercer paso combina ambas llamadas para construir algo útil: una función que trae las N historias más nuevas de portada y las filtra por puntaje mínimo. async function getTopStories(limit = 10, minScore = 100) { const idsRes = await fetch("https://hacker-news.firebaseio.com/v0/topstories.json"); const ids = (await idsRes.json()).slice(0, limit); const stories = await Promise.all( ids.map(id => fetch(`https://hacker-news.firebaseio.com/v0/item/${id}.json`).then(r => r.json()) ) ); return stories.filter(story => story && story.score >= minScore); } getTopStories().then(stories => { stories.forEach(s => console.log(`${s.score} pts - ${s.title}`)); }); Promise.all dispara las N peticiones de item.json en paralelo en vez de esperarlas una por una, lo que reduce el tiempo total de varios segundos a unos pocos cientos de milisegundos. La salida, impresa con forEach, se ve así (los puntajes y títulos cambian en cada corrida real): 342 pts - Rust 1.90 ya esta disponible 218 pts - Show HN: construi mi propio lector de Hacker News 155 pts - A deep dive into io_uring > 💡 Tip: cachear el resultado de topstories.json en memoria durante 60 segundos evita golpear la API en cada render de tu interfaz sin perder frescura real. Agregar un filtro por palabra clave, para armar algo parecido a una alerta temática, es una función más encima de la anterior: function filterByKeyword(stories, keyword) { return stories.filter(s => s.title && s.title.toLowerCase().includes(keyword.toLowerCase()) ); } Secuencia de peticiones sequenceDiagram participant C as Cliente participant F as Firebase HN API C->>F: GET /v0/topstories.json F-->>C: array de hasta 500 ids loop por cada id C->>F: GET /v0/item/id.json F-->>C: objeto story o null end Note over C,F: sin autenticacion, solo HTTPS Node.js incluye fetch de forma nativa desde la versión 18, sin instalar paquetes adicionales. Cómo empezar Requisitos: Node.js 18 o superior y una conexión a internet, sin cuenta ni token de ningún tipo. - Instalá Node en Linux o macOS con nvm (nvm install 22) o el gestor de paquetes de tu distribución; en Windows, winget install OpenJS.NodeJS.LTS hace lo mismo en un solo comando.- Verificá la versión con node -v: debe imprimir v18 o más nuevo, por ejemplo v22.11.0.- Creá una carpeta y un archivo lector.js.- Pegá dentro la función getTopStories del ejemplo anterior junto con el bloque que la ejecuta.- Corré node lector.js desde la terminal.- Deberías ver una lista de líneas con puntaje y título, una por historia que pasó el filtro de score mínimo. Casos de uso reales - Lectores alternativos con diseño propio: Hacker Times es el ejemplo que abrió este artículo, con formato de periódico y ediciones por hora, pero el mismo patrón sirve para un lector minimalista en terminal.- Búsqueda de texto completo: hn.algolia.com es, en sí mismo, un lector construido por Algolia sobre los mismos datos, con un índice de búsqueda que la API oficial no ofrece.- Bots de chat: scripts que postean en Slack, Discord o Telegram las historias que superan cierto puntaje apenas aparecen en topstories.- Dashboards de tendencias: acumular topstories cada hora para graficar qué temas, como IA o seguridad, dominan la semana.- Archivos personales: guardar en una base local las historias con más de X puntos, para tener un archivo histórico propio sin depender del sitio. Errores comunes y buenas prácticas - No filtrar nulls: item.json puede devolver null; acceder a .title sobre null tira un TypeError.- Ignorar dead y deleted: mostrarlos igual puede confundir al usuario final; lo normal es ocultarlos o marcarlos como eliminados.- Pedir items en serie: un for con await secuencial multiplica por N el tiempo de respuesta frente a Promise.all.- Asumir orden estable: topstories.json cambia constantemente, así que para comparar posiciones hay que trackear por ID, no por índice del array.- No cachear: pedir el mismo item.json varias veces en segundos desperdicia peticiones; unos minutos de caché alcanzan para un lector personal.- Polling agresivo: no hay rate limit documentado, pero eso no es invitación a golpear el servidor cada 100 ms; 30 a 60 segundos sobra para cualquier lector humano. > ⚠️ Ojo: un id inexistente responde 200 con body null, no 404. Validar explícitamente ese caso evita que tu lector se caiga entero por una sola historia borrada. Comparativa con alternativas La API oficial no es la única puerta a los datos de Hacker News. Elegir entre ella, Algolia y el scraping depende de si necesitás búsqueda, qué tan en tiempo real querés los datos y cuánto mantenimiento estás dispuesto a asumir. OpciónCuándo usarlaVentajaLimitaciónAPI oficial (Firebase)Lector básico, bots, dashboardsGratis, sin auth, datos casi en tiempo realSin búsqueda de texto; un request por itemAlgolia HN Search APIBuscar por palabra clave o rango de fechasBúsqueda full-text con filtros por fecha y tipoNo la mantiene YC directamente, sino AlgoliaScraping HTMLDatos que ningún endpoint expone (orden visual exacto)Acceso a cualquier dato visible en la páginaRompe con cada cambio de diseño del sitio Profundizando Todo lo anterior usa polling: pedís datos, esperás la respuesta, repetís cada N segundos. Pero como la base es Firebase Realtime Database, existe una alternativa que empuja los cambios en vez de que el cliente los pida: suscribirse directamente con el SDK de Firebase. El propio repositorio de la API de Hacker News documenta esta vía: en lugar de pedir REST, el cliente Firebase se conecta al proyecto y usa .on('value', callback) sobre un nodo como v0/item/{id}, de forma que el callback se dispara solo cuando ese nodo cambia, sin que el código tenga que volver a preguntar. Para detectar cambios sin suscribirse nodo por nodo existe además /v0/updates.json, que devuelve dos arrays: items (IDs de historias o comentarios que cambiaron recientemente) y profiles (usuarios que actualizaron su perfil). Para confirmar que realmente refleja actividad reciente, pedilo dos veces con un minuto de diferencia: con tráfico normal, el array items debería traer IDs distintos en cada corrida. La contrapartida de usar el SDK de Firebase en vez de REST es sumar una dependencia más y un modelo de conexión persistente, en lugar de simples peticiones HTTP sin estado. Para un bot simple o un script que corre una vez al día, fetch alcanza; para un dashboard que necesita reflejar cambios en vivo, la suscripción evita miles de peticiones repetidas. stateDiagram-v2 [*] --> Polling Polling --> Polling : GET cada N segundos Polling --> Desactualizado : intervalo largo [*] --> Suscripcion Suscripcion --> Actualizado : push automatico via SDK Firebase Tu próximo paso: cloná la función getTopStories de este artículo, modificala para guardar en un archivo JSON local las historias con score mayor a 300, y programala con cron para correr una vez por hora. 📖 Resumen en Telegram: Ver resumen Preguntas frecuentes ¿El endpoint público de Hacker News necesita una API key? No. Es de solo lectura, pública y no exige registro, token ni cabecera de autenticación: cualquier petición GET a hacker-news.firebaseio.com/v0/ funciona directo desde el navegador o la terminal. ¿Con qué frecuencia se actualiza topstories.json en Hacker News? No hay un intervalo fijo documentado: el array se reordena cada vez que el algoritmo de ranking del sitio recalcula posiciones, lo que en horas de tráfico alto puede ser cada pocos minutos. ¿La interfaz REST de Hacker News permite buscar por palabra clave? No de forma nativa. Para búsqueda de texto completo, por fecha o por autor, el proyecto comunitario Algolia expone los mismos datos indexados en hn.algolia.com/api. ¿Qué significa que un item de Hacker News tenga dead o deleted en true? dead indica que la historia o el comentario fue removido por moderación pero su ID sigue existiendo; deleted indica que el propio autor lo borró. En ambos casos item.json puede devolver el objeto con esos flags. ¿Hay límite de peticiones por segundo en la Hacker News API? No hay un rate limit documentado públicamente, pero al ser un servicio gratuito mantenido por la comunidad, lo correcto es cachear resultados y espaciar las peticiones: abusar de la frecuencia perjudica a todos los demás clientes que comparten el mismo backend. Referencias - HackerNews/API en GitHub: documentación oficial de endpoints y esquema de datos.- Algolia HN Search API: búsqueda de texto completo sobre el mismo contenido.- Hacker Times: el lector en formato periódico que inspiró este artículo.- Firebase Realtime Database: documentación del servicio sobre el que corre la API. 📱 ¿Te gusta este contenido? Únete a nuestro canal de Telegram @programacion donde publicamos a diario lo más relevante de tecnología, IA y desarrollo. Resúmenes rápidos, contenido fresco todos los días.

@minchoi The download step is easy because HN has an official API with no rate limit. You walk back from /v0/maxitem and pull every comment, no scraping needed HackerNews/API

HN's Firebase-based API (hackernews/api) makes that easy, until you want to reconstruct the threads (i.e. who replied to who) and then it's annoying. For years* I've wanted to replace that API with one we operate ourselves, which will simply return a JSON version of any HN URL. I've finally started working on this! Perhaps my favorite thing about LLMs is how they make projects possible which would otherwise not have gotten started. Edit: perhaps I should explain what I mean by that last bit. The way I want this new API to work is (1) HN should use it internally to generate HTML pages, and (2) there can't be any regressions - nothing should change in the web version. This is a pain to get right, as it's easy subtly alter the details, and it's annoying to regression-test everything. Also, (3) with a design like this, it is easy—and critical not to—leak data that ought to remain private (e.g. people's IP addresses) This is the sort of pain and annoyance that LLMs are amazing at taking care of. I get to focus on the parts I want to think about—the design and (yes) the code—and have a tireless assistant do the gruntwork and review. Without this I don't think I'd have gotten around to working on it. * (edit: over a decade in fact - see collapsed reply)

The Loudest Stories Were Missing from the Map: Learning Topics on Hacker News The five highest-scoring submissions in the six-day comparison all received the same topic: Unknown. That is an awkward opening for a topic-modeling article. It is also the most useful result. Among 5,997 Hacker News submissions dated October 24–29, 2026, a small model assigned 4,432 to no learned topic. Those unknown titles carried 81.55% of the observed points. A chart showing only the named groups would have hidden most of the conversation—and most of its measured attention. The question was straightforward: which subjects appear on HN, which attract attention, and how does the picture differ across sources and periods? Answering it required more than finding words such as “AI.” It required keeping the model's vocabulary, the source population and the popularity calculation separate. This case uses real submissions, a small trained model, and independently callable steps. The examples below run through Splime Public Hub and produce an HTML report with topic summaries, tags, collection details and a linked reading list. Start with the population. The collector retrieved 13,282 unique stories created from October 17 at 00:00 UTC to October 30 at 00:00 UTC, with the end excluded. Collection ran on October 31 from 11:01:59 to 11:03:03 UTC. Training uses the earlier seven days; the later six days supply the descriptive comparisons and temporal evaluation. For historical dates, the implementation queries HN Search, splitting busy time intervals until each query fits within one returned page. This run needed 83 requests and exhausted the requested intervals without reaching the item limit. That establishes retrieval coverage of the search index, not a guarantee that every HN submission is indexed. The returned snapshot records its bounds, limits and collection timestamps. Current Firebase feeds answer a different question. The official API supplies bounded lists of current top, new, Ask and Show stories; filtering those lists by date does not recreate a past front page. The collector exposes both routes and describes their coverage explicitly. Network failures raise errors; they never produce substitute records. Official HN API documentation, HN Search API. “Direction” is therefore not a field in the model. A topic is a learned word group. A source is a feed or index membership. In the historical data, story includes ask_hn and show_hn; these sources overlap. Each HN ID counts once overall, while its memberships remain available for comparisons. Learn a small map of words. An earlier baseline, analyze_titles@1, applied supplied phrase rules. It was explainable, but it had no training step. The new model starts with titles and learns its own word groups. First, fixed text cleanup removes common function words and source prefixes such as “Show HN.” TF-IDF represents each title using words and adjacent word pairs. A term gets more weight when it is informative within the training collection. The vocabulary and document-frequency weights are fitted only on the earlier partition. Next, nonnegative matrix factorization, or NMF, approximates those title vectors with eight reusable word groups. Think of each title as a mixture of a few word lists, with a different weight on each list. This is a standard, inspectable topic-extraction approach; the scikit-learn example illustrates the same TF-IDF/NMF pairing. The baseline caps the vocabulary at 600 terms. Eight topics and that small vocabulary are demonstration choices, not a tuned optimum. From 7,285 earlier submissions, training removed 122 titles with too little usable text and 255 repeated normalized titles. The remaining 6,908 titles supplied the learned state; 6,175 had a nonzero vector in the selected vocabulary. NMF converged after 37 iterations. The topic names are the four highest-weight learned terms, printed without editorial polishing. This matters: open / source / open source / alternative is a useful clue, while video / php / world / context is a diffuse group. Singular agent and plural agents became separate components. The model has learned associations in this corpus, not a comprehensive taxonomy of technology. For a new title, the vocabulary, IDF and components stay fixed. A nonnegative least-squares fit estimates the title's mixture of those components. The strongest component supplies a primary topic when the rejection rules allow it. Very short titles, too few known words, weak matches or ambiguous mixtures become Unknown. Those thresholds are heuristics, not calibrated confidence scores. Tags are even more literal: they are high-weight terms from the assigned topic that occur in the cleaned title; word pairs are adjacent after stop-word removal. A title may receive agent and skills; it does not inherit every word in its topic's name. Nothing in training or inference uses points, comment counts, source membership or future activity as a feature. The map covers a recognizable corner of HN. Here is the later six-day population under that single frozen model: | Automatic topic name | Stories | Share of all stories | Observed points | |---|---:|---:|---:| | Unknown | 4,432 | 73.90% | 95,396 | | ai / ai agents / ai agent / slop | 558 | 9.30% | 9,423 | | agents / ai agents / coding / coding agents | 185 | 3.08% | 1,437 | | code / claude / claude code / codex | 170 | 2.83% | 3,085 | | open / source / open source / alternative | 155 | 2.58% | 1,782 | | agent / ai agent / coding / coding agent | 149 | 2.48% | 832 | | video / php / world / context | 146 | 2.43% | 1,541 | | browser / data / game / pdf | 133 | 2.22% | 1,721 | | model / language / world / first | 69 | 1.15% | 1,758 | The named groups make the corpus look dominated by AI and developer tools. Yet their combined coverage is only 26.10%. For example, the HN submission titled “Anthropic S-1” had 1,708 observed points and remained unknown. So did “We're releasing the weights of Mistral Large 4”, with 1,694 points. The topic map misses even some conspicuous AI-related titles. The five highest-point records were checked against the official Firebase API: their IDs, titles, creation times and observed scores matched. This verifies the submissions and metadata, not the truth of claims made in their headlines. Counts, shares and engagement totals were also independently recomputed from the collected records. A rise in submissions is a different result from a rise in attention. The comparison uses two equal 72-hour windows: October 24–26, with 2,532 stories, and October 27–29, with 3,465. The broad ai / ai agents / ai agent / slop group moved from 223 to 335 stories, or 8.81% to 9.67% of each population: +0.86 percentage points. The singular-agent group rose by 0.68 points. Unknown stayed near three quarters of both windows. !Image description These are changes in the composition of observed submissions. They are not evidence that a topic is gaining users, that sentiment improved, or that its next post will become popular. Both periods use the same model fingerprint. Comparing “topic 3” from two independently trained models would make the numbers look comparable while changing their meaning. Points and comments provide another view. The broad AI group accumulated 9,423 points and 6,835 comments. Its highly discussed examples include “AI promised less work. I've never worked more”, with 684 points and 662 comments, and “No signs of AI payoff outside the tech sector”, with 653 points and 678 comments. These titles suggest debate about AI's practical consequences; comment counts alone cannot establish approval or sentiment. Every topic's median story score was just two. Totals are sensitive to a few exceptional posts. Counts measure publishing volume; shares account for different submission totals; points summarize observed net scoring; comments measure discussion. Missing engagement remains missing and the report shows how many values were observed. Age is another constraint. At collection, the median story was 136.5 hours old in the earlier window and 67.6 hours old in the later one. The older group had more time to accumulate activity. These points and comments were collected after the posting periods, so they are not historical “as of midnight” totals. The case describes accumulated attention and does not claim an age-adjusted trend or a popularity forecast. Ask and Show reveal different coverage. Among the later stories, 179 carried ask_hn and 858 carried show_hn. Unknown accounted for 70.39% of Ask HN, 59.32% of Show HN and 73.90% overall. The open/source group represented 7.11% of Show HN, compared with 2.58% overall; the plural-agents group also represented 7.11% of Show HN. !Image description This is a useful source contrast, although it partly reflects vocabulary coverage: Show HN titles more often match these learned groups. It is not proof that the unassigned stories lack themes. The overall column already includes Ask and Show, so adding their counts to it would count some submissions twice. Inspect errors before celebrating coverage. The temporal evaluation held out submissions created on or after October 24. It conservatively removed 263 later titles whose normalized text or canonical external URL appeared in training, leaving 5,734 evaluation records. That purge applies to evaluation; descriptive counts retain distinct later HN submissions, even when they discuss the same external article. On that purged holdout, the model assigned 25.99% and rejected 74.01%. Mean known-token coverage was 37.38%. These figures describe the model's reach, not its semantic accuracy. No independent human topic labels were available, and no rule-generated labels were used to manufacture an accuracy score. A qualitative review examines deliberately difficult examples, selected by large reconstruction residuals. It is an assistant inspection of titles, not human ground truth or a representative accuracy study. Several errors are easy to understand: - “Open a local LLM as a 3D graph…” landed in the open/source group because “open” can be a verb. - An [Apple iOS-support title ending in “[video]”](news.ycombinator.com) entered the video group because a format marker acted like a subject. - HSBC's data-center debt title matched the browser/data/game/pdf group through “data,” although the title concerns finance. These are specific reasons to improve the representation. A larger vocabulary, phrase handling, longer text or a better tokenizer may help. The next evaluation should use independent human review of future titles. Raising the assignment rate by weakening rejection would not, on its own, demonstrate improvement. Repeat the computation one step at a time. Use Python 3.13 and install these tested package versions in the environment used by your notebook: python -m pip install splime==0.4.13 httpx==0.28.1 numpy==2.3.5 \ scipy==1.18.0 scikit-learn==1.8.0 ipython==9.18.0 An SPH consumer needs no registration or API key. In embedded mode the published code is downloaded and runs locally; trust=True explicitly permits its execution. The following version table is generated only from confirmed publication receipts: | Public object | Confirmed immutable version | |---|---| | fetch_snapshot | [fetch_snapshot@2](splime://@splime.project/case4-hn/fetch_snapshot@2) | | normalize_stories | [normalize_stories@2](splime://@splime.project/case4-hn/normalize_stories@2) | | train_topic_model | [train_topic_model@1](splime://@splime.project/case4-hn/train_topic_model@1) | | predict_topics | [predict_topics@1](splime://@splime.project/case4-hn/predict_topics@1) | | evaluate_topics_temporally | [evaluate_topics_temporally@2](splime://@splime.project/case4-hn/evaluate_topics_temporally@2) | | summarize_topics | [summarize_topics@2](splime://@splime.project/case4-hn/summarize_topics@2) | | hn_topic_model | [hn_topic_model@1](splime://@splime.project/case4-hn/hn_topic_model@1) | | tag_titles | [tag_titles@1](splime://@splime.project/case4-hn/tag_titles@1) | | render_digest | [render_digest@3](splime://@splime.project/case4-hn/render_digest@3) | | hn_topic_digest | [hn_topic_digest@4](splime://@splime.project/case4-hn/hn_topic_digest@4) | In a notebook, set up the client and exact references: from spl import SPLClient from IPython.display import HTML, display client = SPLClient.embedded(registry_url="https://splime.io", run_receipts=False) base = "splime://@splime.project/case4-hn" Fetch and normalize independently. Collection limits are visible rather than hidden in a utility: snapshot = client.call( f"{base}/fetch_snapshot@2", kwargs=dict(source="algolia", feeds=["story"], start_utc="2026-10-17T00:00:00Z", end_utc="2026-10-30T00:00:00Z", max_items=20000, page_size=500, max_requests=200, timeout=15, attempts=3, budget_seconds=300), trust=True, ).value assert not snapshot["source_info"]["truncated"], "Increase limits or narrow the period" snapshot = client.call( f"{base}/normalize_stories@2", kwargs=dict(records=snapshot["records"], source_info=snapshot["source_info"]), trust=True, ).value Training remains an explicit operation. Its result is a JSON-compatible model containing the learned vocabulary, weights and automatic labels: earlier = dict(snapshot, records=[r for r in snapshot["records"] if r["created_at_utc"] < "2026-10-24T00:00:00+00:00"]) model = client.call( f"{base}/train_topic_model@1", kwargs=dict(snapshot=earlier, n_topics=8, max_features=600, random_state=42), trust=True, ).value The published predictor includes one frozen model as a required native scalar dependency. Its learned state is serialized as JSON text inside the signed release, and a restore step checks the model fingerprint before prediction. The bundle carries the model itself; it does not retrieve an author's file or retrain when called. The consumer uses tag_titles with only a snapshot; its bound model is mandatory. The model artifact can also be fetched independently through hn_topic_model, while predict_topics accepts an explicit model for experiments. Consumers of the fixed pipeline never search for a local model file, pass one manually or retrain. Repeated live queries can change as the index and scores change. Every comparison here uses the same frozen model, while the reported totals refer to the collection time stated above. later = dict(snapshot, records=[r for r in snapshot["records"] if r["created_at_utc"] >= "2026-10-24T00:00:00+00:00"]) predictions = client.call( f"{base}/tag_titles@1", kwargs={"snapshot": later}, trust=True, ).value["result"]["default"] analysis = client.call( f"{base}/summarize_topics@2", kwargs={"predictions": predictions, "window_hours": 72, "anchor_utc": "2026-10-30T00:00:00Z"}, trust=True, ).value html = client.call( f"{base}/render_digest@3", kwargs={"analysis": analysis, "limit": 12}, trust=True, ).value display(HTML(html)) The HTML is an escaped fragment with container-scoped CSS, so displaying it does not restyle the notebook. Use full_document=True only when saving a standalone page. Finally, the ready pipeline does collection, preparation, fixed-model inference, popularity calculation and rendering in one call. Training is absent: html = client.call( f"{base}/hn_topic_digest@4", kwargs=dict(source="firebase", feeds=["newstories", "topstories"], max_items=15, max_requests=50, timeout=15, attempts=2, budget_seconds=180, limit=10), trust=True, ).value["result"]["default"] display(HTML(html)) All ten releases passed an anonymous run in a clean Python 3.13 environment with a fresh cache on 1,210 real HN stories. Its 15 checks exercised the separate stages, the complete one-call pipeline, fixed-model identity and empty-input behavior. The companion notebook also completed all 17 code cells against a fresh 13,282-story historical download and a current-feed pipeline call. The experiment leaves a practical reading tool and an equally practical warning about its interpretation. The learned groups reveal recurring vocabulary, source differences and modest changes in submission share. Keeping unknowns visible shows exactly where that map stops helping. A useful report makes those boundaries as easy to inspect as its most appealing chart.

I checked 11 launch platforms for an llms.txt. Four answered, one sent a challenge page On October 19 I ran a small, boring test: fetch /llms.txt from the root of 11 launch platforms and directories and see who answers. Ten days later, on October 29, 2026, I ran it again before writing this up. The picture had already moved. llms.txt is a proposal for a plain markdown file at a site's root that tells a language model what the site is and where the useful pages are. Nobody is required to publish one, and Google's own guidance says you don't need new AI text files to appear in its AI features. So a missing file isn't a failing. I just wanted to know what an agent finds if it goes looking, because more of my own work now starts with an agent fetching a URL. How I checked One curl per domain, following redirects, with an ordinary desktop Chrome user agent: curl -sL -A "Mozilla/5.0 ... Chrome/130.0.0.0 Safari/537.36" \ -o /dev/null -w '%{http_code} %{content_type} %{url_effective}' \ https://<domain>/llms.txt I also tried /llms-full.txt, the longer companion file some sites publish. I counted a site as answering only when it returned a 200 with text/plain. A 404 counts as "no file". A 403 challenge page counts as "blocked the fetch", not "missing", because those are different things. What came back | Platform | /llms.txt | What the file says | |---|---|---| | DevHunt | 200, text/plain, 6 KB | What DevHunt is, free listing vs. paid launch options, key pages, URL patterns for tool, alternatives and compare pages, categories. No submission API listed. | | TinyLaunch | 200, text/plain, 42 KB | An "Agent Guide": a full submit flow over its REST API, email OTP auth, maker profile, startup, launch scheduling, categories, limits, error codes. | | BetaList | 200, text/plain, 3 KB | What gets featured, links to an Agent API for drafting a submission without an account, which a human founder then claims; submission criteria and help pages. | | Uneed | 200, text/plain, 792 KB | What Uneed is, a read-only MCP server and REST API for product data, a link to an agent launch guide (email OTP), then long lists of products, reviews and alternatives. | | Product Hunt | 403 to curl (Cloudflare challenge) | Not readable with plain curl today. A different fetcher did get it: site overview, URL patterns, leaderboards, categories, and an attribution line. | | Hacker News | 404 | No file | | Fazier | 404 | No file | | Open Launch | 404 | No file | | SaaSHub | 404 | No file | | Peerlist | 404 | No file | | Microlaunch | 404 | No file | On /llms-full.txt: Uneed serves one (about 9 MB of text). TinyLaunch's path redirected to a login page, which returns 200 but is HTML, so I don't count it. Everyone else returned a 404 or the same challenge page. What changed in ten days Three results flipped since October 19: - BetaList returned a 404 on the 19th. Today it serves a short, well-organized file. - Uneed blocked my fetch on the 19th with a challenge page. Today it answered with the largest file in the set. - Product Hunt answered on the 19th. Today plain curl got a Cloudflare "Just a moment..." page with cf-mitigated: challenge, on both the apex and www. The file is still there; the same URL fetched through a different tool came back fine, with links dated today. I don't know why my request was challenged this time, only what came back. That last one is the practical lesson. Whether an agent "can read" a site isn't a property of the site alone. It depends on who is asking, from where, and on which day. A single check is a snapshot. Three kinds of file Reading the four files that answered, they're doing different jobs. The map. DevHunt's file, and Product Hunt's when it's reachable, explain what the site is and how its URLs are shaped: /tool/{slug}, /compare/{a}-vs-{b}, /products/{slug}/alternatives, daily and weekly leaderboard paths. That's what an assistant needs to answer "what are alternatives to X" by going straight to the right page instead of guessing. Product Hunt's also asks for something back: list a product's name and tagline first, and attribute the data to Product Hunt. The runbook. TinyLaunch's file opens with "You are launching a startup on TinyLaunch on behalf of a user, end-to-end, without a browser." It then lists the eight endpoints the flow needs, which ones need a bearer token, and how to handle the email code. It even tells the agent to use its host's structured-question tool rather than printing options as text. That's written for a coding agent mid-task, not for a crawler. The handoff. BetaList's file points to an Agent API where an agent drafts the submission and a human founder claims it. Uneed's does both a read side, an MCP server and REST API for its product data, and a write side, a separate launch guide for agents. I checked that the linked pages exist (betalist.com/agent and uneed.best/launch.txt both returned 200). I did not run either flow. What none of the files did was hide the human step. Every write path I saw still routes through a person: an emailed code, or a founder claiming the draft. I think that's the right shape. What the 404s mean Not much on their own. Hacker News is a discussion site with an official read API documented on GitHub. The others have perfectly readable HTML pages, and a search crawler doesn't need a root markdown file to index them. The 404 only matters in one narrow case: an agent that checks /llms.txt first will learn nothing about how submissions work there and will fall back to reading the HTML or asking you. What I'm taking from it - If you run a site that takes submissions, the most useful thing an llms.txt can carry is the submission path, plainly written, including the step that needs a human. - Write it for the agent that will act on it, not just the crawler that indexes it. TinyLaunch's step list and error codes are a good model of that. - Re-check anything you cite. Three of eleven results changed in ten days, and one of them seems to depend on how it was fetched. - Keep the raw responses. I saved every body and header from today's run, which is how I could tell a missing file from a challenge page. The file I keep for LaunchSignal at launchsignal.ai is held to the same test: it names the JSON API and MCP submit paths, the error codes, and the fact that a submission stays pending until a person reviews it.

HN Algolia API Reference: Search, Tags, Filters, Limits The Hacker News Algolia API is a free, keyless REST API at https://hn.algolia.com/api/v1/. Use /search for relevance order or /search_by_date for newest first. Filter with tags (story, comment, show_hn, ask_hn, author_NAME, story_ID) and numericFilters on points, num_comments and created_at_i. One query returns at most 1,000 hits. This page is the reference I wanted when I built on it: every endpoint and parameter, what each one does, and the limits. Each example below was run against the live API on 2026-10-14. The official documentation is the HN Search API page, which is short; Algolia's general search parameters apply on top of it. The four endpoints | Endpoint | Returns | Notes | |---|---|---| | GET /api/v1/search?query=... | Hits sorted by relevance, then points, then number of comments | The official page gives that sort order. | | GET /api/v1/search_by_date?query=... | Hits sorted newest first | Same parameters as /search. | | GET /api/v1/items/:id | One item with its whole comment tree nested in children | A missing ID returns HTTP 404 with {"error":"Not Found","status":404}. | | GET /api/v1/users/:username | username, karma, about | A missing user returned HTTP 500 (an HTML error page), not 404. | Every response is JSON and the API sends Access-Control-Allow-Origin: *, so browser fetch() works without a proxy. Search parameters | Parameter | Example | What it does | |---|---|---| | query | query=sqlite | Full-text search. Wrap a phrase in double quotes for an exact phrase: "rust compiler" matched 202 stories, against 1,068 for the two words unquoted. | | tags | tags=story | Filters by type, author or story. See the next section. | | numericFilters | numericFilters=points>100 | Filters on created_at_i, points or num_comments with <, <=, =, >, >=. | | page | page=2 | Zero-based page number. | | hitsPerPage | hitsPerPage=100 | Defaults to 20. Values above 1,000 are cut to 1,000. | | restrictSearchableAttributes | restrictSearchableAttributes=url | The official page's example for matching URLs only. | | attributesToRetrieve | attributesToRetrieve=title,points | Algolia parameter. Trims each hit to those fields (plus objectID and _highlightResult). | | typoTolerance | typoTolerance=false | Algolia parameter. Turns off fuzzy matching. | The official page says "The complete list of search parameters is available in Algolia Search API documentation." The two Algolia parameters above are the ones I tested; others may be restricted by the index settings. Tags The official list is story, comment, poll, pollopt, show_hn, ask_hn, front_page, author_:USERNAME and story_:ID. In practice you write author_pg and story_8863, with no colon. | Tag | Matches | Tested | |---|---|---| | story | Submissions (including Show HN and Ask HN, which also carry show_hn or ask_hn) | yes | | comment | Comments | yes | | show_hn, ask_hn | Show HN and Ask HN posts | yes | | front_page | Stories on the front page right now | 30 hits; 29 of them were also in the Firebase topstories top 30 at the same moment | | poll, pollopt | Polls and poll options | yes | | author_pg | Items by user pg | 10,723 items | | story_8863 | The story and every comment in its thread | 72 hits (1 story + 71 comments) | | job | Job posts | Works, though it isn't in the official list | | launch_hn | Launch HN posts | Works, though it isn't in the official list | Combining tags. Commas mean AND. Parentheses mean OR. The official example: tags=author_pg,(story,poll) means author is pg AND (type is story OR poll). tags=story,author_pg stories by pg (699 hits) tags=author_pg,(story,comment) stories or comments by pg (10,684 hits) tags=comment,story_38309611 every comment in one thread (2,530 hits) numericFilters The filterable fields are created_at_i (Unix seconds), points and num_comments. Commas mean AND. Parentheses mean OR, as with tags. # Stories with more than 1,000 points: 2,684 hits curl -s "https://hn.algolia.com/api/v1/search?tags=story&numericFilters=points%3E1000&hitsPerPage=3" # At least 500 points AND at least 500 comments: 3,593 hits curl -s "https://hn.algolia.com/api/v1/search?tags=story&numericFilters=points%3E%3D500,num_comments%3E%3D500" # More than 5,000 points OR more than 4,000 comments: 7 hits curl -s "https://hn.algolia.com/api/v1/search?tags=story&numericFilters=(points%3E5000,num_comments%3E4000)" %3E is > and %3D is =. Python's requests and JavaScript's URLSearchParams encode these for you. Watch out: a misspelled field doesn't raise an error. numericFilters=score>10 returned HTTP 200 with 0 hits. HN's Firebase API calls the field score, but here it is points. The response { "hits": [ ... ], "nbHits": 60688, "page": 0, "nbPages": 500, "hitsPerPage": 2, "exhaustiveNbHits": false, "query": "rust", "params": "query=rust&tags=story&hitsPerPage=2&advancedSyntax=true&analyticsTags=backend", "processingTimeMS": 18 } That was query=rust&tags=story&hitsPerPage=2. Note advancedSyntax=true in params: the server adds it, which is why quoted phrases work. Fields in each hit: | Field | Story | Comment | Notes | |---|---|---|---| | objectID | yes | yes | The HN item ID, as a string. | | title, url | yes | | url is missing for Ask HN text posts. | | story_text | Ask HN, text posts | | HTML. | | comment_text | | yes | HTML. | | author, created_at, created_at_i | yes | yes | created_at_i is Unix seconds. | | points, num_comments | yes | | Null on comments. | | story_id, parent_id | story_id | both | Comments also carry story_title and story_url. | | children | yes | | IDs of the top-level replies only. | | updated_at | yes | yes | When Algolia last refreshed the record. | | _tags, _highlightResult | yes | yes | Tags, and the matched text wrapped in <em>. | Don't treat nbHits as a count when exhaustiveNbHits is false. On 2026-10-14, tags=comment with no query reported 303,748 hits, while tags=story reported 45,522,013. HN has far more comments than stories, so both are estimates. With a narrow filter, exhaustiveNbHits came back true and the count matched the real total: a one-day window summed correctly across seven days. The 1,000-hit cap nbPages never goes past what 1,000 hits allow. Asking for page 50 at 20 per page (hit 1,001 onward) returned HTTP 200, no hits, and this message: you can only fetch the 1000 hits for this query. You can extend the number of hits returned via the paginationLimitedTo index parameter or use the browse method. You can't change paginationLimitedTo or use browse on someone else's index, so the fix is to narrow the query. Split it into created_at_i windows that each hold 1,000 hits or fewer. Search Hacker News by date range or points has a script that splits windows automatically. Python: a small client import time import requests BASE = "https://hn.algolia.com/api/v1" s = requests.Session() def hn(path, **params): for attempt in range(4): r = s.get(f"{BASE}/{path}", params=params, timeout=30) if r.status_code == 200: return r.json() if r.status_code == 404: return None time.sleep(2 ** attempt) # 1, 2, 4 s on 429 or 5xx r.raise_for_status() top = hn("search", tags="story", numericFilters="points>1000", hitsPerPage=3) for h in top["hits"]: print(h["points"], h["num_comments"], h["title"]) latest = hn("search_by_date", query="sqlite", tags="comment", hitsPerPage=1)["hits"][0] print(latest["created_at"], latest["story_title"]) print(hn("users/pg")) print(hn("items/999999999")) Output on 2026-10-14 (the comment line changes every few minutes): 6015 436 Stephen Hawking has died 5771 967 A Message to Our Customers 5710 2530 OpenAI's board has fired Sam Altman 2026-10-14T15:01:57Z Strafe: A new open-source, E2E Encrypted, and federated alternative to Discord {'about': 'Bug fixer.', 'karma': 157316, 'username': 'pg'} None With no query, /search puts the highest-scoring stories first, so it doubles as an "all-time top stories" endpoint. Limits and behaviour | Topic | What I found | |---|---| | Rate limit | "10,000 per hour" per IP, from the official API page. | | Authentication | None. | | CORS | Access-Control-Allow-Origin: *. | | Freshness | The newest item in /search_by_date had the same ID as Firebase's maxitem, 27 seconds after it was posted. Points and comment counts on the five top stories matched the Firebase API exactly. | | Page size | Default 20, maximum 1,000. | | Depth | 1,000 hits per query. | | Missing user | HTTP 500, so treat a 500 from /users/ as "maybe missing", not only as an outage. | Related guides for this API - The Hacker News Search API: free, no key, the shorter introduction. - Get all comments for a Hacker News story, for /items and the comment tree. - Hacker News Firebase API vs Algolia, for which API to use when. - Search Hacker News by date range or points, for queries past 1,000 hits. Do it without code The free Hacker News search tool calls this API live from your browser. Type a keyword, pick a type (story, comment, Show HN, Ask HN), sort by relevance or date and set a minimum score. You get real results with points, comments, author and date, and no key. At scale The Hacker News Scraper actor on Apify uses both official APIs. It has nine modes: top, new, best, ask, show and job lists, keyword search, a user's submissions, and the monthly "Who is hiring?" threads as job rows. It can attach comment trees and author karma, filter by score, comment count, date and domain, and export JSON, CSV or Excel. It's free to start, then pay-as-you-go. Disclosure: I build the datatooly tool and the Apify actor. Everything above works with the public API alone. Sources: HN Search API (official page), Algolia search API parameters, Hacker News API (Firebase) on GitHub, and live responses from hn.algolia.com/api/v1 on 2026-10-14.

Export a Hacker News thread without claiming the comment tree is complete <!-- apify-article:hacker-news-stories-comments-scraper-evidence-guide-20261021 --> A comment export can contain valid stories and comments while leaving part of the thread unvisited. That is not a contradiction. Individual items and traversal coverage are separate units, and a useful research dataset should preserve evidence about both. The Hacker News stories and comments Actor reads items and bounded feeds through the official Hacker News API. It retains source IDs and parent relationships. It does not fetch the external articles linked from stories, so the dataset describes the discussion source rather than the full linked publication. Start with one historical item The installed prefill uses item 8863 as a known historical story. With includeComments disabled, this is a small input for checking the item envelope. Values such as scores and descendant counts remain source snapshots rather than timeless assertions about that story. { "itemIds": [ 8863 ], "feed": "none", "includeComments": false, "maxDepth": 2, "maxItems": 20, "keywords": [] } You can supply up to twenty item IDs. Alternatively, a selected current feed requires an empty itemIds array. The feed modes are bounded current selections, not a historical search index. Preserve the discussion graph Keep each item's id, parent, rootId and relative depth. Those fields let a consumer reconstruct relationships after sorting or exporting the dataset. A chronological list of comment text alone loses which statement each reply addressed. Preserve deleted, dead and source-status evidence where it is returned. Do not replace a missing or deleted item with an empty comment that looks like ordinary content. Source identity is still useful when the requested content is unavailable. Bound traversal deliberately When comments are enabled, the runtime traverses child comments breadth-first within the depth and item limits. maxDepth ranges from zero through three, and maxItems can allow up to five hundred complete items. An additional HTTP work cap and working deadline also bound source work. A free partial notice explains known remaining children or descendants excluded by the selected depth. Complete items already delivered can remain useful while the thread set is incomplete. Retain the notice next to the exported item collection instead of discarding it during a found=true filter. Understand what keyword filtering means keywords supports a small list of case-insensitive substrings applied to fetched titles and HTML text. It is not a global search over all past Hacker News content. An empty keyword result therefore describes the items actually checked within the chosen source scope. If a research question needs a full historical population, identify a source and collection method that can establish that population. A bounded feed snapshot can support exploratory reading, but its convenience should not turn into an unsupported coverage claim. Count content and notices separately The result unit is one complete story or comment. Free status rows do not consume the same result allowance, so a dataset can contain more rows than maxItems. Count complete content separately from diagnostic and coverage rows when reporting the export. For a handoff, provide the input, run identity, complete-item count and remaining-scope notices. That lets another analyst see exactly which discussion evidence was collected and which portions were left outside the export. A clear coverage statement is more useful than calling every nonempty thread dataset complete. Cost and verification scope The current FREE-tier result price is $0.0003 for the documented unit: One complete Hacker News story or comment with its metadata. The listing also has a $0.005 start event, charged per GB of Actor memory with a minimum of one event. A free diagnostic row does not refund that start event. Discounted platform tiers have their own rates; inspect your applicable tier before estimating a batch. These are listing prices read on 21 October 2026, not a receipt or a claim about customer spending. The contract for this draft was checked against installed build 0.1.3 (K4E1G3AtaSlcyGbCU). The input above is an illustrative schema example, not a newly executed run. Current source responses and useful output still need inspection when you choose to run it. No new cloud build or run was made to prepare this article. Open the Actor and inspect its input and pricing. AI-assisted draft; publication requires independent technical review.

Hacker News official API is a realtime database. Firebase at /v0/. No rate limit stated. Nothing marks it as a public dataset. Enthusiasm for public data and the tools people actually build on top of it. HackerNews/API