How to Download an Entire Website With wget (--mirror Alone Breaks Links)
A fansite about cartoon dogs with tea-leaf ears went online in 2004 and barely changed after 2005. A few weeks ago one of its old visitors went back and found the domain serving online gambling ads. They work in web archiving, and they still never saved a copy. That's the story behind the post If you love a website, back it up, and the HN thread under it is mostly people listing their own dead bookmarks.
This happens constantly. Pew Research found that 38% of webpages that existed in 2013 were no longer accessible a decade later, and 54% of Wikipedia pages had at least one reference link pointing at a page that no longer exists. "It's on the Wayback Machine" doesn't help much either. One HN commenter found most of a forum had never been crawled. Another hit "This URL has been excluded from the Wayback Machine" because whoever squats the domain now put up a robots.txt, and it hid the old snapshots too.
So if you want to download an entire website, do it yourself, and do it while it still loads. Below is the wget command I use, plus the tools I reach for when wget can't handle a site (HTTrack, monolith, SingleFile, ArchiveBox, Browsertrix) and where each of them breaks.
The short version
| Tool | Best for | Runs JavaScript? | Output |
|---|---|---|---|
| wget | Static sites, old docs, university pages | No | Folder of HTML + assets, optional WARC |
| HTTrack | Same as wget, with a GUI | No | Browsable folder |
| monolith | One page, one file | No (unless you pre-render) | Single HTML |
| SingleFile CLI | One page or a shallow crawl, JS included | Yes (real Chrome) | Single HTML / ZIP |
| ArchiveBox | Bookmark-style archiving, many formats per URL | Yes (headless Chrome) | HTML, PDF, PNG, WARC, media |
| Browsertrix Crawler | Full crawls of JS-heavy sites, logins | Yes | WACZ |
My opinion: wget is still the right first move for anything that renders without JavaScript. If the content only shows up after JS runs, skip straight to Browsertrix and stop fighting wget flags.
wget: the mirror command that actually rewrites links
Plain wget --mirror is a trap. Per the GNU Wget 1.25.0 manual, --mirror is just -r -N -l inf --no-remove-listing: recursion and timestamps. It won't fix links, so your local copy keeps pointing at the live site, and once that site dies your copy is broken too.
This is the line you want:
wget --mirror --convert-links --adjust-extension --page-requisites --no-parent \
--wait=2 --random-wait --limit-rate=200k \
https://example.com/
- --convert-links rewrites links after the download so the copy works offline.
- --adjust-extension appends .html to pages served from URLs like /about or /post?id=3, so your browser opens them properly.
- --page-requisites pulls the CSS, images and scripts each page needs.
- --no-parent keeps wget from climbing out of the directory you pointed it at.
- --wait=2 --random-wait spaces requests between 1 and 3 seconds (the manual says 0.5 to 1.5 times the wait), and --limit-rate caps bandwidth.
The HN shorthand for the core of this is wget -mkpnp. Add -E for adjust-extension.
If you want a preservation-grade copy and not just something you can browse, add --warc-file=example. You get example.warc.gz next to the mirror, which replay tools and the Internet Archive both understand. One catch on 1.25.0: wget prints "WARC output does not work with timestamping, timestamping will be disabled," so the -N half of --mirror is gone and re-running the command re-downloads everything.
Where wget fails:
- SPAs: wget follows links in HTML and CSS. It doesn't execute JavaScript, so a React app gets saved as an empty <div id="root">.
- Logins: you can pass browser cookies with --load-cookies cookies.txt. It works until the session expires halfway through the crawl.
- Bot walls: one commenter put it bluntly, recursive wget "isn't going to pass Anubis." Cloudflare challenges stop it as well.
- robots.txt: wget respects it by default during recursion. You can turn that off with -e robots=off. Don't do it on a site run by a single person on a $5 VPS.
HTTrack: wget with a window
If you don't want to live in a terminal, HTTrack does the same job with a GUI on Windows (WinHTTrack) and a CLI everywhere else:
httrack "https://example.com/" -O ~/mirrors/example "+*.example.com/*" -c2 -%c2 -A100000 -v
The options guide says the default is 8 simultaneous connections (-c8) and 10 connections per second (-%c10). That's rude to a small site, so I drop both to 2. -A100000 caps transfers at about 100 KB/s. -s2, the default, means it always obeys robots.txt. Leave it there.
HTTrack has the same blind spot as wget: no JavaScript. But it's old and stable, and it handles the sites most likely to vanish just fine: 2005-era fansites and abandoned project pages.
One page, one file: monolith and SingleFile
Sometimes you only need the article and not the whole site.
monolith is a single Rust binary that inlines every asset as a data URL:
monolith https://example.com/post -o post.html
The README admits it "doesn't feature a JavaScript engine." The documented workaround is to let headless Chromium render the page and pipe the DOM in:
chromium --headless --window-size=1920,1080 --run-all-compositor-stages-before-draw \
--virtual-time-budget=9000 --incognito --dump-dom https://github.com \
| monolith - -I -b https://github.com -o github.html
SingleFile came up repeatedly in the HN thread for quick one-offs, and one commenter pointed out that Zotero uses it under the hood to save web pages. The CLI drives Chrome (or another Chromium browser) over the DevTools protocol, so JS-rendered content shows up:
npx single-file https://www.wikipedia.org wikipedia.html
It can also do shallow crawls, which a lot of people don't realise:
single-file https://example.com --crawl-links=true --crawl-inner-links-only=true \
--crawl-max-depth=1 --crawl-save-archive=true
ArchiveBox: every format, self-hosted
ArchiveBox takes a URL and runs everything at it: SingleFile, a wget clone with WARC, Chrome PDF, screenshot, DOM dump, readability text, yt-dlp for media, and git for repos. Getting it running:
mkdir -p ~/archivebox/data && cd ~/archivebox/data
docker run -d --name archivebox -v $PWD:/data -p 5797:5797 archivebox/archivebox:dev
Or install it with uv and use the CLI:
uv tool install --python 3.13 --prerelease explicit --upgrade 'archivebox>=0.9.0rc0,<0.10'
archivebox init
archivebox add 'https://example.com'
archivebox server 0.0.0.0:5797
For recurring snapshots it has archivebox schedule, which runs through the same orchestrator as the server. The syntax changed in the 0.9 rewrite, so check archivebox schedule --help rather than copying an old blog post.
ArchiveBox is great at saving the 400 links you bookmarked and terrible at "mirror this 20,000-page forum." In the thread people complained that self-hosting it gets annoying, that their queues fill up with pages that failed to capture, and that server-side capture runs into Cloudflare. The maintainer replied that 0.9 is basically a complete rewrite, bundles ArchiveWeb.page as a plugin, and has a browser extension that captures from your own browser and uploads the result to your server. It's still on release candidates (>=0.9.0rc0), so expect some rough edges.
Browsertrix: for JavaScript-heavy sites and logins
This is the one I'd use for a modern site. Browsertrix Crawler runs real browsers in Docker and writes a WACZ, one file you can open in replayweb.page with no server:
docker run -v $PWD/crawls:/crawls/ -it webrecorder/browsertrix-crawler crawl \
--url https://example.com/ --scopeType host --generateWACZ --text \
--collection example --workers 2
The output ends up in crawls/collections/example/example.wacz. --scopeType defaults to prefix (same directory). Use host for the whole site and page-spa for single-page apps that route through hashes. --limit, --sizeLimit and --timeLimit keep a crawl from running away.
For sites behind a login, you run create-login-profile, sign in once through the embedded browser on port 9223, and pass the result with --profile /crawls/profiles/profile.tar.gz. That works far better than exporting cookies for wget.
If Docker is too much, the same team makes the archiveweb.page extension: click Start from the extension icon, browse the site, click Stop. Because it captures from your own logged-in browser, bot walls mostly stop mattering, which was the point several HN commenters kept coming back to.
How to stay polite
Small sites die when someone hammers them, too. Before you crawl:
1. Read /robots.txt and respect it unless you have a real reason not to (the site is already dead and you're working from a cache, for example).
2. Use one or two connections, not eight.
3. Add a wait. Two seconds with random jitter costs you nothing on a one-off mirror.
4. Scope tightly. --no-parent, --scopeType prefix and --limit are there for exactly this.
5. If it's a community site, ask the admin. They often have a database dump they'd be happy to hand over.
And then keep the copy in more than one place. "Lots of copies keeps stuff safe," as the original post puts it. Upload a WARC to the Internet Archive, give a WACZ to a friend, anything other than one folder on one laptop.
Which website would you be gutted to lose tomorrow, and have you actually backed it up? If your answer is "wget is all you need" or "just print to PDF," tell me in the comments what you do about the JavaScript.
Sources
- If you love a website, back it up and the Hacker News discussion
- Pew Research: When Online Content Disappears
- GNU Wget manual
- HTTrack options guide
- monolith, single-file-cli, ArchiveBox
- Browsertrix Crawler docs
Mentions
How to Download an Entire Website With wget (--mirror Alone Breaks Links) A fansite about cartoon dogs with tea-leaf ears went online in 2004 and barely changed after 2005. A few weeks ago one of its old visitors went back and found the domain serving online gambling ads. They work in web archiving, and they still never saved a copy. That's the story behind the post If you love a website, back it up, and the HN thread under it is mostly people listing their own dead bookmarks. This happens constantly. Pew Research found that 38% of webpages that existed in 2013 were no longer accessible a decade later, and 54% of Wikipedia pages had at least one reference link pointing at a page that no longer exists. "It's on the Wayback Machine" doesn't help much either. One HN commenter found most of a forum had never been crawled. Another hit "This URL has been excluded from the Wayback Machine" because whoever squats the domain now put up a robots.txt, and it hid the old snapshots too. So if you want to download an entire website, do it yourself, and do it while it still loads. Below is the wget command I use, plus the tools I reach for when wget can't handle a site (HTTrack, monolith, SingleFile, ArchiveBox, Browsertrix) and where each of them breaks. The short version | Tool | Best for | Runs JavaScript? | Output | |---|---|---|---| | wget | Static sites, old docs, university pages | No | Folder of HTML + assets, optional WARC | | HTTrack | Same as wget, with a GUI | No | Browsable folder | | monolith | One page, one file | No (unless you pre-render) | Single HTML | | SingleFile CLI | One page or a shallow crawl, JS included | Yes (real Chrome) | Single HTML / ZIP | | ArchiveBox | Bookmark-style archiving, many formats per URL | Yes (headless Chrome) | HTML, PDF, PNG, WARC, media | | Browsertrix Crawler | Full crawls of JS-heavy sites, logins | Yes | WACZ | My opinion: wget is still the right first move for anything that renders without JavaScript. If the content only shows up after JS runs, skip straight to Browsertrix and stop fighting wget flags. wget: the mirror command that actually rewrites links Plain wget --mirror is a trap. Per the GNU Wget 1.25.0 manual, --mirror is just -r -N -l inf --no-remove-listing: recursion and timestamps. It won't fix links, so your local copy keeps pointing at the live site, and once that site dies your copy is broken too. This is the line you want: wget --mirror --convert-links --adjust-extension --page-requisites --no-parent \ --wait=2 --random-wait --limit-rate=200k \ https://example.com/ - --convert-links rewrites links after the download so the copy works offline. - --adjust-extension appends .html to pages served from URLs like /about or /post?id=3, so your browser opens them properly. - --page-requisites pulls the CSS, images and scripts each page needs. - --no-parent keeps wget from climbing out of the directory you pointed it at. - --wait=2 --random-wait spaces requests between 1 and 3 seconds (the manual says 0.5 to 1.5 times the wait), and --limit-rate caps bandwidth. The HN shorthand for the core of this is wget -mkpnp. Add -E for adjust-extension. If you want a preservation-grade copy and not just something you can browse, add --warc-file=example. You get example.warc.gz next to the mirror, which replay tools and the Internet Archive both understand. One catch on 1.25.0: wget prints "WARC output does not work with timestamping, timestamping will be disabled," so the -N half of --mirror is gone and re-running the command re-downloads everything. Where wget fails: - SPAs: wget follows links in HTML and CSS. It doesn't execute JavaScript, so a React app gets saved as an empty <div id="root">. - Logins: you can pass browser cookies with --load-cookies cookies.txt. It works until the session expires halfway through the crawl. - Bot walls: one commenter put it bluntly, recursive wget "isn't going to pass Anubis." Cloudflare challenges stop it as well. - robots.txt: wget respects it by default during recursion. You can turn that off with -e robots=off. Don't do it on a site run by a single person on a $5 VPS. HTTrack: wget with a window If you don't want to live in a terminal, HTTrack does the same job with a GUI on Windows (WinHTTrack) and a CLI everywhere else: httrack "https://example.com/" -O ~/mirrors/example "+*.example.com/*" -c2 -%c2 -A100000 -v The options guide says the default is 8 simultaneous connections (-c8) and 10 connections per second (-%c10). That's rude to a small site, so I drop both to 2. -A100000 caps transfers at about 100 KB/s. -s2, the default, means it always obeys robots.txt. Leave it there. HTTrack has the same blind spot as wget: no JavaScript. But it's old and stable, and it handles the sites most likely to vanish just fine: 2005-era fansites and abandoned project pages. One page, one file: monolith and SingleFile Sometimes you only need the article and not the whole site. monolith is a single Rust binary that inlines every asset as a data URL: monolith https://example.com/post -o post.html The README admits it "doesn't feature a JavaScript engine." The documented workaround is to let headless Chromium render the page and pipe the DOM in: chromium --headless --window-size=1920,1080 --run-all-compositor-stages-before-draw \ --virtual-time-budget=9000 --incognito --dump-dom https://github.com \ | monolith - -I -b https://github.com -o github.html SingleFile came up repeatedly in the HN thread for quick one-offs, and one commenter pointed out that Zotero uses it under the hood to save web pages. The CLI drives Chrome (or another Chromium browser) over the DevTools protocol, so JS-rendered content shows up: npx single-file https://www.wikipedia.org wikipedia.html It can also do shallow crawls, which a lot of people don't realise: single-file https://example.com --crawl-links=true --crawl-inner-links-only=true \ --crawl-max-depth=1 --crawl-save-archive=true ArchiveBox: every format, self-hosted ArchiveBox takes a URL and runs everything at it: SingleFile, a wget clone with WARC, Chrome PDF, screenshot, DOM dump, readability text, yt-dlp for media, and git for repos. Getting it running: mkdir -p ~/archivebox/data && cd ~/archivebox/data docker run -d --name archivebox -v $PWD:/data -p 5797:5797 archivebox/archivebox:dev Or install it with uv and use the CLI: uv tool install --python 3.13 --prerelease explicit --upgrade 'archivebox>=0.9.0rc0,<0.10' archivebox init archivebox add 'https://example.com' archivebox server 0.0.0.0:5797 For recurring snapshots it has archivebox schedule, which runs through the same orchestrator as the server. The syntax changed in the 0.9 rewrite, so check archivebox schedule --help rather than copying an old blog post. ArchiveBox is great at saving the 400 links you bookmarked and terrible at "mirror this 20,000-page forum." In the thread people complained that self-hosting it gets annoying, that their queues fill up with pages that failed to capture, and that server-side capture runs into Cloudflare. The maintainer replied that 0.9 is basically a complete rewrite, bundles ArchiveWeb.page as a plugin, and has a browser extension that captures from your own browser and uploads the result to your server. It's still on release candidates (>=0.9.0rc0), so expect some rough edges. Browsertrix: for JavaScript-heavy sites and logins This is the one I'd use for a modern site. Browsertrix Crawler runs real browsers in Docker and writes a WACZ, one file you can open in replayweb.page with no server: docker run -v $PWD/crawls:/crawls/ -it webrecorder/browsertrix-crawler crawl \ --url https://example.com/ --scopeType host --generateWACZ --text \ --collection example --workers 2 The output ends up in crawls/collections/example/example.wacz. --scopeType defaults to prefix (same directory). Use host for the whole site and page-spa for single-page apps that route through hashes. --limit, --sizeLimit and --timeLimit keep a crawl from running away. For sites behind a login, you run create-login-profile, sign in once through the embedded browser on port 9223, and pass the result with --profile /crawls/profiles/profile.tar.gz. That works far better than exporting cookies for wget. If Docker is too much, the same team makes the archiveweb.page extension: click Start from the extension icon, browse the site, click Stop. Because it captures from your own logged-in browser, bot walls mostly stop mattering, which was the point several HN commenters kept coming back to. How to stay polite Small sites die when someone hammers them, too. Before you crawl: 1. Read /robots.txt and respect it unless you have a real reason not to (the site is already dead and you're working from a cache, for example). 2. Use one or two connections, not eight. 3. Add a wait. Two seconds with random jitter costs you nothing on a one-off mirror. 4. Scope tightly. --no-parent, --scopeType prefix and --limit are there for exactly this. 5. If it's a community site, ask the admin. They often have a database dump they'd be happy to hand over. And then keep the copy in more than one place. "Lots of copies keeps stuff safe," as the original post puts it. Upload a WARC to the Internet Archive, give a WACZ to a friend, anything other than one folder on one laptop. Which website would you be gutted to lose tomorrow, and have you actually backed it up? If your answer is "wget is all you need" or "just print to PDF," tell me in the comments what you do about the JavaScript. Sources - If you love a website, back it up and the Hacker News discussion - Pew Research: When Online Content Disappears - GNU Wget manual - HTTrack options guide - monolith, single-file-cli, ArchiveBox - Browsertrix Crawler docs