Have you heard the good news about the terminal savior asciinema -- https://asciinema.org/
embedding-shape 5 hours ago [-]
It's a cool tool/platform, but very different. Asciinema tries to make the "multimedia" itself better by making it actual text instead of being video/images, while the CLI command above turns actual text into multimedia supported by platforms already. Both are useful, both have their use cases :)
How can you do it? I don't see an SVG output from ascii-gif.
embedding-shape 4 hours ago [-]
Very cool, never thought of that! "way smaller" is almost an understatement, when it's 50kb :P Neat that it loads in GitHub READMEs as well, which is probably a large reason people use .gif today.
tamnd 6 hours ago [-]
I have a bunch of opinionated/personal-use binaries like this in my $HOME/bin/, like delete-all-npm, clean-rust-cache, download-youtube-playlist, and get-markdown <url>. It feels good, and I don't need to remember any commands. Sometimes my coding agent can figure out how to call some of those tools too ;))
stavros 6 hours ago [-]
VHS is fantastic for scripting cli video generation.
alterom 8 hours ago [-]
FYI, on other platforms (Windows/MacOS), LiceCAP is a fantastic tool to record screen into compact GIFs by the author of Winamp and Reaper DAW:
One use I'd have for this is company wikis that you want to give folks easy offline access to (maybe the wiki has documentation that's useful at sites that don't have cellular coverage).
Cool!
It would be especially cool to have a version that didn't require the separate serving process - even though it's nifty you can package up a whole site as a single binary.
Maybe a single HTML entrypoint shim with a bit of javascript that could index into an archive (potentially embedded) of the site's content?
tamnd 10 hours ago [-]
Submitting this to Hacker News is the right place! Thanks for your idea. I will consider implementing that :)
Also, in my mind, I already have a script/program to convert HTML to Markdown, so it could actually store everything on disk as a folder of Markdown files, and then commit them to a Git repo.
mcdonje 2 hours ago [-]
Not to load you up with too many ideas, but a markdown folder sounds a lot like obsidian, which has a plugin system now.
Epub would also be a great target.
mgiampapa 7 hours ago [-]
I think the zim flow was perfect for offline use. I know I will be making use of it as soon as I can figure out how to pass chrome the cookies so I can be signed into the site. Didn't see it in the page, but I didn't look closely yet.
tamnd 5 hours ago [-]
Not yet supporting cookies, since I created this tool for shadowing public websites first. I will add options to pass cookies later. It will pass them to the underlying Chrome/Chromium process, so it should not be hard to do.
smeej 3 hours ago [-]
I would use the shit out of this. I'm a heavy user of Logseq (OG, the md file-based version). Would LOVE to save my favorite web resources this way.
gwern 3 hours ago [-]
> Maybe a single HTML entrypoint shim with a bit of javascript that could index into an archive (potentially embedded) of the site's content?
If the result is static why does it need a server? Isn't it possible to make it so that it can simply be opened by the browser? Like:
$ firefox $HOME/data/kage/paulgraham.com
Then the result would be useable on machines without kage nstalled.
tamnd 6 hours ago [-]
You could use python -m http.server instead. I haven't tried it yet, but it should work.
Actually, Kage has two parts: a crawler that crawls pages and converts them to clean HTML by capturing the DOM after rendering in Chrome/Chromium, and a pack/serve component that packages the result as either a ZIM file for Kiwix or an executable file.
doctoboggan 9 hours ago [-]
Usually JavaScript is blocked when you load pages that way.
dmazzoni 8 hours ago [-]
Not all JavaScript, but a lot of APIs are restricted
embedding-shape 8 hours ago [-]
Since when? You won't be able to make HTTP requests to localhost, as it'd be a different Origin, but I don't think any mainstream browser blocks JS outright when you use file:// to load and view HTML files.
Yeah, but that's fine, the document is .html, and it can load ./app.js or ./style.css just fine even if loaded by file:// (as long as it isn't initiated by JS itself, then Origin starts to matter a lot more), otherwise basically every single local HTML file would suddenly be broken, I don't think anyone would have accepted that even with the origin changes.
pixelatedindex 8 hours ago [-]
I thought all the JS was stripper?
recursive 8 hours ago [-]
I am quite familiar with this and it is factually false
danielheath 7 hours ago [-]
Js modules don’t work on file urls (classic js does).
recursive 2 hours ago [-]
They can be made to work with blob urls. I have done this.
afavour 8 hours ago [-]
You’ll likely run into a ton of CORS issues doing that.
embedding-shape 8 hours ago [-]
I don't think so, there is no HTTP requests being done from JS as it's stripped away, and all the other resources are pulled down (and I'm assume their reference made relative), so really shouldn't be any issues because of CORS at all.
maxloh 11 hours ago [-]
I find SingleFile [0] to be a much more robust version of this.
It strips out all the JavaScript too, but also packs everything into a single HTML file that is easy to transfer. Binary assets (like web fonts and images) are packed as base64 strings.
What I'm implementing here is mirroring a whole website, with all its subpages, so you can browse it all offline. For example, all essays from paulgraham.com.
maxloh 9 hours ago [-]
Oh, I see. In that case, feature-wise, it is actually a modern alternative to HTTrack.
I think the misunderstanding stems from the browser's "Save As" reference in the description. It is misleading. You use "Save As" to save a single page, not an entire website.
Also, the description lacks a clear explanation of the project's purpose. It would be helpful to include a sentence explaining that the program downloads an entire website, not just a single page.
I highly recommend reading the singlefile source or https://archiveweb.page/ to see how they handle closed shadow DOMs, cross-origin iframes, websockets, media urls, deduping large assets, etc.
sillysaurusx 4 hours ago [-]
> For example, all essays from paulgraham.com
Not the same thing, but I made a clone of pg’s website which can be used for exactly that: https://github.com/shawwn/pg
If you want to read all essays, just clone the repo and open any of the .html files. Or any of the .page files which generated them.
sdevonoes 10 hours ago [-]
[flagged]
sermah 10 hours ago [-]
Um. Whose website are you on right now?
ivangelion 10 hours ago [-]
Don't come here to laugh but always great when it happens anyways.
wamatt 8 hours ago [-]
Love love love SingleFile too. The FF extension works pretty well for a clean save.
That said, Kage looks promising if OP can combine SingleFile reproduction quality with the HTTPTrack spidering approach. SPA's are kinda tricky with archiving and do wonder how well Kage would handle that
initramfs 7 hours ago [-]
I've seen the option in IE- .mhtml.
For some reason it displays in IE better but I don't recall seeing this option in chrome of Firefox recently..
tamnd 11 hours ago [-]
And thanks for the link. Let me implement this single HTML feature, it looks nice to have!
maxloh 9 hours ago [-]
Yeah. An idea on top of that is to bundle an entire website into a single HTML page, with vendored JavaScript to enable client-side routing (all of the original pages' JS is still stripped out).
That way, the page is self-contained as it is, but requires no bundled binary code to serve the site. It is actually safer security-wise.
The vendored script can be as simple as this:
const site = {
"path-1": "<!DOCTYPE html><html> ... </html>",
"path-2": "<!DOCTYPE html><html> ... </html>",
// More paths
}
function attachListeners() {
for (const [path, html] of Object.entries(site)) {
document.querySelector(`a[href=${path}]`).onclick = () => {
document.documentElement.outerHTML = html
attachListeners()
}
}
}
document.addEventListeners("DOMContentLoaded", attachListeners)
HelloUsername 10 hours ago [-]
What's the difference with, any webbrowser on a computer, File -> Save as ?
nmstoker 10 hours ago [-]
That's for a single page, this handles the whole site. Also the browser Save As options often work poorly.
dmazzoni 8 hours ago [-]
Save As works fine for simple websites with static content.
Let's say you have a site that fetches content from a database. If you Save As, then at best you'll get a local copy of an HTML page with JS that loads the content from the same remote database. It might not work (since the local copy has a different origin), or if it does, it requires you to be online, which defeats half of the purpose.
What this project, and SingleFile, both do is save a snapshot of what the rendered page actually looks like at that moment in time. The scripts are stripped out so it runs locally and has no external dependencies.
arikrahman 8 hours ago [-]
This is what I first thought and it's a very elegant solution, and not needlessly overcomplicated.
9 hours ago [-]
telesilla 9 hours ago [-]
I've been using httrack (https://www.httrack.com) to download wikis to read on flights, which isn't perfect but better than I'd found previously. I'll try this out, I'd be delighted to have good results. Thanks for the post.
throwaway219450 7 hours ago [-]
Specifically for wikis, is there a reason you wouldn't use Kiwix? For non "official" releases it's more complicated, but there are some services to generate the ZIM files. The desktop reader app is pretty good in my experience.
Kiwix has readers for almost every platform, Android, desktop, iPhone. That's why I made Kage produce ZIM file.
The executable file is mostly for people who don't have Kiwix installed yet, or just want to run the archive directly.
telesilla 6 hours ago [-]
Thanks, never knew about this and great to hear about it.
tamnd 5 hours ago [-]
This brings back memories. Around twenty years ago, internet was still expensive dial-up, so I used to go to an internet cafe, run HTTrack to download websites and manga, copy everything onto my tiny 128MB USB stick (felt very large at that time), then bring it home and read offline ;))
nikisweeting 6 hours ago [-]
https://github.com/archiveteam/grab-site or browsertrix may be easier to use for some, it's what was used to save a lot of the data.gov stuff before it got taken down.
gregwebs 11 hours ago [-]
This seems like it has potential to create a lot of load on a site- are there settings to set how fast it clones or avoid images/videos?
Is there a way to only get a subset of a website?
tamnd 10 hours ago [-]
Could you help create a new issue for that? I will do it later. It is already 1:00 AM my time, but I am happy that anyone is interested in it. : )
But will look into this now, see if we can swap some stuff out. We’ve really liked the idea of an offline mirror, makes a lot of collaboration use cases simpler
Both of these projects have completely different purposes and use cases.
Have you even read the first line of the readme of the project you're commenting on?
coffeecoders 8 hours ago [-]
I've accumulated a bunch of old website archives over the years. The funny thing is the ugly HTML dumps have been more useful than the "perfect" archive.
It's one of the reasons I've become a bigger fan of RSS over time. A feed from 10-ish years ago is often more usable today than a carefully preserved (application) website.
tamnd 5 hours ago [-]
I have a project for creating and archiving RSS feeds, keeping the full history from the time the crawler starts. I need to clean up a bit, then will open source it soon.
dimiprasakis 10 hours ago [-]
Neat project, I like the idea.
One thing from a quick read: you launch Chrome with --no-sandbox. Is there a good reason for that? Security wise it's probably not a good idea. If there is no reason, I'd suggest leaving the sandbox on!
In any case, cool stuff :)
nikisweeting 6 hours ago [-]
--no-sandbox is needed in docker, maybe they assume it will mostly run in docker?
tamnd 5 hours ago [-]
Exactly. For downloading, Kage requires Chrome or Chromium. Running it inside Docker makes setup easier and keeps cleanup simple:
Btw, let me think the way to only enable this when running inside Docker.
nikisweeting 4 hours ago [-]
Docker is designed to be undetectable by default, the best way I have found is to set env IN_DOCKER=True manually in your Dockerfile + check that there is no $DISPLAY configured + that you're on linux. Usually if all/most of those are true you can safely add --no-sandbox --disable-setuid-sandbox --disable-dev-shm-usage etc. all the docker-specific flags. Thats what we do in https://github.com/ArchiveBox/ArchiveBox/blob/dev/Dockerfile...
carsonye 52 minutes ago [-]
This is interesting. Is the intended use case mostly read-only websites like blogs/docs/essays? How well does it handle sites where navigation, search, dropdowns, or other UI interactions depend on JavaScript?
amatecha 4 hours ago [-]
Suddenly remembering the days of dialup and your browser serving a fully-functional cached copy of a webpage when you try to access it and you're not online...
Compared to that is there anything kage does better?
smusamashah 1 hours ago [-]
What if I wanted to download all Confluence docs at work?
sanqui 10 hours ago [-]
Cool concept. I would like to see this combined with mitmproxy for archive grade fidelity. You could be saving exactly the data served and at the same time a representation by a modern (contemporary) browser, with all JS having run. This combination would be my perfect replacement for the WARC format.
tamnd 10 hours ago [-]
I'm working on WARC too, with format from Common Crawl!
By converting it to Markdown, we save a lot of space, but it is for a different purpose and a different project: https://github.com/tamnd/ccrawl-cli
sanqui 10 hours ago [-]
That's neat! In my opinion, the WARC format is quite tricky and underspecified especially since HTTP2 introduced new semantics. It encodes too much in-band and requires rewriting of the server data. A mitmproxy capture is higher fidelity and supports capturing modern features such as WebSockets. I think if we could wrap Kage's crawler interactions by it and store its capture (the intercepted traffic), we could make a potentially nice new archival format.
tamnd 10 hours ago [-]
I tried to follow well-known formats first, such as WARC and ZIM from Kiwix, so we could benefit from existing tooling support.
For my own custom data format, I have a lot of private code that I plan to release soon. It is optimized for compression, fast lookups, and more. I have been working on it for two years.
This is part of a larger, ambitious umbrella project: I am building Google from scratch (all open source), something that anyone can host, including the crawler, indexer, storage, and serving layers. Stay tuned!
sanqui 10 hours ago [-]
I'm a fan of compatibility with established formats!
Sounds awesome. There is a lot of untapped potential with respect to efficiently archiving and indexing websites. I saw the impressive things Marginalia Search is doing in this area (the blog is great when it gets technical). There is also a lot of very complete archives of websites out there which are not being indexed at all, and I would love to make them available for researchers. In any case, I'm interested in your project!
threecheese 6 hours ago [-]
OK, sounds fascinating; following! (your GH)
tamnd 5 hours ago [-]
Thanks ;)
Prime_Axiom 9 hours ago [-]
Looking forward to the next project! I love these kinds of archiving tools.
Dhavidh 10 hours ago [-]
sound interesting
jyscao 4 hours ago [-]
I tried to clone a HTTP (not HTTPS) site, and it's giving me `navigation failed: net::ERR_NAME_NOT_RESOLVED`. Even when I explicitly included the protocol with `http://<FQDN>`.
rahimnathwani 10 hours ago [-]
So this is like using wget --mirror except that it works on pages that require javascript, right?
tamnd 10 hours ago [-]
Yeah, it is. For example, openai.com is rendered with Next.js, so I will try to mirror it tomorrow.
10 hours ago [-]
Igor_Wiwi 10 hours ago [-]
This is quite useful tool, especially for the cases where internet access is limited (the flights for example). I implemented it as a separate feature in mdview.io: for example you can export a document as a html file for offline usage, with all the presentation features like reach tables, mermaid and etc built in. Example https://mdview.io/s/why-markdown-became-default-format-for-a... then try to Export - Export HTML
lolpython 10 hours ago [-]
This is cool. I could see myself downloading the articles behind the first couple pages of hacker news with this, for viewing on a flight or long distance train ride with spotty internet
godot 4 hours ago [-]
the readme uses paulgraham.com as an example (which is text articles mostly) and I never use "Save As" for a web page (for the reasons the author states), I always just print as PDF and save the PDF file.
for an entire website though of many pages I can see this can be useful.
G_o_D 2 hours ago [-]
How its different then MHTML ??
latexr 9 hours ago [-]
For those with an eReader, one thing that works really well is using pandoc to download and convert a webpage to EPUB that you can then load to your reader.
pandoc --from html --to epub --output /PATH/TO/FILE.epub https://example.com
arikrahman 8 hours ago [-]
Thanks, will try this out on the Kobo later.
calrizien 7 hours ago [-]
Does this work for the Apple Docs website? Really tricky to get those offline.
tamnd 5 hours ago [-]
Making docs available offline was one of my main motivations for building this tool. I will try Apple Docs too.
I previously downloaded the Snowflake docs, and it was something like tens or even hundreds of thousands of pages, I do not remember exactly. The output ended up being very large.
By the way, I forgot to add zstd compression support to my ZIM reader/writer. I will implement that in the next version.
nitotm 7 hours ago [-]
I was looking for something like this the other day, it can be very helpful.
10 hours ago [-]
chfritz 6 hours ago [-]
how is this different from using puppeteer to load the page and save the DOM as HTML?
daviding 10 hours ago [-]
Nice idea!
fwiw, false positives and all, but the Windows 11 default Windows Security doesn't like it:
`leakless.exe: Operation did not complete successfully because the file contains a virus or potentially unwanted software.`
KellyCriterion 8 hours ago [-]
Sounds like .MCH-files re-invented? (-:
cynicalsecurity 7 hours ago [-]
Binary app is a really bad way of storing data. No one would ever want to run a binary shared with them or found online.
tamnd 5 hours ago [-]
For sharing, better use the html folder or zim format, Kage supports both of them.
soulofmischief 8 hours ago [-]
Cool project! I know it's written in go, but it would be cool to see something like this which uses Cosmopolitan Libc + redbean or something similar to create a binary which runs anywhere. Would be fun to be able to pass around self-executable website archives.
Turns out it's using another project by the same author: https://github.com/tamnd/ascii-gif
The script used for the demo is at https://github.com/tamnd/kage/blob/01e75b87ecc893bbba7943c63... and has a comment showing how to run it:
Looks like it's an opinionated wrapper around https://github.com/charmbracelet/vhshttps://www.cockos.com/licecap/
Cool!
It would be especially cool to have a version that didn't require the separate serving process - even though it's nifty you can package up a whole site as a single binary.
Maybe a single HTML entrypoint shim with a bit of javascript that could index into an archive (potentially embedded) of the site's content?
Also, in my mind, I already have a script/program to convert HTML to Markdown, so it could actually store everything on disk as a folder of Markdown files, and then commit them to a Git repo.
Epub would also be a great target.
So something like SingleFileZ https://github.com/gildas-lormeau/SingleFileZ or Gwtar https://gwern.net/gwtar ?
If the result is static why does it need a server? Isn't it possible to make it so that it can simply be opened by the browser? Like:
$ firefox $HOME/data/kage/paulgraham.com
Then the result would be useable on machines without kage nstalled.
Actually, Kage has two parts: a crawler that crawls pages and converts them to clean HTML by capturing the DOM after rendering in Chrome/Chromium, and a pack/serve component that packages the result as either a ZIM file for Kiwix or an executable file.
Related WHATWG discussion: https://github.com/whatwg/html/issues/3099
It strips out all the JavaScript too, but also packs everything into a single HTML file that is easy to transfer. Binary assets (like web fonts and images) are packed as base64 strings.
They also offer a CLI powered by Puppeteer. [1]
[0]: https://github.com/gildas-lormeau/singlefile
[1]: https://github.com/gildas-lormeau/single-file-cli
What I'm implementing here is mirroring a whole website, with all its subpages, so you can browse it all offline. For example, all essays from paulgraham.com.
I think the misunderstanding stems from the browser's "Save As" reference in the description. It is misleading. You use "Save As" to save a single page, not an entire website.
Also, the description lacks a clear explanation of the project's purpose. It would be helpful to include a sentence explaining that the program downloads an entire website, not just a single page.
I highly recommend reading the singlefile source or https://archiveweb.page/ to see how they handle closed shadow DOMs, cross-origin iframes, websockets, media urls, deduping large assets, etc.
Not the same thing, but I made a clone of pg’s website which can be used for exactly that: https://github.com/shawwn/pg
https://shawwn.github.io/pg/
If you want to read all essays, just clone the repo and open any of the .html files. Or any of the .page files which generated them.
That said, Kage looks promising if OP can combine SingleFile reproduction quality with the HTTPTrack spidering approach. SPA's are kinda tricky with archiving and do wonder how well Kage would handle that
For some reason it displays in IE better but I don't recall seeing this option in chrome of Firefox recently..
That way, the page is self-contained as it is, but requires no bundled binary code to serve the site. It is actually safer security-wise.
The vendored script can be as simple as this:
Let's say you have a site that fetches content from a database. If you Save As, then at best you'll get a local copy of an HTML page with JS that loads the content from the same remote database. It might not work (since the local copy has a different origin), or if it does, it requires you to be online, which defeats half of the purpose.
What this project, and SingleFile, both do is save a snapshot of what the rendered page actually looks like at that moment in time. The scripts are stripped out so it runs locally and has no external dependencies.
https://wiki.openzim.org/wiki/Build_your_ZIM_file
EDIT: https://get.kiwix.org/en/solutions/applications/kiwix-reader...
The executable file is mostly for people who don't have Kiwix installed yet, or just want to run the archive directly.
But will look into this now, see if we can swap some stuff out. We’ve really liked the idea of an offline mirror, makes a lot of collaboration use cases simpler
Have you even read the first line of the readme of the project you're commenting on?
It's one of the reasons I've become a bigger fan of RSS over time. A feed from 10-ish years ago is often more usable today than a carefully preserved (application) website.
In any case, cool stuff :)
https://github.com/tamnd/kage/blob/main/Dockerfile
Btw, let me think the way to only enable this when running inside Docker.
Compared to that is there anything kage does better?
By converting it to Markdown, we save a lot of space, but it is for a different purpose and a different project: https://github.com/tamnd/ccrawl-cli
For my own custom data format, I have a lot of private code that I plan to release soon. It is optimized for compression, fast lookups, and more. I have been working on it for two years. This is part of a larger, ambitious umbrella project: I am building Google from scratch (all open source), something that anyone can host, including the crawler, indexer, storage, and serving layers. Stay tuned!
Sounds awesome. There is a lot of untapped potential with respect to efficiently archiving and indexing websites. I saw the impressive things Marginalia Search is doing in this area (the blog is great when it gets technical). There is also a lot of very complete archives of websites out there which are not being indexed at all, and I would love to make them available for researchers. In any case, I'm interested in your project!
for an entire website though of many pages I can see this can be useful.
I previously downloaded the Snowflake docs, and it was something like tens or even hundreds of thousands of pages, I do not remember exactly. The output ended up being very large.
By the way, I forgot to add zstd compression support to my ZIM reader/writer. I will implement that in the next version.
https://github.com/jart/cosmopolitan
https://justine.lol/cosmopolitan/index.html
https://redbean.dev
(Certificates just expired for justine's website, just ignore the warning.)
I did something like that a very long time ago (Of course, I have forgotten)
Is the code also AI slop?