Servo integration?

[mre]

Recently, servo got released to crates.io. 🎉

That is a pretty big milestone for the project and means that Servo can be used as a headless browser in other Rust projects without any external dependencies or DLL bindings.

For lychee, the consequence is that we could add support for executing JavaScript. This way, websites that heavily depend on JS could be rendered correctly for the first time and we could use the output for link detection. I tried it in a prototype and it worked pretty nicely.

Note that it doesn't necessarily mean that we'll detect more links, just different links. The mental model is that JS execution will take a part of the HTML and replace it with different HTML. That could also mean that it replaces a bunch of HTML which contain links with a "rendered" version that no longer contains links.

I ran a quick spike against 13 sites to get a clearer picture, fetching each one both statically and via headless Servo, then comparing the extracted links. This is purely for research/discussion purposes.

The results fell into three buckets:

1. SSR sites
   example.com, remix.run, wikipedia.org, figma.com, vercel.com showed no diff at all. These ship fully-rendered HTML, so JS execution adds nothing.

2. SSR + hydration (Next.js etc.): static finds more than Servo
   This was the surprising one. react.dev, nextjs.org, and supabase.com are all Next.js apps whose raw HTML already contains all pre-rendered links. But Servo extracted significantly fewer, because during hydration, React rewrites the DOM and in doing so drops links that existed in the original server-rendered HTML. This is because of image URL replacement for example, where it selects the best image based on the client.

3. Ad-heavy media/commerce sites: Servo finds more than static
   techcrunch.com, theverge.com, and nike.com showed 20–30 JS-only links. They're almost exclusively ad infrastructure: Google Tag Manager, DoubleClick, HubSpot, OneTrust cookie banners, third-party chat widgets. Still might be useful for link checking.

Bottom line: static-first should remain the default I think. It's not just faster, it's actually more complete for most pages I tested. (Might just be a bad dataset, though! Feedback wanted.) Servo would add the most value for true SPAs with no server-rendering at all, which are a bit rare.

In my opinion, a reasonable approach would be to make headless rendering opt-in, or to fall back to it only when the static fetch returns suspiciously few links. I think we could add a servo feature and enable it conditionally. For now, people, could pass it during compilation and build a custom version of lychee. Of course, there's a ton of work that needs to be done to push it over the finishing line. We'd basically have to add a full prerender step, so that's a lot of work.

The goal is to start the conversation and get some feedback on what to do next. I'd love to hear people's thoughts on this and whether there's a use-case for browser-based rendering.