The Regrettable State of Cross-Browser Testing

My last post about the perceived “slow” adoption of new CSS sparked some great discussion. But I also got a few bug reports about my site design. I am not enormously concerned about supporting older or unusual browsers, but friends, the site looked bad.

“Best practice” says I should test the site in many browsers and many different versions. But I don’t. Not this site, anyway, for two reasons:

1. I have limited time and no one is paying me for any of this work.
2. It is difficult and/or expensive to test multiple versions of each browser on each OS.

In my defense, though, the only way I was able to reproduce and diagnose the problem, then verify a fix, was to install an old version of Firefox through a Docker container. It’s not something a casual cruise through typical browsers would have revealed.

By the way, this is an excellent way to report that someone’s website is borked (thank you!):

Heads up: On Firefox 115.12.0esr (Debian 12 system version) the page layout is broken (at all viewport widths)?

This will get you blocked:

Why do I get the impression you are either a chrome fan, into the newest, hottest shit, or one of those mobile-first people? :-P

(If you are an indie web person and/or just getting started in web development and someone speaks to you this way, absolutely block, block, block. It’s not you, it’s them.)

This specific problem occurred because I was (and am still) using native CSS nesting instead of running my CSS through a pre-processor. However, I am using the old syntax, which has been generally well-supported across all modern browsers for just shy of a year. Most people don’t have any problem seeing this because browsers all push updates by default, the status quo for many, many years now. I am loathe to add PostCSS back into my tool chain but also really like working with nested CSS, which has been my practice for ten years at least.

The folks having problems with this design were all running the Firefox Extended Support Release, locked at the time to version 115 — two releases behind the native CSS nesting support. Of course, Mozilla doesn’t make it easy to determine what version ESR is running. A Mastodonian helpfully linked to a third-party site explaining the update schedule for the ESR releases.

Third party!

There is no incentive for browser makers to allow you to install old versions side-by-side, and every incentive for them to force updates.

Options for browser testing

If you develop on a Mac, you can easily test using:

• Firefox
• Safari
• Chrome
• Chromium-based Microsoft Edge (Yes, really!)
• Android with the Android SDK
• iOS Safari with Xcode

If you develop on Windows you can test using:

• Firefox
• Chrome
• Chromium-based Microsoft Edge
• Android with the Android SDK

Cross-browser testing is not easy. Or cheap.

Once upon a time you could run multiple versions of a web browser on your own machine, with no need for Docker, but I don’t think it’s worked that way for any of them for many years now.

I have been told that you can do live browser testing using Playwright. I haven’t tried that out yet, but neither had the person who suggested this to me. My experience with Playwright tells me the setup is not beginner-friendly.

I was able to lean on my experience with Docker to pull and run a virtual machine locally with Firefox 115 on it to verify my “fix” worked. That is also not a trivial task for someone unfamiliar with Docker, and it was (at least) frustrating for me.

So in order to do a handful of minutes of testing I now have an Android developer toolkit, the full Xcode application, and Docker Firefox image all installed on my machine. The only reason I have to open any of these tools is to test web sites, which is nuts.

Or there’s BrowserStack, an expensive solution at US$39.00/month. Out of reach for many independent hobby-level developers. Certainly more than I want to pay for the tiny amount of time I need it. I tried a competitor, but once I’d signed up for the free tier I discovered that live testing was limited to three-minutes. On the plus side, I was subjected to the full-court press of commercial sales efforts and got to exercise my unsubscribing finger.

The Indie Web is going to have rough edges

I suppose I spent most of a Saturday morning isolating the cause of this issue, implementing a rough “fix” (not really a fix, just a very simple fallback), and then verifying that the change worked. In my professional environment, I have the support of a team and access to paid, professional tools. This task goes much quicker. But on my personal work, it’s just me responsible for the design, the implementation, the testing, the dev ops, the writing, and most of the photography. I like doing this stuff, but I feel pretty stretched and the things I know I should do is larger than the things I have time to do by several orders of magnitude.

That has to be even more difficult for beginners or folks dabbling in making websites. The surface level of web development is easy to get hold of, but the attack surface for critique is massive. Is the HTML semantic? Is the site accessible? Did you choose tooling (Wordpress, NextJS, Tailwind, Substack) that people have a technical or political beef with? Did you use vanilla CSS and JavaScript? Did you use any JavaScript? Did you use versions of it that were old enough? Is your site contributing too much to greenhouse emissions because your images aren’t properly compressed? Are you stealing everyone’s art by using an AI image somewhere?

It’s real easy to see how people might decide that making their own web site is not worth the scrutiny. We need to be sensitive to that if we want independent creators writing stuff, especially without the expectation of it being monetized or a side hustle.

As far as browser testing goes, the typical user experience is having one of the Big Three browsers installed and on something relatively close to the most recent version. This works for most web site visitors, except those who have not upgraded — either because they can’t or because they won’t.

The solution is not to limit ourselves to markup, layouts, and CSS features available circa 2007 (a suggestion I actually got). For all the device compatibility the old stuff has, it was also much more difficult to make accessible. Oh, and required much more extensive browser testing. (Anyone who thinks float-based layouts are “bulletproof” has never had to support IE 6, I assume.)

I don’t even know what the solution is for website backwards compatibility, at least for the indie web producer. If it’s live cross-browser, cross-version testing, it will need to be something at least as easy to use as BrowserStack but much less expensive.

In the meantime, some patience and tolerance will go a long way. On the indie web, people do what they can with the time and resources they have, and what they give us is usually a gift.

This by no means is to say you shouldn’t raise an issue if you can provide enough context. But please don’t respond to that gift with a sneer or a scowl because of technical decisions you have made. Remember to keep your bug reports concise, include the browser type and version, and the OS. And don’t be surprised to find out there’s very little they are equipped to do.