Checking where website requests are coming from

Exciting news this week. I’ll be continuing my work with The Green Web Foundation into 2023 as a part-time member of the team! This will allow me to keep building out developer documentation, tooling (like CO2.js), and get working on prototypes, ideas, and concepts for making the web more sustainable. I’m really excited for the work we’ll be doing. And, on a personal note, the peace of mind this opportunity provides will make for a nice change.

Talking about building things, this week I’m sharing a beta preview of a tool I’ve been working on. It’s called ReqCheck, and is built to help folks find out where all the different requests made by a web page are served from. I’m hoping it can be a tool for both web sustainability and web performance folks to use when auditing websites.

A few weeks ago, I wrote up a website sustainability audit of the COP27 homepage. It was a really fun exercise, going beyond website carbon calculations and really looking under the hood at how the web page comes together.

One of the things I got into as part of that audit was finding out where the COP27 homepage was being hosted. Using a combination of WebPageTest, and The Green Web Foundation’s IP to Carbon Intensity API, I was able to determine that the site was served from a host based in Egypt. This allowed me to starting build a sustainability profile for the page.

After publishing the article, I got to thinking a bit about some of the things I’d learnt in the process. One thought that stuck with me was “sure, finding where the site is hosted is a good start, but knowing where all requests on a page are hosted would give a more complete picture”.

After a bit of digging around, I discovered that I could use public WebPageTest results and the IP to Carbon Intensity API to uncover this information. But who wants to do that manually for every request on a page? Not me. So, nerd sniped, I started building a way to automate the process.

Introducing ReqCheck (Beta)

ReqCheck (currently in beta) is a tool that takes the results of a WebPageTest run, and surfaces information about where each request is served from. The results show the countries from which resources are served, as well as information about the carbon intensity of that location. Requests made from green web hosts are also highlighted.

Why use WebPageTest results?

WebPageTest allows for tests to be run from a number of different locations around the world. Using WebPageTest results as the data source for ReqCheck opens up the possibility for folks to check how requests for a page are served depending on where a user might be visiting from. By doing this, the impact of different hosting providers and CDNs should become clearer. It should also help to make visible the carbon intensity differences in global energy grids, and how this can effect a page’s sustainability profile.

Using ReqCheck

To use ReqCheck, you’ll first need to run a test on WebPageTest. You will also need to ensure that the test results are public (this is on by default, but can be turned off if you’re logged in with a WebPageTest account).

Running a test

With your result URL in hand, you can then:

  1. Head over to https://reqcheck.fershad.com.
  2. Enter a public WebPageTest result into the form, and click Submit.
  3. The results are then scanned, and a set of unique IP addresses used for requests is compiled.
  4. This is then passed into The Green Web Foundation's IP to Carbon Intensity API to check the location and grid intensity of each IP address.
  5. The tool also uses The Green Web Foundation's Greencheck API to test if a request is served from a green host.

It should be noted that ReqCheck uses the Median Run, First View results from a WebPageTest run. A future version might include Repeat View results to show the impact of caching, but don’t hold me to it!

Reading the results

Once ReqCheck has finished analysing the site, you’ll be taken to the results page (here’s a demo page that you can refer to). Here, you’ll be presented with:

  • Some brief details of the WebPageTest run that was analysed
  • A summary of the ReqCheck test analysis
  • A breakdown of IP addresses by country

For the rest of the time, we’re going to focus on the IP address breakdown, because that’s where the detail lies.

IP addresses by country

The countries listed on the results page are sorted from the one with the most unique IP addresses to the least. Sometimes, the IP to Carbon Intensity API isn’t able to work out where an IP address is located. If this happens, those results are shown at the bottom of the page.

Under each country’s name and flag, you’ll get a short summary of:

  • How many requests came from there
  • How many host IP addresses served those requests
  • The carbon intensity of that country’s electricity grid
  • The percentage of annual electricity generation from fossil fuels

Additionally, if a country has a grid intensity of less than 100 grams of carbon per kilowatt-hour, and fossil-fuel generation of less than 10% then it will be flagged as a "green region". This can start to give a sense of how carbon intensive resources hosted in that country might be.

Below this summary are details of each individual IP address from the country in question. Each block represents a unique IP address, and presents information about:

  • The host that is served from that IP address
  • Whether it uses a CDN (if so, which one)
  • How many requests are served from that address
  • And whether or not the host domain is a known green web host

The green hosting information comes from Greencheck, another API from The Green Web Foundation. If a green host is found, the IP address block is turned green and a leaf icon is shown in the top right.

This additional information about each IP address allows developers to be more informed about how a web page’s resources are hosted. Perhaps requests are being served from a region that has a grid with high carbon intensity, but are hosted on a green web host. In this case, changing how these requests are served could be given a lower priority than ones which are not served from a green host.

Let me know what you think

ReqCheck is still in beta while I work through a few last design and technical things. In the meantime, you’re welcome to give it a try and let me know what you think! DM me on Twitter, Slack, or just drop me an email if you’ve got any feedback.

The next issue of Optimised will be in your inbox on December 23rd. The website has an archive of all previous emails. It's a good place to recap on anything we've covered, and also handy to share with friends or colleagues. As always if you've got any feedback or specific topics you want to be covered then just reply to this email.

Keep safe, stay well.
Fershad.