Introduction
The Hyper Text Transfer Protocol (HTTP) is one of the most common communication protocol on the internet.
There are three major versions of HTTP that you should be aware of: 1.1, 2.0, 3.0.
Versions 1.1 and 2.0 are widely used today, so this tutorial mainly focuses on these two well-known standards.
Version 3.0 is still fairly new, so you will probably will not see it adopted widely yet; you should at least be aware of it because you might run into it in the near future. Version 3.0 is in the Proposed Standard state, which means that it is stable, but lacks maturity.
This article aims to teach you the basics of HTTP debugging, giving you enough of a starting point, so that you know where to start looking whenever you run into a problem.
The HTTP Connection
For the majority of the time, the tools that we use every day, such as web browsers or mobile apps, abstract away most of the inner workings of HTTP connections. For this lesson, we will use the command line tool curl to dig a little deeper into how these connections work.
When using curl with the flags -vL pointing to https://google.com,
curl -vL https://google.comThe output is:
* Trying 142.251.15.100:443...
* Connected to google.com (142.251.15.100) port 443 (#0)
…
* SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
…
> GET / HTTP/2
> Host: google.com
> user-agent: curl/7.79.1
> accept: */*
>
< HTTP/2 301
< location: https://www.google.com/
…
* Issue another request to this URL: 'https://www.google.com/'
* Trying 64.233.177.105:443...
* Connected to www.google.com (64.233.177.105) port 443 (#1)
…
* SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
…
> GET / HTTP/2
> Host: www.google.com
> user-agent: curl/7.79.1
…
< HTTP/2 200
< content-type: text/html; charset=ISO-8859-1
< set-cookie: …
< set-cookie: …
< set-cookie: …
<!doctype html>
…
</html>I have already removed most of the noise from my curl output above. So let’s walk through it to figure out what exactly happened.
-
Before an HTTP request can be sent, a connection must be made between the client and the server. The end points of these connections are the network sockets of each system. At this step, the terminal outputs:
* Trying 142.251.15.100:443... * Connected to google.com (142.251.15.100) port 443 (#0) -
Because the URI scheme was https, curl will attempt to perform a TLS handshake.
* SSL certificate verify ok. -
curl has identified that the server supports HTTP version 2, so it defaults to use that.
* Using HTTP2, server supports multiplexing -
After TLS handshake completes, curl now makes an HTTP GET request with
curlas the user-agent.> GET / HTTP/2 > Host: google.com > user-agent: curl/7.79.1 -
The server responds with an HTTP code 301 to redirects the client to the domain with the www.
< HTTP/2 301 < location: https://www.google.com/ -
curl makes another TCP connection to the new address with the www.
* Issue another request to this URL: 'https://www.google.com/' * Trying 64.233.177.105:443... * Connected to www.google.com (64.233.177.105) port 443 (#1) -
curl verifies TLS again and makes another HTTP GET request.
> GET / HTTP/2 > Host: www.google.com > user-agent: curl/7.79.1 -
The server responds with an HTTP code 200 (Success), declaring that the content-type of the response body is
test/html.< HTTP/2 200 < content-type: text/html; charset=ISO-8859-1 -
The server also attempts to set cookies on your machine.
< set-cookie: … < set-cookie: … < set-cookie: … -
Lastly, the server includes the HTML document in the response body.
<!doctype html> … </html>
The Server Responds Differently to Different Clients
Depending on the client that you are using, the server might respond differently. user-agent might be used to decide your client type, but other request headers might be used as well.
For example, when we used the tool `curl, the server did not encode the response body at all. But if we were to use Chrome, Safari, or Firefox, we receive the content encoded in different formats.
-
Chrome automatically sends the header accept-encoding with the values
gzip, deflate, br, so the server sent the response as a br file for the browser to decompress.
-
Safari sends the same accept-encoding as Chrome does, but it is not sure why I received a gzip file instead of a br file. It could very well be just the edge node that I am connected to.
-
Similar to Chrome, I also received a br file on Firefox.
Common Debugging Tools
Besides using built-in developer tools from web browsers or terminal utilities such as curl. You can also use dedicated debugging tools. They offer an easy-to-use interface to input query, headers, or even support authentication. You can also save and re-use your queries easily.
The most popular among these tools seems to be Postman.
But the interface of Postman is a bit bloated, so sometimes I prefer simple tools such as Insomnia.
