Gain a deeper understanding of a public or private API, especially for ones that aren’t well-documented
Joyce Lin · Follow
Published in · 6 min read · Jan 2, 2019
When you think about reverse engineering an API, perhaps corporate espionage or something legally or ethically questionable comes to mind. Hackers gonna hack. However, there’s also legitimate reasons for reverse engineering an API.
APIs aren’t always documented. When you’re debugging an API, you can diagnose and resolve issues more quickly when you can visualize all the data that is sent and received. Doing this allows you to gain a deeper understanding of an API.
Reverse engineering, also called back engineering, is the process by which a man-made object is deconstructed to reveal its designs, architecture, or to extract knowledge from the object.
Knowing how an API actually behaves enables you to identify flaws and security vulnerabilities like accidental data leakage. This also performance tests your API so you can isolate bottlenecks that could benefit from additional caching and compression.
Of course, there’s also selfish reasons for reverse engineering an API.
- Accessing a service programmatically that doesn’t offer a public API
- Creating an interface that suits your needs because theirs just ain’t cutting it
- Contribute to a feature that the engineering team is slow 🐌 to deliver
- Hack with the hopes that you’ll dazzle the engineering team and they’ll offer you a job
- Order extra bacon on your pizza when the available app limits you to a sub-optimal amount of bacon
- Cheat at games — there’s an entire subreddit dedicated to this among others
- Scale up your catfishing side business by unleashing chatbots on popular dating apps
When you’re reverse engineering a private API that isn’t yours, make sure to check the terms of service. Some developers wear their cease and desist orders like a badge of honor, but you might get sued or banned from their services.
A web proxy server is like a middleman that sits between your client application and the server. The web proxy is a substitute server that can intercept HTTP traffic between a web browser and the website’s real server.
It’s all there! Black and white, clear as crystal!
— Willy Wonka, The Chocolate Factory
Reasons to use a web proxy
- Monitor and eavesdrop on HTTP network traffic by recording and displaying all traffic between your client and server
- Hide your public IP address while browsing websites and to access blocked content
- Filter or redirect requests to limit access to content or optimize systems performance
Free web proxy tools
- Postman is a free tool with a bunch of different proxies, including a built-in proxy to capture HTTP requests in the native apps for Mac, Windows, or Linux
- Mitproxy is an open-source proxy with a command line interface, web interface, and Python API
- Fiddler is a free web debugging proxy with support for a wide variety of browsers, systems, and platforms
- Burp has a free community edition of a web proxy server that lets you view and modify requests and responses
The remainder of this article will focus on how you can use Postman to intercept the traffic between your client and server.
It’s time to get our hands dirty, sniffing and inspecting to our heart’s content. First, we’ll import a single request into the Postman app. Then we’ll use Postman as a proxy to capture a stream of HTTP/S requests from a variety of clients like a desktop web browser, a mobile device, and an Electron app.
To follow along in Postman, click the orange +New button in the top left of the Postman app. Under the Templates tab, search for Reverse engineering an API, and import the sample collection into your instance of the Postman app.
Read through the descriptions in the Postman app for details, or check out the web documentation for step-by-step instructions and screenshots.
Import a single request
If you want to inspect a request in Postman, here’s a super simple way to import it as cURL from your browser. In this example, we will use Chrome DevTools to inspect and select a request. You can follow similar steps when using other web browsers.
Inspect a stream of requests
If you want to inspect a stream of requests from your client, you can use the Postman built-in proxy to capture these requests. Postman has a bunch of different proxies. In this scenario, we’ll rely on the Postman built-in proxy in the native apps for Mac, Windows, or Linux. You can capture requests sent from your client, like a desktop web browser, mobile device, or an Electron app.
You can pipe this stream of requests to your Postman History and save them to a Postman Collection. Then you can revisit these requests for inspection at a later date, or share them with a teammate.
Currently, the Postman built-in proxy in the native apps only captures HTTP request traffic. Fortunately, most websites have HTTP Strict Transport Security (HSTS) enabled for an additional layer of security. Unfortunately, this means the Postman built-in proxy cannot capture requests sent over HTTPS if the website has HSTS enabled.
Note: As of the publication of this article, the Postman built-in proxy captures HTTP traffic, but not traffic from HTTPS websites with HSTS enabled. Interceptor integration and HTTPS proxy is slated for development in the Postman roadmap for developers.
This recipe is just the tip of the iceberg of how you can sniff and inspect HTTP traffic to start understanding what’s going on under the hood. For tougher nuts to crack, you may have to dig into SSL certificate pinning, spoof particular client attributes, or learn how to sign and authenticate more complex requests.
Although these tools and methods are powerful and can be used for selfish reasons, they can and should be used for good.
Be excellent to each other.
Tools like Postman can enhance the visibility of client requests, making them easier to replicate and tweak, so you can diagnose and resolve issues faster. Ultimately, this will help you gain a deeper understanding of a public or private API, especially for APIs that aren’t well-documented.
