#charles proxy

Objective

In recent times more websites & mobile apps are using WebSockets to handle communication. Typically mobile Apps make use of WebSockets to send and receive messages, make various API calls for handling various features on the app and a lot more.

Background – What is WebSocket ?

The WebSocket API differs from the standard SOAP or REST API by virtue of the nature of its traffic.

If I was…

I've been an iOS developer (professionally) for over six years. There is one constant in every iOS app that I've written for someone else. They all communicate with a web service to read and write data from the iOS app/client. Even my personal side project apps are moving from completely isolated on the device to talking to a web service.

If you work on any sort of mobile client that communicates with a web service, you should learn how to use Charles Proxy. I've been using this tool for years and find it absolutely indispensable.

This post will give a primer on using Charles with a simple iOS project. I wrote a very simple project that will load the Curated Images from Unsplash. You can grab the source code from GitHub.

All of the setup in this post is for the iOS Simulator. But setting up Charles to talk to other devices (tvOS or real iOS/tvOS devices), is easy enough. I have written a separate guide to setting up Charles Proxy for an Apple TV device, which you can find it here: Setting up Charles Proxy on Apple TV (tvOS).

Please Note: The way I’ve outlined the instructions here is not the only way to do things. Feel free to explore the app on your own.

Also Please Note: I’ve included short YouTube videos showing the steps for the instructions below. Some of the instruction sets can be complicated and the YouTube videos should help out.

Charles Proxy Details

Before we get started, here are some details about Charles Proxy. Directly from the developer’s website:

Charles is an HTTP proxy / HTTP monitor / Reverse Proxy that enables a developer to view all of the HTTP and SSL / HTTPS traffic between their machine and the Internet. This includes requests, responses and the HTTP headers (which contain the cookies and caching information).

Website: http://www.charlesproxy.com

Cost: $50 USD

Free trial? Yes, for 30 days

Charles Proxy Overview

Charles Proxy is an HTTP proxy/monitor application. It allows you to monitor web traffic from a variety of sources, including macOS/iOS/tvOS devices and the iOS/tvOS simulator. This means that you won’t have to write temporary code like this:

print("JSON Response: \(JSON)")

or

NSLog(@"JSON Response: %@", JSON)

Instead, you'll be able to setup your device/simulator to use Charles Proxy and watch the inbound/outbound HTTP(S) traffic without making a single code change.

Setting up Charles

The steps for configuring Charles to watch iOS Simulator traffic have gotten so much easier over the years. Follow these steps to get started:

Launch Charles Proxy

Make sure that you have run the target iOS Simulator at least once.

Quit the iOS Simulator

Open Charles and navigate to Help -> SSL Proxying -> Install Charles Root Certificate in iOS Simulators

If your app is talking to a HTTPS website (as it should be), there may be a few extra steps you need to do in order to actually see the data. By default, the HTTPS/SSL response from a web service may look like this:

In order to unscramble that mess, you need to enable SSL Proxying for the domain. Follow these steps to enable SSL Proxying for the Unsplash API domain:

Open Charles and navigate to Proxy -> SSL Proxying Settings

Click on the SSL Proxying tab.

Make sure Enable SSL Proxying is enabled.

Click the Add button at the bottom.

Enter api.unsplash.com into the Host field and click OK.

Thats it. When you restart your app, Charles should start capturing your web traffic. The HTTP response will look more like this:

If you'd like to know how to install the SSL Certificates for other devices, the Charles website has instructions on how to do this.

You can use Charles as it is, or you can customize things. I don't like having all of my web traffic recorded, so I limit what Charles records on my behalf. In order to do this, follow these steps:

Open Charles and navigate to Proxy -> Recording Settings

Click on the Include Tab tab.

Click the Add button at the bottom.

Paste http://api.unsplash.com/* into the Host field and press the Tab key on your keyboard. All off the fields will be populated here.

Click OK.

Once you start adding entries here, Charles will not record traffic to other domains. This really helps in keeping your traffic logs nice and clean. It really helps when you submit defects/tickets to other people (say the web service team) with a nice clean Charles log showing what happened.

Watching Traffic

If you've made it this far, you've already seen the first major feature of Charles. You can view web traffic. This feature alone is worth purchasing a Charles license. Being able to see live data from a web service during development, testing and (even the) production phases is invaluable.

During the development phase, it's great to see the data you are getting back in your app. Sure, you can add some debug print lines (shown above), but recompiling an app just to add logging statements you'll need to remove later can be a pain.

Not only can you view traffic, but you can save the Charles session (File -> Save Session) for later inspection. In my day job, we have often asked QA for Charles logs along with defects that are created. This helps us know if the API (web service) is returning bad data or if the app is showing the wrong thing.

This technique of watching traffic and saving it later works for production issues too. If a single user (that you have physical access to) is having an issue, you can (temporarily) install the Charles SSL Cert and watch their traffic through Charles. This particular user may have something odd in their setup or account that's causing something funky going on.

Please Note: This last technique where you view a customer's data should be used sparingly. We never used it with external, paying customers. We always used this technique with internal customers. You should have the user’s permission before doing so. Having access to your customers’ logs can mean you may have access to their login information. We always have a user log into the system before we start tracking their logs. Just be careful here.

Mapping

Ok. So, I can watch traffic with Charles. What else does the app do?

One of the major benefits of using Charles is to map responses somewhere else. You can map responses to either a local file, or to a remote server. In this blog post, we’ll demonstrate mapping an API call in my Unsplashed demo app to a remote service that always returns a 500 HTTP Status code.

Using this technique is perfect for testing failures from a remote API that your app depends on without changing code to simulate the error or actually taking down the service. You can test this on an app build that is ready to ship without having to make changes. The thought of taking down a service so you can test error conditions is terrible. Don’t do that.

Remote mapping to 500 service

Great. You’re on board and want to try out this mapping thing. In the GitHub repo, there is a directory called “Charles Rules” that contains the first rule we’ll be importing into Charles. You don’t need to import this rule, you can create it yourself, but I’ve added it for simplicity.

Here’s how you import the existing rule from the GitHub Repo:

Open Charles and navigate to Tools -> Map Remote…

Click on the Enable Map Remote checkbox to enable it.

Click the Import button at the bottom.

Navigate to the Charles Rules/500HttpStatusRemote.xml file and click Open.

Click OK.

Or if you’d rather create the rule yourself, follow these steps:

Open Charles and navigate to Tools -> Map Remote…

Click on the Enable Map Remote checkbox to enable it.

Click the Add button near the bottom.

Paste https://api.unsplashed.com/photos/curated/?client_id=* into the Map From Host field and press the Tab key on your keyboard. All off the fields will be populated here.

Paste http://httpstat.us/500 into the Map To Host field and press the Tab key on your keyboard. All of the fields will be populated here.

Click OK.

Click OK.

Now, if you relaunch the app, you should be presented with an error message and see that there is now traffic being routed to http://httpstat.us/500.

The service http://httpstat.us can serve up all of the HTTP status codes including 418 I'm a teapot. You can use this technique to test all sorts of error conditions. The sample project responds the same way based on any non-200 OK response, but your app may need to respond differently based on various HTTP status codes.

When you look at the Charles traffic, you will no longer see the original URL being mapped. This also means that you’ll need to listen for the mapped URL in addition to the original one. Please read above for notes on adding additional URLs.

If you look at the response overview, you’ll notice that Charles has included a note about the response:

Mapped from remote URL: https://api.unsplash.com/photos/curated/?…

This note is important so that you know what’s happening with your messages. It’s also helpful for debugging in the next section.

Rewriting Traffic

Mapping traffic can be a really powerful tool. But Charles can do more. You can also rewrite responses from an API. Why would you want to do this? I can think of two very specific examples:

First, it’s useful when you want to return a known response on every call. This could be useful for testing with an empty result set, or testing with a funky response. Either way, you can always rewrite the response so that it includes exactly the data you’re looking for.

Second, you can handle APIs that do a bad job of error handling. I’ve worked with a few APIs that would return a 200 OK response for an error, but include JSON or XML that would include an error message. I know, we all make mistakes. Services like http://httpstat.us can’t handle this. Rewriting traffic will allow you handle any case you need to.

Rewriting

I’ve included a few example rewrite rules in the Charles Rules folder in the GitHub repo. You can import these using the following steps.

Open Charles and navigate to Tools -> Rewrite…

Click on the Enable Rewrite checkbox to enable it.

Click the Import button at the bottom.

Navigate to the Charles Rules/422RewriteError.xml file and click Open.

Creating a rewrite rule from scratch is not always easy. They can be tricky to setup and it’s easy to forget steps.

This example will return a HTTP status code of 422 (Unprocessable Entity) along with an array of error messages. This is taken directly from the Unsplash API Error Messages docs. If I were to take this example app further, I would add code to handle these error responses.

There are a lot of steps here. Please bear with me. Or watch the video below.

Open Charles and navigate to Tools -> Map Remote…

Click on the Enable Rewrite checkbox to enable it.

Click the Add button near the bottom.

Type a name into the Name field.

Click the Add button under the Location section

Paste https://api.unsplash.com/photos/curated/?client_id=* into the Host field and press the Tab key on your keyboard. All off the fields will be populated here.

Click OK

Click the Add button under the bottom Type/Rules section

Change the Type dropdown value to Body

Click the Request checkbox to that it is disabled

Click the Response checkbox to that it is enabled

Paste { "errors": ["Username is missing", "Password cannot be blank"] }  into the Value field in the Replace section at the bottom.

Click OK

Click the Add button under the bottom Type/Rules section

Change the Type dropdown value to Response Status

Paste 422 Unprocessable Entity into the Value field in the Replace section at the bottom.

Click OK

Click Apply

Click OK

When creating these rewrite rules, it’s valuable to make sure the responses are either valid XML or JSON before using them. It can be painful to edit parts of responses in the Charles rewrite modal view. I’ll usually verify that the JSON is valid before pasting it in there. In the past, I’ve spent a lot of time trying to debug my rules only to find out the response JSON is invalid.

Conclusion

I hope this primer has been useful on getting started with Charles Proxy as a development tool. There are many more features to Charles Proxy that I haven’t even touched on.
 Charles Proxy is invaluable to me as a client developer. There are too many features to cover in this article. In fact, I’m still discovering new features that I hope to use in the future.

As I continue to grow and learn more, I may cover new topics in a subsequent post, if there’s enough interest.

I use Charles Proxy as one of my development tools. I have done so for years. It’s invaluable to me. If you do any sort of client development that calls any sort of web service, you should look into Charles Proxy too.

This blog post is mostly taken from this excellent post on WillowTree development blog post called: 12 Steps For Setting Up Charles Proxy With tvOS. I recently tried to follow their steps, but was unable to get the SSL cert I needed from the the URL outlined (www.charlesproxy.com/getssl). You can now get the SSL cert directly from the Charles Proxy app. This post includes most of their steps, but have an updated step for getting the SSL cert. They deserve all of the credit here for the original post.

Set up a profile

In order to set up tvOS, you need to use a profile to setup a proxy.

Download and install Apple Configurator.

Open Apple Configurator and navigate to File -> New Profile.

Give your profile a name and a unique identifier. Typically you’ll be setting it up to go through the same PC you’re using, so indicating this is encouraged (e.g. Charles-JoeLaptop).

Select Global HTTP Proxy* on the left and click configure.

Input your IP address of the machine running Charles and 8888 (default) for the port.*

If you need SSL, you need to add the Charles SSL cert. (if you’re not tracking SSL you can ignore this part):

Open Charles and navigate to Help -> SSL Proxying -> Save Charles Root Certificate.

This will save a *.pem file containing the certificate you need. You need the certificate in a different format to add to your profile. (There are different ways you can do this, I found the simplest way was to add it to my keychain then export it as a *.cer file.)

Back in Configurator, select Certificates on the left and click Configure.

This brings up a dialog to select the relevant certificate, select the one described above and click Open.

It may say, “This root certificate is not trusted.” This just reflects the status of the user you’re using on the Mac.

Save and close the Profile.

Prepare Apple TV in Configurator

In order to be able to install profiles onto your Apple TV, the device needs to be prepared and supervised.

With the Apple TV turned on, connect it to your Macbook using the USB-C adapter.

Open Apple Configurator.

The Apple TV should show up under All Devices.

Select the device and click on the Prepare button at the top of the window.

Follow the On Screen prompts. At the “Enroll in MDM Server” prompt, I selected “Do not enroll in MDM”. I selected the defaults for everything else.

Install the profile

Now you will need to load the Profile onto the Apple TV. This requires a USB-C adapter.

With the Apple TV turned on, connect it to your Macbook using the USB-C adapter.

Open up Apple Configurator.

The Apple TV should show up under All Devices.

Select the device and click on the Add button at the top of the window.

Select Profiles and find the profile you created above, then click Add Profiles.

You should start to see web traffic in Charles Proxy from your Apple TV.

I’ve got another blog post about Charles Proxy and using it in a development/QA process. I hope to have that post out shortly, it’s still a work in progress.

Test DTM rules using charles proxy

        I am sure that while reading this post you will start doing the handson with charles proxy. Yeah, It is so good to dig in to something new. Many people out there might have done this already. Before the weekend started, I had to implement DTM on one of the client’s site but there was no header and footer code added by their developers. It was hard to get the DTM rules and Data elements…

The ability to debug web applications is baked into every major browser -- just click Inspect Element and you'll see lots of information. It's not quite as easy to do this with native apps, especially if you don't have their source code. I'd like to show you how to understand the behavior of an application by inspecting its network requests (with or without SSL). I'll also discuss some security implications relevant to developers who are building their own API (private or public).

Note: I am not a lawyer and this is not legal advice, however using these techniques may violate a product’s TOS and may be illegal in some cases. Consult a lawyer if you have any doubts.

Setup

To get going, we need to install a web debugging proxy on a computer (we'll use Charles Proxy), configure our native device to use that proxy, and install a root CA certificate on our device.

You can skip right to the Traffic Analysis section to see this in action.

1. Install Charles Proxy

Get it online at http://www.charlesproxy.com/. You can start out by using the trial version, but I highly recommend buying it. It's a great use of $50 for all of its features. Besides, you'll probably find it annoying that it closes after 30 minutes of use :-)

2. Configure your device to use the computer as a proxy

Make sure your device and computer are on the same wireless network. If you're using an iPhone or iPad: go to Settings -> Wi-Fi, then click the blue arrow on the far right of your network. Scroll down to HTTP Proxy and choose Manual. Enter your computer's IP address and 8888 for the port.

If you're using an Android: go to Settings -> Wi-Fi, long press your network, then press Modify network. Press Show advanced options, set Proxy settings to Manual, and similarly put in your computer's IP address and port 8888. If you don't see that setting, make sure you have a recent version of Android. Most older versions of Android before Ice Cream Sandwich don't let you configure the HTTP proxy, so you won't be able to use this technique.

3. Install the Charles root certificate on your device

The above setup will let you intercept regular traffic, but you won't be able to make sense of encrypted traffic. Many apps use HTTPS for their network communication, so this step will let you analyze them as well.

You'll need to install the Charles root SSL CA certificate on your device. As of May 2013 this can be found at http://charlesproxy.com/charles.crt, but check the latest documentation (here and here) if that link doesn't work. Put that URL into Safari or Chrome on your device, approve it, and you're almost good to go.

Behind the scenes, Charles will effectively perform a man-in-the-middle attack on your encrypted traffic. It will supply its own certificate to your device and the destination host, transparently encrypting and re-encrypting all data sent and received.

SECURITY NOTE: Make sure to remove the Charles certificate when you're done debugging, or your legitimate HTTPS traffic could be compromised later. See Final Notes below.

3a. Turn on SSL Proxying

Inside Charles, go to Proxy -> Proxy Settings and click the SSL tab. Ensure that Enable SSL Proxying is checked. Click Add and type in * for the host field. This will allow Charles to intercept traffic sent to any host. For fine-grained debugging, you should replace the wildcard with only the hosts that you specifically care about.

If Charles asks you to automatically configure your computer's Network Settings, go ahead and reject it. This post isn't about debugging local applications on your computer (although this has many possibilities).

NOTE: Some apps may stop working or report connection errors if you try to inspect their traffic this way. This is likely due to additional security on their part (see SSL Pinning, below).

4. Start analyzing

Open up a browser on your device. Charles should display a prompt asking you to approve your device. Click Allow and you should start seeing data appear in the main window.

Traffic Analysis

For this walkthrough, I wrote a simple iOS app that does simple authentication against a backend, sends the user's current latitude and longitude, and displays a list of nearby places returned by the backend. However, for the sake of science let's assume we're analyzing an app that we downloaded from the App Store, and we know nothing about how it works. Let's see how that looks inside Charles.

Clicking on the individual rows will show the request and response bodies. Charles comes with some very nice functionality by default, such as JSON parsing, timing charts, and more.

We can now deduce quite a bit of information:

The app is communicating with a backend at sandbox.nickfishman.com

It's using Flurry Analytics and Apple Push Notifications

It's using a RESTful web API and supplying an access_token parameter to each request

Playing with the app further would let us enumerate most of the backend's API endpoints and learn how they operate. We also have a valid access_token, so we can probably masquerade as a legitimate client and hit the API programmatically (even if the API is private and undocumented).

Useful Charles features

Setting breakpoints and modifying requests in-place: One of the most powerful features of Charles is the ability to set breakpoints on certain URL patterns. When a request triggers such a breakpoint, Charles lets you modify the request headers and body in-place before sending them on to the server. This lets you instantly test how the backend reacts to changing parameters, without needing to build even a simple custom HTTP client. In our example, we could forge the latitude/longitude parameters sent by the client and see how the backend responds to different locations around the globe.

Replaying requests: We can replay any of the requests in the list to see how the backend reacts. Right click on a request and click Repeat. We can also stress test a particular API endpoint by issuing many requests in parallel (Repeat Advanced).

Simulating latency: Charles also lets us place artificial limits on the bandwidth and latency of requests. This helps simulate slow network conditions, which sometimes cause apps to severely misbehave. For Mac users, you can do even more testing along these lines using Apple's Network Link Conditioner.

Saving sessions: You can save an entire recording session for later reference or processing.

See the documentation for more details on these features and more.

Security implications and preventive measures

This example shows how easy it is to intercept and even actively modify a native app's network traffic. Furthermore, it shows that even HTTPS APIs aren't safe from this kind of meddling. How can we build secure web APIs in the face of such challenges?

Never trust the client

Perhaps the most important conclusion: never trust the client. It's easy to masquerade as the client without disrupting any of the cryptography behind the authentication process. A client with a valid auth token can still misbehave -- either intentionally or due to buggy code. We shouldn't assume the client will always use the API as intended.

Well-known public APIs go to great lengths to handle abusive clients. Some preventive measures include rate limits, internal alerts for clients that supply invalid parameters or make strange API calls, and offline data mining to classify abusive or fraudulent behavior. You might consider implementing some of these techniques even if your API is private or undocumented. Check out nginx's HttpLimitReqModule and Netfilter limits as starting points.

Understand the limitations of HTTPS

Using HTTPS allows your app to establish a secure channel of communication with the server that prevents attackers from reading, modifying, or replaying data sent along the channel. The cryptography behind HTTPS is incredibly sound, but the protocol itself has some important limitations.

We've seen one of those limitations in this walkthrough. It's easy to intercept a secure channel with a tool like Charles if you can get the device to trust your (rogue) root CA certificate. This is straightforward if you have physical access to the device. This doesn't mean HTTPS is broken, but it does mean you should carefully consider the assumptions you're making by using HTTPS.

In general, you can assume that only your app and your backend can read data transferred over HTTPS. However, you cannot assume that the user of your app is unable to read that data. The user can discover any private API endpoints you use and learn more about how your app works under the hood. These discoveries are sometimes embarrassing (this is how Arun Thampi found that Path was uploading a user's entire address book to its servers without permission). Don't send anything over the network that you don't want the user to discover.

NOTE: The rogue root CA certificate issue can also be exploited by governments for surveillance purposes. This is outside the scope of this post, but you can learn more here.

SSL Pinning

One of the most promising solutions to the rogue root CA certificate problem is SSL pinning, also known as certificate pinning or certificate validation. In this technique, a known copy of the server's certificate is bundled with the app itself. When the app makes an HTTPS request, it validates the server's provided certificate against its known copy. If the OS validates the certificate chain against a (potentially rogue) root CA but the certificate doesn't match the app's expectations, it will reject the connection and display a network error.

The Chrome and Twitter iOS apps have already implemented this technique. For example, here's what happens if I try to use Chrome through Charles:

You can learn more about how to implement SSL pinning in this blog post and in Twitter's Security Best Practices guide. Keep in mind that SSL pinning can also be broken by jailbreaking or rooting the device and modifying the app binary (see the iSEC Partners presentation at blackhat USA 2012). However, this attack is more complicated and requires a motivated attacker. SSL pinning is still a good approach for keeping your app's API traffic hidden from casual observers.

Final notes

When you’re done debugging, it’s a good idea to remove the Charles root CA certificate from your device. Otherwise, there’s a chance that legitimate encrypted traffic from your device (such as usernames, passwords, Facebook session tokens, payment information, etc) could be intercepted by someone else on your network with Charles. It’s unlikely, but don’t take that chance. On iOS, go to Settings -> General -> Profiles, and remove the Charles Proxy one. On Android, go to Settings -> Security -> Trusted credentials -> User, and remove the Charles one.

In this guide we used Charles Proxy because it's super flexible and yet easy to use. A powerful, free, and open-source alternative is mitmproxy -- it's definitely worth checking out.

As you've hopefully seen, viewing native app network traffic is almost as easy as using Inspect Element in the browser. In general, you're in trouble if the security of your app depends on nobody discovering the inner workings of your private API. On the other hand, you're in much better shape if your app keeps all its confidential and valuable business logic in the backend, the backend uses a sensible authentication protocol, and the only real attack against your API is a DDoS.