Methodology

Reviewing VPNs is an ongoing 10+1 step process:

The word review dates back to the 15th century. It’s a noun denoting a formal inspection of military or naval forces. Term “review” originates from obsolete French reveue, from revoir – ‘to see again‘. 

VPN service reviews come from all directions, but nothing makes us happier than finding a leakproof provider who has made an effort for its customers and of course the hidden gems – services that are truly undervalued.

It all starts with a list of companies that have a ready-to-use product and could be considered as trustworthy. Trust is the core of all VPNs.


1. Company Profile

We search their court case history, logging, location, whether they have separate companies for payment processing, who are the owners, reputation, and more. We sometimes even use lawyers to take a look at their documents.

Company profiles usually give us an overview of the companies’ business practices. Privacy policy and terms of service are the first to look into:

HotSpot Shield VPN logging

2. Customer Support

We ask them a wide range of questions to see how they interact with customers. There have been cases when a provider had a live chat feature, but the operators never reacted to the initiated conversation.

Sometimes it takes a lot of back-and-forths.

Some companies have flawless support. The agents are personal and interested in making the customer feel at ease. Yet, some have enormously strict rules about responses, which makes us feel like we’re talking to robots.

Did you know?: Out of 91 contacted VPN services Cloak VPN took the longest time to respond to our e-mail – 11,841 minutes.
(Nonetheless, there are companies which never respond.)

 

3. Service Tests

We purchase VPN services for testing

4. Setup Files Are Virus-Tested With VirusTotal

After installing a VPN service, all setup files will be tested to find suspicious malware. This is both for Android and desktop devices:

VPNinja Virustotal.com test

5. Usability

You’ll never know what’s coming..

Our testing device desktop.

Setting up the software is part of the user experience and it’s important to take note of the time spent on configuring the client. Not all VPNs have a standalone program. If the configuration files have to be integrated with VPN implementing software like OpenVPN, we note it as a drawback.

Usability comes next. We will tune every knob and checkbox and see which of them gives errors.

6. Speed Tests

ShadeYouVPN. Maybe a bit too low.

We do one test without VPN and right after that try 3 countries from their server list. The countries include USA, UK, and China, though we have had to use Japanese servers due to lack of available servers.

This gives an overview of speeds on 3 continents.

However, note that the speed varies by ISP and location. The speed tests are mainly done to get an overall comparison of every VPN.

7. DNS leak Tests

Leak tests are done once we’re connected to the provider’s service (random server). We use 5 main web tools to detect DNS leaks.

8. Netflix and Torrenting Availability

At least 3 servers, will be tested for Netflix and torrenting. If a connection is established the first time when torrenting with VPN server, we don’t repeat it the process. However if it fails, we will try 2 more random servers.

9. VPN Abroad (China Availability Tests)

Traveling to China is expensive. Occasionally, when one of our team-mates visits mainland China, some tests are made. Yet most of the times, we’ll use the help of our partner.

Tests are conducted with China Unicom Shandong (CHINA UNICOM china169 Backbone) ISP.

10. Usability Tests

If the provider offers service on iOS devices, we also do a simple usability test to see how navigation through the application feels for the users.

Example: PIA lets you choose between light and dark mode on your iOS device.