SUS-759: Sandvine and Internet blocking in Azerbaijan

With the permission of https://www.qurium.org I am posting forensic report about internet blocking in Azerbaijan. This article has a lot of technical details.

Stockholm, 9th January 2019, 9:19 AM

Background

The government of Azerbaijan deployed Allot Communications (Israel) deep packet inspection to block websites in Azerbaijan in March 2017.
In April 2018, a new type of active blocking was introduced to block several websites we host after we managed to find a way to bypass Allot’s traffic inspection engine DART.
This article summarizes how we discovered that (1) Procera-Sandvine (Canada) is used in conjunction with Allot Communications (Israel) and (2) how Azerbaijan seeked technical support from Waterloo office in Canada to block azadliq.info
… and read more about SUS-759 Sandvine support ticket and how their technicians in India looked for ways to block our mitigation techniques.

During the first week of April 2018, we started to monitor connections performing daily tests to check the site azadliq.info. The tests included performing crafted requests to the website to learn more about the techniques we used to unblock the sites in the country.

Testing blocking from Canada and Azerbaijan

These connections were very likely originated by the same person using the following addresses:

62.212.224.24. 62.212.224.27, 94.20.30.47, 213.154.29.10 and 167.88.32.136,

Let us provide you a quick summary of what these addresses are:

62.212.224.24 and 62.212.224.27 (Baku, Azerbaijan): These IPs belongs to the network used by providers that peer in the Delta Internet Exchange. The name server of Delta Telecom is hosted at 62.212.224.21
94.20.30.47 (Stockholm, Sweden): This server hosts a VPN server running SoftEther (port 5555). During the multiple attempts to block the site, the person testing from 62.212.224.0/24 used a VPN connection in 94.20.30.47 to verify the status of the blocking. This network is announced in Delta peering in Sweden at Netnod.
213.154.29.10 (Baku, Azerbaijan): This IP belongs to bestcomp.net. Bestcomp Group is the supplier of DPI technology for Bakcell in partnership with Sandvine.
167.88.32.136 (Waterloo, Ontario, Canada): This IP is connected with Sandvine infrastructure in Canada.

Fact 1: The same user(s) (using the same browser) tested from different locations that status of the blocking of the sites. To test remotely, he connected to different VPN servers in Sweden (Delta at Netnod) and Canada.

Who runs 167.88.32.136?

A careful look into this address shows that it also hosts a VPN IPSEC server. The server also hosts the Sandvine’s open stack development environment nubo.sandvine.rocks

We looked into the domain sandvine.rocks domain and found this development connected to the GitHub account of Sandvine and the openstack account of “don“.

Who is Sandvine and Don?

Sandvine, is a networking equipment company specialized in network traffic management and Deep Packet Inspection based in Waterloo, Canada. In July 2017 Sandvine was adquired by PNI Acquireco Corp., an affiliate of Francisco Partners and Procera Networks.

In March 2018 the Citizen lab published a report showing strong evidence that equipment from Sandvine could have been used to deploy government spyware in Turkey and redirect Egyptian users.

The server with IP 167.88.32.136 that is used to VPN the traffic from Bestcomp Group in Azerbaijan hosts a development site from “Don”.

Don is Don Bowman, the co-founder of Sandvine Corporation (former CTO and now working in the start up Agilicius)

As explained in this video, Sandvine uses DPI technology to “enforce the policies” of the carriers.

Fact 2: Infrastructure setup by former CTO of Sandvine is used to VPN traffic from Azerbaijan to test the status of the Internet blocking

Who is Bestcomp Group?

According to Sandvine press release back in December 2015, Bakcell purchased a Policy Traffic Switch (PTS) from Sandvine and roll out the solution with Bestcomp Group (bestcomp.net) in Azerbaijan. Bestcomp is a well connected IT company that frequently joins the State Agency for Public Service and Social Innovation in their visits abroad.

Fact 3: According to our forensic investigation, someone from Bestcomp Group supporting Bakcell operations has privileges to analyze our current anti-DPI setup using Sandvine infrastructure in Canada.

Fact 4: Sandvine has been partner of Bestcomp Group at least since 2015.

And suddenly SUS-759 appeared in our logs..

SUS-759: Our Sandvine support ticket

Until then, we did not fully understand what role Sandvine was playing in this case and how they are currently supporting Internet blocking in Azerbaijan but we can see that our efforts to keep the sites unblocked did trigger a few support tickets. Some of these websites are blocked even without a legal case or court order against them.

Both the 7th and 16th of April, we received connections from Bangladesh and India for what it seems a Customer Support Center dealing with a case that involved the domain “azadliq.info”

The Customer Support, clicked on the ticket that included the domain azadliq.info and leaked it to us.

103.58.92.X - - [07/Apr/2018:19:58:38 +0000] "GET / HTTP/1.1" 200 33304 [522] "https://sandvine.my.salesforce.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" "---" "BD" "AS134201 Metaphor Digital Media" "www.azadliq.info" "tcp443" "TLSv1.2" "0.000" "6.18"
223.226.77.X - - [16/Apr/2018:18:25:19 +0000] "GET / HTTP/1.1" 200 32939 [512] "https://jira.sandvine.com/browse/SUS-759" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" "---" "IN" "-" "-x-" "www.azadliq.info" "tcp443" "TLSv1.2" "0.000" "6.26"

The domain “jira.sandvine.com” is not publicly available in the Internet and seems to be an internal domain name only available to the technical support.

Is there a customer support center in India?

It seems like Sandvine has a Quality Assurance and customer support in India.

The 16th of April, we learned that our case number is “SUS-759”

Fact 5: A support ticket was filled to help blocking the websites: SUS-759

A sample of the log of their activity is available here: sandvine_bakcell.log

How does this blocking look like in our infrastructure?

The mechanisms used to block the websites used by Sandvine and other DPI vendors generate storms of “RST traffic” against our servers. In same cases, for every connection attempt inside Azerbaijan we receive up to 10 RST packets back from their DPI equipment. Ultimately, their DPI equiment is flooding our network with RST traffic.

Summary

This is what we believe happened during April-May 2018 once we unblocked the websites in Azerbaijan behind Allot Communications equipment.

In April 2018, someone from Bestcomp Group, the official partner of Sandvine in Azerbaijan and with DPI equipment in Bakcell, troubleshot during weeks why the websites were online and not blocked. The person troubleshooting the setup used different connections from inside and outside the country (Canada, Sweden) to verify the blocking.

As he was not able to block the websites he/she submitted a ticket (SUS-759) to the internal support server of Sandvine: jira.sandvine.com and the support of the Canadian company outsourced in India/Bandgladesh handled the support request.

A few days after, the Sandvine equipment started to block the websites.