Running a free public API, a post-mortem
Frederic Cambus November 30, 2018 [Miscellaneous]It's been a little bit more than three years since Telize public API was permanently shut down on November 15th, 2015. I have previously written about the adventure itself, and about the decommissioning of the API.
Before shutting down the public API of Telize, a paid version was launched on Mashape to ease the transition for those who couldn't host their own instances. The Mashape API Marketplace became a part of RapidAPI last year, the service is still running and will keep doing so for the foreseeable future. You can support my work on Telize by subscribing to the service.
While a small fraction of the userbase switched to the paid API, the vast majority didn't, and the number of requests exploded due to retries, as detailed in the article about the API decommission. One thing I wondered at the time was how long it would take for the traffic to become negligible. The Internet is a very strange place and things can go unnoticed for a very long time, years sometimes. Of course, Telize case is no exception.
Every year since the public API was closed down, I've been logging requests for a few days in a row to get a glimpse of how many of them were still being made. While the number of unique IP addresses querying the API kept decreasing, the amount of requests themselves went up again compared to last year.
2016-11-06 - Requests: 51,896,923 - Unique IPs: 2,543,814
2016-11-07 - Requests: 56,427,258 - Unique IPs: 2,756,065
2016-11-08 - Requests: 53,641,121 - Unique IPs: 2,746,005
2016-11-09 - Requests: 53,704,140 - Unique IPs: 2,536,632
2016-11-10 - Requests: 53,194,946 - Unique IPs: 2,525,167
2016-11-11 - Requests: 50,444,003 - Unique IPs: 2,652,730
2016-11-12 - Requests: 49,224,863 - Unique IPs: 2,670,926
2016-11-13 - Requests: 48,526,303 - Unique IPs: 2,492,765
2017-11-10 - Requests: 35,325,037 - Unique IPs: 1,736,815
2017-11-11 - Requests: 33,582,167 - Unique IPs: 1,613,161
2017-11-12 - Requests: 33,334,836 - Unique IPs: 1,587,549
2017-11-13 - Requests: 36,131,909 - Unique IPs: 1,593,255
2017-11-14 - Requests: 34,457,433 - Unique IPs: 1,571,144
2017-11-15 - Requests: 33,225,149 - Unique IPs: 1,563,845
2018-11-12 - Requests: 50,612,559 - Unique IPs: 611,302
2018-11-13 - Requests: 50,858,236 - Unique IPs: 640,836
2018-11-14 - Requests: 51,991,454 - Unique IPs: 661,410
2018-11-15 - Requests: 53,008,712 - Unique IPs: 689,646
2018-11-16 - Requests: 51,651,814 - Unique IPs: 686,646
2018-11-17 - Requests: 49,236,779 - Unique IPs: 662,717
2018-11-18 - Requests: 47,237,596 - Unique IPs: 692,718
2018-11-19 - Requests: 51,679,888 - Unique IPs: 735,396
2018-11-20 - Requests: 50,245,134 - Unique IPs: 755,177
2018-11-21 - Requests: 50,745,725 - Unique IPs: 773,949
2018-11-22 - Requests: 50,609,750 - Unique IPs: 786,963
2018-11-23 - Requests: 49,991,775 - Unique IPs: 687,652
2018-11-24 - Requests: 47,479,703 - Unique IPs: 584,058
2018-11-25 - Requests: 47,346,829 - Unique IPs: 597,153
Bandwidth usage, measured with nload:
Incoming:
Curr: 2.44 MBit/s
Avg: 2.08 MBit/s
. .... ... . ... ...... ... .. ... Min: 1.76 MBit/s
#########################|######################|################# Max: 2.70 MBit/s
################################################################## Ttl: 2217.35 GByte
Outgoing:
Curr: 1.22 MBit/s
Avg: 1.07 MBit/s
Min: 904.10 kBit/s
.... .. . Max: 1.45 MBit/s
################################################|################# Ttl: 1111.68 GByte
So more than 3 years after the decommission, I'm still getting around 50 millions daily requests. I'm honestly quite astonished to notice that the numbers went up again significantly this year.
Below is a report of user agents which performed more than 1M daily requests on November 12th 2018, top offenders being Android applications and Wordpress sites... How surprising.
Apache-HttpClient/UNAVAILABLE 24,729,923
Dalvik/2.1.0 1,113,530
WordPress/4.2.21 3,200,750
WordPress/4.1.24 2,223,350
WordPress/4.3.17 1,212,849
On a more positive note, those are recent Wordpress releases, which means it might be possible to identify the plugins performing those requests and contact their authors.
Regarding the open source project itself, I released 2.0.0 back in March, which is now using GeoIP2/GeoLite2 databases, as GeoIP/GeoLite databases have been deprecated since April. I'm currently working on a rewrite in C using Kore, which will bring in a couple of improvements compared to the current version. I will write about the new iteration in a following post.