Twitter releases open source Anomaly Detection tool

Most of the time we’re pretty normal.

We do the same thing every day. We get up, go to work, come home, go to sleep. At weekends we pop out to the cinema, and maybe take our partner to a restaurant.

Nothing very unusual.

But every now and then, fed up with the bad weather or just tired of the same-old same-old, we book an overseas holiday.

And for two weeks our lives are full of fun and sunshine. We go out to dinner every day, rent a boat to take us around the island, and spend a fortune on gifts for the friends we left back at home.

From a credit card’s point of view, that behaviour looks unusual. Indeed, you might have had your bank ring you up, asking to verify that you really are visiting country, perhaps worried that your card might have been stolen.

What the bank’s computer systems have detected is an anomaly. And as part of the fight against fraud, they are checking in with you that everything’s okay before they cancel your card. This, by the way, is a good reason why it can be helpful to tell your bank if you are travelling to the different part of the world for a short time.

Anomalies, of course, don’t just happen with banks.

They can happen on websites too.

Twitter this week released an open source tool that can be used by software developers to detect various types of anomalies.

Twitter has been using the tool, called AnomalyDetection, to detect anomalies on its platform.

Anomalies might occur on Twitter when, say, there is a breaking news story, or the final episode of a cult TV show airs.

In a blog post about its anomaly detection technology, Twitter describes the spike in the number of photos it saw uploaded to Twitter on Christmas Eve, Christmas Day and New Year’s Eve.

That’s clearly a legitimate anomaly, because of the days on which it happened, but the tool could also be used to detect malicious behaviour on Twitter, such as the spike in traffic that could be caused by spammers or bot activity designed to augment follower counts.

If you didn’t think you were being watched on Twitter, think again.

As we engage more and more with online systems and cloud services, chances are that internet companies will be trying to learn more about our activities and find a way to make use of big data.

Sometimes it might be to determine if newly rolled-out systems are working properly, if a user interface change is making a difference in engagement, or to predict if malicious activity is taking place. Other times, it might be as a simple as determining group behaviours in order to make sense of the chaos, and make recommendations and change advertisements on the fly.

Twitter says it is open sourcing AnomalyDetection in order to encourage others to contribute their expertise to the project:

“Early detection of anomalies plays a key role in ensuring high-fidelity data is available to our own product teams and those of our data partners. This package helps us monitor spikes in user engagement on the platform surrounding holidays, major sporting events or during breaking news. The package can be used to find such bots or spam, as well as detect anomalies in system metrics after a new software release. We`re open-sourcing AnomalyDetection because we`d like the public community to evolve the package and learn from it as we have.”

You can read more about AnomalyDetection on Twitter’s engineering blog.