Facebook has launched a new artificial intelligence system to better understand memes that are often posted on its platform and identify offensive ones.

According to a blog post, Facebook underscored the need for such a system as its platform is flooded by photos that contain text in various forms, whether overlaid on an image in a meme or inlaid in a photo of a restaurant menu.

Facebook is overwhelmed by the sheer volume of photos shared each day, and understanding the text in images poses an enormous problem for the social media company. Facebook already has its traditional optical character recognition (OCR) systems, which gives it the capability to recognize characters but cannot understand the context of the associated image.

To address this problem, Facebook has created and deployed a large-scale machine learning system named Rosetta. The AI has the capability to extract text from more than a billion images and video frames on Facebook and Instagram daily and in real time. The system then inputs it into a text recognition model that is trained and designed to understand the context of the text and the image together.

How The System Extracts Text From Images

Facebook performs text extraction on an image via detection and recognition. In detection, the platform detects rectangular regions that potentially contain text, while recognition involves a convolutional neural network (CNN) that helps recognize and transcribe the word in the region.

The platform also adopted a system based on Faster R-CNN for text detection.  Faster R-CNN is a state-of-the-art object detection network that performs detection and recognition at the same time, which would allow Facebook to better understand the text in a meme and determine if they are offensive.

According to Facebook, the text extracted from the image is being used to improve the photo search in terms of quality and relevance, while automatically pinpointing which content violates the platform's hate speech policy in different languages. The system will also help improve the users' News Feed by giving them more personalized content.

Rosetta Widely Used By Different Facebook Products

Currently, Facebook revealed that Rosetta is already being widely used by various products and programs within its platform and Instagram.  Facebook admitted that if faces an uphill battle with the influx of images and videos that are being shared in its platform, but the creation of an AI system is a good start.

Facebook added that extracting text from images and videos is an exciting challenge for the company, since people are finding more ways to share content via the social media platform in different forms and languages.