Microsoft launched a new online tool called CaptionBot that leverages AI to try and figure out what's going on in your photos.
The whole process is pretty straightforward. You upload an image of your choice and let the Microsoft CaptionBot analyze it, after which it comes up with a description of the image.
It's worth pointing out, however, that the CaptionBot will hold on to any image you submit so it can use it to improve its accuracy.
"I can understand the content of any image and I'll try to describe it as well as any human," reads the CaptionBot description. "I'm still learning so I'll hold onto to your photo but no personal info."
After analyzing a number of photos with the new tool, we've come to the conclusion that the Microsoft CaptionBot is surprisingly accurate. It can discern whether your photo features a dark, cloudy sky, a pet, a concert or some other scenario.
The tool relies on two neural networks to deliver its image descriptions. The first one handles image recognition, while the other uses natural language processing. Using the two neural networks, the Microsoft CaptionBot takes only a few seconds to come up with a caption for your photo.
The software studies labelled images in order to match image features with human descriptions of what the images captured, so it can apply what it learns when it stumbles upon similar images.
As expected, the Microsoft CaptionBot doesn't get it right 100 percent of the time. When faced with more abstract images or something that's not so obvious, it can be hit-and-miss. The image below, which features a tiny toy ladybug and a small flower, gets mistaken for a pair of scissors.
In some cases, the CaptionBot also seems to interpret some images as inappropriate, though it should not be the case. For instance, when uploading an image of a dog sleeping on a bed, covered in blankets, the CaptionBot returned the result "I think this may be inappropriate content so I won't show it."
Trying the same image again to see if the CaptionBot would offer the same result apparently puzzled the software some more. This time, it reckoned that it couldn't discern what was going on there.
Nevertheless, the results are quite impressive overall and the fact that the software is constantly improving means that even if it botches some image descriptions now, it will likely get things right in the future. The more photos you submit, the more the CaptionBot learns and improves.
The latest Microsoft CaptionBot joins a growing list of apps developed by the company's research team. Other such tools try to guess your age and gender, or detect emotions in photos. The tools may not always get things right, but at least they don't fail as hard as Microsoft's Tay AI chatbot did.
