MIT and Microsoft build a new algorithm to find hidden links between historic paintings

If you use Twitter regularly, you must’ve come across celebrities as food tweet threads. Some examples are Rihanna as Indian food and Olly Alexander as cakes. The idea is to match the look and color of the food to the celeb’s clothes.

Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) and Microsoft have created a new algorithm that matches paintings from the Metropolitan Museum of Art and Amsterdam’s Rijksmuseum through hidden stylistic connections. 

The MosAIc system is inspired by the “Rembrandt and Velazquez” exhibit in the Rijksmuseum that has paired up paintings which might seem different but actually have some connection in terms of interpretation or style. For example, in the exhibit Francisco de Zurbarán’s, The Martyrdom of Saint Serapion is paired up with Jan Asselijn’s The Threatened Swan; both of them have visual similarities in terms of posture.

The researchers spun up an algorithm that finds similar images to a query. For example: “which musical instrument is closest to this painting of a blue and white dress?” They found that result is a blue and white porcelain violin — that also helped them draw cultural exchanges between The Dutch and the Chinese. 

MosAIc is similar to Google’s X degrees of separation experiment which connects two images or artworks through a series of paintings. However, MosAIc only requires one image to find similar stylistic images. Through the input images, the algorithm tries to find matching work in different cultures.

Mark Hamilton, the lead author on the paper and a student at MIT, said that building the algorithm was challenging as their aim was to match images that weren’t just similar in color and style, but also meaning and theme. 

Hamilton and colleagues used a new K-Nearest Neighbor (KNN) data structure that puts similar images in a tree-like structure and traverses it till they find the closest result. This algorithm was then applied to the combined open-access collections of the MET and the Rijksmuseum.

Researchers also found that this method can be applied to find out the limitation of GAN (Generative Adversarial Network) based deepfake algorithms and where they fail. Though, it’s not clear if the algorithm can help to differentiate deepfakes from genuine images.

You can read the full paper on the algorithm here.