When AI image generator services were opened to the public on the vast web in 2021, they quickly became viral sensations—people used them as weapons in the constant effort to make more and more absurd memes, creating increasingly unhinged images: under the TikTok hashtag “cursed dall e mini,” popular for a time on the app in 2022, one can find images generated in Dall E mini with prompts ranging from the relatively innocuous “minion committing arson,” and “a bottle of ranch dressing testifying in court,” to the racier “Donald Trump twerking” and “Jesus break dancing at the capital riots.” Even the Pope was not immune from AI: the Balenciaga Pope became an instant meme earlier last year, fooling several who believed the image to be real. But it wasn’t: it was made with Midjourney.
I, like many others, started to wonder about the possible uses of AI in historical visualizations and reconstruction—will we ever reach a time when we can ask an AI model to create a snapshot of the ground of a Buddhist monastery in Pakistan in the 3rd century CE and wait for the program to work its magic? Or—more realistically—can AI generate reconstructions that can be useful tools for the art historian and archaeologist alike? The answer is both yes and no.
AI certainly seems to be a promising tool in the restoration of damaged objects and in the reconstruction of partially or totally destroyed artefacts or buildings of which some documentation exists. The AIs are indeed quite good at filling in the gaps when the big picture plan or pattern is known and can therefore be trained to master it. In 2021, for example, an AI model was trained to mimic the painting style of Dutch master Rembrandt van Rijn (1606-1669), so that the machine could reproduce the missing parts of the Night Watch. The painting had been trimmed down in 1715, some forty years after the artist’s death, to fit between two doors in Amsterdam’s city hall, with the cut-out parts being thrown away or lost. The AI managed to successfully reproduce the missing part after training on a high-resolution scan of the original Night Watch and a copy of the untrimmed painting made by Gerrit Lundens in the mid-17th century.
The Night Watch (1642) by Rembrandt at the Rijksmuseum in Amsterdam. The AI painted the panels added around the border of the large canvas. Photo: Piroschka van de Wouw/Reuters.
AI models versed in inpainting—the conservation process where missing parts are filled in to present a full picture—are used time and time again to digitally restore damaged art: in 2022, a team of researchers asked DALL-E 2 to reconstruct the missing pieces of some Roman mosaics using its feature called “outpainting,” which allows the user to submit incomplete images to the AI and ask it to fill the lacunae. The outcome was mixed: while at a first glance the results look somewhat credible, the AI generations break down at a closer inspection: for example, in a 2nd–4th CE mosaic from Antioch representing the fight between a warrior and an amazon, DALL-E not only gave the warrior’s horse an extra leg, but it also made its face into a monstrosity right out of a H. R. Giger’s illustration.
AI’s reinterpretation of a horse in the virtual restoration of the Amazon battle in Antioch. From Moral-Andrés, Fernando, Elena Merino-Gómez, Pedro Reviriego, and Fabrizio Lombardi. “Can artificial intelligence reconstruct ancient mosaics?” Studies in Conservation (2023): 1-14, p. 6.
In the Mogao Caves in Dunhuang, deep-learning models are being used to fill in the damaged parts of the murals and restore them to their original look. Clearly, as the researchers acknowledge in this study, the artifacts themselves remain unchanged: “In virtual restoration, the operation targets are images instead of objects themselves. This approach can avoid damages to the original ancient paintings, and minimize the risk of physical protection and repair process.” But there’s more. Already in 2018, researchers working in collaboration with the Dunhuang Academy had developed a style-aware model to extract line drawings from mural paintings. The final results, though not quite perfect, are rather incredible and make for a great base for further drawing and refinement by human hand.
On the left, the original painting; on the right, the AI model’s line drawing. From Pan, Gang, Di Sun, Rui Zhan, and Jiawan Zhang. “Mural sketch generation via style-aware convolutional neural network.” In Proceedings of Computer Graphics International, 2018, pp. 239-245, p. 244.
Proponents of AI advocate for its use in other fields too, especially in connection with the analysis of great quantities of geospatial and archival data: deep-learning technology, together with LiDAR (a remote sensing method used in professional mapping) has been used in the heavily forested region of Utrechtse Heuvelrug, in the Netherlands, to identify and classify archaeological features in the landscape; CNNs (Convolutional Neural Network) can classify objects in typologies—especially pottery—and it has been used successfully for Tusayan White Ware from Arizona, for artefacts from the southern Levant, and for sherds from the site of Guadalupe in Honduras, among others; recently, machine learning methods helped uncovered hidden writing and preparatory drawings in the Buddhist caves of Mogao, in Dunhuang. This is just a glimpse of the many tasks AI and its subsets can be put to.
But what about archaeological and historical reconstructions? What if an AI is asked to recreate the original look of a Ghaznavid mosque of the 11th century, or that of a 3rd century Buddhist monument, now almost lost to time? Things are a tad more complicated here. While the publicly available AI models can mix and match whatever sources they have been trained on to generate pseudo-historical fantastical vignettes, they seem to fail at recreating ancient architectures with the accuracy needed for any usable scientific reconstruction. Truth to be told, the final result is heavily dependent on the prompts these image-generator models are fed; at the same time, there’s only so much a prompt can do. To satiate my curiosity, I attempted to generate a viable reconstruction of a 2nd-3rd century stupa from Gandhāra (an area in modern Northwestern Pakistan) using two freely available AI image-generators (Canva and Leonardo.ai): in both models, I started with the simplest prompt (“a Gandhāran stupa”) and added details with every new generation (type of stone, decoration, shape of the podium, style of the columns, etc.).
The results are a mixed bag, and you can see some of the final images here. Despite describing the stupa in the final prompts as a “solid mound-like/hemispherical structure” the majority of the images show either structures with doors, tower-like architectures, or both; the AI models also ignored my requests for “a finial on top of the stupa made of at least four stone disks on top of each other decreasing in size,” and “a square solid platform.” They all ignored my description of the stone as “dark grey” or “grey with a blue tint,” and only in three images did the AIs try to depict the stairway towards the stupa drum that I asked for. None of the generated images are satisfactory from a scientific point of view—unlike the reconstruction of the stupa of Amluk Dara made by Francesco Martore—but they are certainly entertaining.
“Gandhāran” stupas generated with Canva.
“Gandhāran” stupas generated with Leonardo.ai.
The reconstruction of the main stupa of Amluk Dara, in the Swat Valley. Drawing by F. Martore.
As many critics say, what AI models lack is imagination—even more poignantly for us, they lack a certain kind of sensible and learned imagination that joins historical evidence and creativity, a skill that is at the core of images made for historical reconstructions.
That might change in the near future, however. AI technology evolves incredibly fast, at a speed that might barrel us into the realm of creative and imaginative AI sooner than we think. This poses new challenges for us scholars, art historians, archaeologists, and artists alike, challenges regarding personal privacy, the sustainability of these models, their relation to the onslaught of ‘fake news,’ and so on. A hotly contested debate concerns the ethics of AI, especially in relation to issues of creative ownership and copyright. Artists have loudly protested the advent of AI art—and rightly so: while AI proponents hail these image generators as agents of democratization in the arts—according to these factions, now everyone can make art freely, regardless of their talent or skillset—the truth is that AI models are often trained on real artists’ work that has been lifted from the internet without their consent. Additionally, image generators are often used to create images mimicking specific artists’ style without giving them any credit; these plagiarized images then often compete against the originals for a fraction of the price (and oftentimes for a fraction of the quality as well). Artists across the board—and creatives working on scientific and artistic reconstructions are a vocal part—are fighting back with calls for AI bans and restrictions, but also with lawsuits.
The path ahead seems difficult and uncertain—while there are still many flaws right now, I wonder if soon enough AI will be able to produce visualizations that can be useful for us–for example, it might speed up the process by providing base images that can be modified and “corrected” by human hand, or it might be used to generate multiple options at a faster pace so that the expert can have more creative possibilities at their hand when deciding what kind of final image they want to produce, and so on. Nothing can substitute drawing by hand as an analytical tool, that is true, but it is also true that AI seems to offer a new set of interesting tools that is able to produce images faster than the human hand. One thing is sure: despite the pushback among the many fearful cries that AI is going to take over the world, AI is here to stay. We in the Humanities must not only confront it, but perhaps even learn to accept it and embrace it as a new productive tool for our own work.
The Digital Orientalist 
One thought on “AI, Machine Learning, and Neural Networks: Friends or Foes in the Historical Investigation?”