ASU students share America’s untold stories using AI, AR

When the topic of how artificial intelligence (AI) might impact teaching and learning among educators, Retha Hill was already ahead of the curve. 

As a professor and the executive director of the New Media Innovation and Entrepreneurship Lab at the Walter Cronkite School of Journalism and Mass Communication, Hill has been incorporating the use of new and emerging technologies in her curriculum for immersive, enriching storytelling for nearly a decade. 

"In our lab, we focus on the latest technology to better get news and information out to our media consumers,” she said. “We've experimented with augmented and virtual reality, AI, 360 videos, and more. Innovation is about combining the best of visual and auditory storytelling to engage the audience in a format they love."

Each semester, Hill's course challenges students to tackle storytelling using new tools and technology. 

And in fall 2023, students took a journey back to revolutionary times as America celebrates the 250 Anniversary of the Boston Tea Party this Dec. 16, 2023.

Traveling to the past using AI and augmented reality 

Among the nation’s most iconic events and one that propelled America down the road to independence, students were tasked with telling the stories of ordinary people involved in the American Revolution. Inspired in part by historical portraits that hang in the Capitol Rotunda, the concept of a "Hall of Ordinary Revolutionists'' was born. 

The projects, which were completed by students over the 16-week semester, offer entirely 3D worlds detailing a historical scene through the experiences of the “ordinary revolutionist” they identified. These individuals are depicted inside the virtual world through full-color portraits, historic scenes, and audio stories, all developed using generative AI tools.

For example, ChatGPT and Transkribus were used to help translate old English text into more modern language, MidJourney and Dall-e helped to generate images, D-id.com to make talking portraits and Eleven Labs recreated the audio narration. The immersive digital environment was powered by Unity while Spatial helped create historically accurate 3D settings.

Students spent time researching records, diary entries, letters and ancestral photographs to bring the story of their character to life in an authentic way. Dialect and accent study informed the AI-generated audio for each individual to even further immerse the viewer in the story. 

For example, one project followed the experience of Hannah Till, who was the pastry chef of George Washington during the encampment of Valley Forge. 

ASU students bring American heroes to life

Meet Hannah Till, presented by Devin Harrell

As the students noted in their presentation, Till represents the many enslaved people and women who helped contribute to the revolution and encampment. Although she was born into slavery, she arranged for her freedom and the freedom of others in exchange for preparing meals for Washington and his officers.

Winters at Valley Forge were brutally cold with howling winds and barren forests. The presentation combined visual and audio elements, created using generative AI and free special effects files, to depict the scene at what is now the historical park.

Using generative-AI created audio, the students are able to recreate Till’s voice to offer a narration of her experience: 

“My owner, Reverend John Mason lent me to General George Washington to serve him as a pastry cook and made it so that some of the shillings he's paid is set aside so that I may one day purchase freedom for myself and the child I'm carrying. I started with my husband and some others in the house where General Washington stays. This encampment is a dreary place that doesn't provide to keep general Washington and his officers fed is a struggle. There is little food and men are starving. Soldiers are hunting and fishing, and we've run out of vegetables. So General Washington sent out some parties to find farmers with offers to pay for food. A summer of battles have left these men discouraged and disheartened. But being in general Washington's trust, it means my duty to raise spirits by keeping everyone fed, so they can continue the fight. It also means making sure the food is safe. So no harm is brought upon General Washington or his officers.”

Meet William Tennent III, presented by Nate Fox

A second project shared by Hill’s students told the story of William Tennent III. Tennent was Presbyterian preacher and fierce defender of religious freedom – one of his documented speeches is at the crux of the scene presented.

Taking place in 1775 outside a small clapboard chapel in the backwoods of South Carolina, Tennent is shown standing before an audience on a mission to persuade loyalists to support the revolution. 

The student who presented on Tennet noted they used Transkribus, an AI powered text translation tool, to translate the original speech, which was in Old English and fine script. Here’s a clip of a documented speech given by Tennent:

“While our current government might be inclined to uphold religious freedom, future administrations could potentially restrict the liberties of certain Christian denominations to favor one in particular. It is imperative therefore to secure the strongest possible safeguards against any encroachments on religious freedom by any sect or group to ensure the enduring protection of this utmost freedom. In matters of faith, we must earnestly petition for the inclusion of a specific clause in our state's foundational constitution. This clause should decree that no single Protestant denomination or set shall ever be established as superior in this state. Furthermore, it should guarantee that no Protestant citizen will be denied their civil rights based on their religious beliefs ensuring that all Protestants who respect and abide by the laws of this constitution shall enjoy equal and unfettered civil and religious liberties.”

While Tennent's historical speech reflects similar constructs in what would later become the First Amendment, Tennent himself is not recognized for contributing to drafting of the Constitution.

Students also covered other Ordinary Revolutionists, including:

• Nancy Morgan Hart and Esther Debert Reed by graduate student Katherine  Hebert
• Reverend William Emmerson by Darrell Harrell
• Margaret Catherine Moore Barry and Deborah Sampson Gannett by Sabrina Baker
• Prince Dunsick and Sarah Osborn Benjamin by Nick Parks
• Joshua Muriel Mullins by Nate Fox

“Students went on an exciting learning ride, blending tech and history. From exploring with AI to crafting the Hall of Ordinary Revolutionists, they got hands-on with the American Revolution,” Hill said. “This project isn't just about history; it's about students diving in and really understanding the different stories that shaped our nation."