Staying Ahead of Evolving Security Threats
AMY WHIPPLE
In the midst of last winter’s holiday season, an alert popped up in Marinus Analytics’ Missing Persons Watch. The picture of a teenage girl, reported missing a month earlier, now sighted and matched with an online advertisement for commercial sexual services. The alert immediately kicked off a federally driven task force. Within days, the girl was recovered.
Without that alert, authorities might never have known to open a sex-trafficking investigation in the first place.
Most cases of missing teenagers start with local authorities searching last known and familiar locations. The federal government and its resources might never be involved if the missing person is more than 12 years old. “Sometimes these youth might be falsely assumed to be runaways instead of victims of serious crimes. Missing Persons Watch is helping to break cases,” said Marinus Analytics co-founder Cara Jones.
Missing Persons Watch passively monitors bulletins of disappearances and checks them against the millions of records gleaned through Marinus’ flagship program, Traffic Jam. The program does what individual detectives cannot: it uses AI to comb public web advertisements for online sightings and evidence of human trafficking. It indexes sites and proactively detects vulnerability indicators of trafficking so they can be accessible to law enforcement and other front-line investigators to aid the interdiction of trafficking and accelerate missing persons investigations.
In the first year of Missing Persons Watch, Marinus scanned 20,000 records. They found 500 missing persons in commercial adult services advertisements — 415 were girls between the ages of 13 and 17. “This is a systematic defense in this day and age that is required to protect the next generation,” Jones said.
Marinus Analytics spun out of the Auton Lab in the Robotics Institute, where co-founder Emily Kennedy (DC 2012) created Traffic Jam as part of her undergraduate thesis.
Cara Jones, Co-founder of Marinus Analytics
Emily Kennedy (DC 2012), Co-founder of Marinus Analytics
The company’s work has always been grounded in software-as-a-service, which Jones believes was still somewhat novel in the public sector at the beginning. And, by nature of its specialty, they had also already thought through AI policies surrounding data protection, privacy and ethics by the time those became a larger industry conversation.
That larger conversation includes other security-based spinouts, startups and alumni from CMU’s School of Computer Science (SCS). They, alongside Marinus, must grapple with the speed of AI change and the ever-increasing complexity of contemporary cybersecurity.
Jones sees AI fundamentals as part of Marinus’ motivation. “In this digital age, how do we protect the vulnerable from technology-enabled harm? Often, safety is an afterthought.”
Co-founders of Gray Swan
Zico Kolter, MLD Department Head
Matt Fredrikson, Associate Professor in CSD and S3D
Andy Zou, Ph.D. Student in CSD
Zico Kolter, deparment head of MLD believes that AI models bring a new and different perspective on what security and safety mean.
Kolter is also the chief technical advisor for Gray Swan, which he founded with fellow SCS faculty member Matt Fredrikson and SCS Ph.D. student Andy Zou. The company spun out from the trio’s research in creating effective safeguards for large language models (LLMs).
“Traditional computer programs do exactly what the developer said,” noted Kolter. “We still have security flaws in traditional programs, of course, but it’s basically because the developers make mistakes.” Responses in modern AI systems come from the combined internal data used to build them and are then used to fine tune them or to prompt them. “AI is bringing a very different perspective on what security and safety mean,” Kolter said.
Fallout from security flaws in generative models could range from an individual crisis, like exploiting someone’s AI email assistant, to a mass casualty — creating bioweapons from LLM instructions. “Most of the agents that are currently deployed actually have a lot of these vulnerabilities literally right now,” said Zou. “You can use them for targeted manipulation, misinformation or getting harmful content instructions.”
Initially, targeting malicious requests involved AI models responding with a simple refusal. However, Kolter said, “You could just trick the model, basically, into ignoring its instructions not to do something.”
Also, training models with known attacks only served to make the models refuse known attacks. Zou said their research found that malicious requests follow a generalized pattern that can be interrupted. In teaching models to focus on the pattern and “circuit breaking,” the specific prompt becomes irrelevant.
To test safeguards built on circuit breaking, Gray Swan created the Arena, a digital battleground for red teamers — professional adversaries — to try to successfully deploy an attack against various AI models. They add new challenges regularly with different themes and, often, monetary awards.
The Arena highlights AI’s ever-changing security risks that, Zou said, none of the academic, static benchmarks get at. More so, companies “rush to deploy AI agents” in increasingly complex environments, said Zou. “Lack of tested safeguards and unmonitored models can result in actions that are in many cases irreversible.”
Yinglian Xie (SCS 2002, 2005), Co-founder of DataVisor
One irreversible action both Zou and Kolter noted is financial fraud. That problem is the heart of DataVisor, another security platform that deals with fraud and risk, co-founded by Yinglian Xie (SCS 2002, 2005).
Xie first examined network security as a Ph.D. student in SCS in the early 2000s. She later moved into application-level work, which led her to fraud and abuse concerns. There, she realized the need for large-scale data processing to safeguard against malicious acts.
She and a colleague founded DataVisor in 2013 after websites like Yelp, Pinterest and Facebook approached them for user security data analysis and problem solving. “Whatever functionality you push out, there are ways to commit fraud,” said Xie. The most lucrative of which is in the financial service sector. “It’s an abusive situation with a massive impact,” she said. As AI advances, attacks become increasingly intelligent and aggressive.
A particular challenge is that the financial sector, by its regulated nature, is conservative when it comes to adopting change. The old way of doing things — the pace of researchers working and publishing, someone turning those findings into a market reality and allowing a product to mature — no longer best serves fraud, financial crime and cybersecurity.
“Nowadays, they’re mixed together,” said Xie. “You have to build it and launch it to the market and show the impact.”
Xie said she and fellow CMU alumni are notably suited to the task at hand. The university’s culture of hands-on learning pushed her to not only research and create on a superficial level but to make her ideas a functional reality. “You have to build. You have to make things work so somebody finds them useful.”
Predictive Analytics for Real-Time Decision-Making
Lovelace AI, co-founded by former dean of SCS Andrew Moore, is useful in high-stakes scenarios that include national security, disaster response efforts and infrastructure resilience.
Leah Nicolich-Henkin (SCS 2016) is a senior machine learning engineer at Lovelace who received her master’s degree in the Language Technologies Institute. “We’re building real-time predictive analytics to help organizations make smarter decisions by understanding global movement patterns,” said Nicolich-Henkin. In part, Lovelace AI pulls data from traditional sources like news articles and textual content. “What sets us apart is when we’re dealing with real things moving around in the physical world,” she noted.
Leah Nicolich-Henkin (SCS 2016), Senior Engineer at Lovelace
Nicolich-Henkin’s work involves parsing data from satellite photos and images created from synthetic aperture radar data to give potential meaning to the locations of ships and airplanes. “Just having all that data is not useful to anybody,” she said. “How can we take those images and find useful information with them and fuse them with the other data?”
Images of two ships in one location, for instance, might become interesting. Two ships that reported being in a location when images show that they were not, even more so. “It raises a lot of questions that we can also answer that nobody else has been able to,” she said.
Giving real-time information means not just creating a system that can ingest up to a million data points per minute but also one that can navigate the ever-evolving machine learning AI base. In the two years since Moore founded Lovelace, Nicolich-Henkin thinks “the conception of what our machine learning and AI would look like is not the same as what it is now.” This has been particularly true in incorporating large language models and generative AI techniques in both the product and the development process.
Nicolich-Henkin credits the abundance of CMU alumni on staff for Lovelace’s ability to stay up to date. “In this community, I’m working with a lot of really smart people who have a lot of experience. Seeing everything come together into a real product has been exciting.”
SCS’s nature of encouraging researchers to follow their interests and build from them has led to a lasting imprint on security across a wide variety of fields.
Back at Marinus Analytics, Jones noted, “We’re able to do some pretty big things because we are empowered by that foundation of long-standing research and development that came out of Carnegie Mellon.”
Jones once thought enough positive changes would come from their work that Marinus could put itself out of business. But, she said, “the work continues.”
The next step for Missing Persons Watch is working directly with child welfare agencies. Children who are in, or who have run away from, foster care are at an increased risk of experiencing sex trafficking. “This has been a long-standing problem, specific to the United States,” said Jones. “It’s a hard problem, and we’re doing it.”
Since there can be evidence of exploitation prior to a teenager’s last-seen date, they are looking for ways to serve youth before they become missing in the first place. Jones is grateful for the opportunity. “Even though this is a dark space, it’s a joy to deliver innovation to the public sector and to all these professionals who have dedicated themselves to serving our society.” ■
