Street-Legal Physical Adversarial Rim for License Plates

Eight months ago, I saw this YouTube video by Benn Jordan highlighting the rapid growth of license plate readers across the country, and arguing that private companies like Flock Safety were using this technology to track average people without their consent. He proposed a transparent covering for your license plate with adversarially-placed “spots” (resembling dirt) that could disrupt the computer vision algorithm responsible for locating and reading the license plate in the camera’s images. The implementation of this is open-source, though it just randomly applies shapes over a single image until a license plate reader fails, and claims that this is a reliable method for physical-world attacks. This seemed far too simple to be true; surely a simple random search couldn’t be sufficient to reliably confound a realistic and modern ALPR system?

I tested out his code on my own license plate, and indeed my intuition was correct. Though it was possible to find a pattern that causes the license plate reader model to fail on a single image, when I printed out this pattern and applied it over my own license plate, I was completely unable to replicate his results from the video. I experimented a bit with the approach to get it to transfer into the real world, but quickly realized that even if that were possible, covering any part of my license plate could be grounds to get pulled over and ticketed. Maybe that’s a risk real attackers could take, but for someone just wanting to preserve their privacy? There had to be a better, legal way.

So I invited one of my good friends, Sahana Ganapathy, over to rewatch the video and think through what that better approach might look like. Before long, we were scouring the prior literature to see what others had tried, and eventually discovered a 2024 paper that managed to disrupt a license plate detector with just an ornamental border (obscuring nothing on the real license plate). Their results were promising, causing CNN-based license plate detectors to fail in over 90% of cases, when the border was applied digitally.

Zhang et al.’s adversarial border insertion method, applying the border as a rescaled rectangle over the license plate.

But here, too, we saw issues. The authors inserted the adversarial border rectangularly on the image plane, rather than transforming it into the same orientation as the license plate. The straight-on viewing angle was a pretty big assumption that would not hold up under a real attack scenario, where the license plate camera could be at a high incident angle. Moreover, the authors claimed “physical-world validation,” but essentially just tested their printed border on a single image of a single car at an ideal viewing angle. It was a cool visual, to be sure, but not sufficient evidence that the approach could work in real attack conditions.

Zhang et al.’s only provided data for their physical-world evaluation. Yes, just these three images.

Of course, anybody can sit on the sidelines and retrospectively criticize the decisions of expert researchers. Actually building a better system was a much taller order, especially since we were just two undergraduates without any formal lab support. But the ML problem was too interesting, and the real-world implications too significant, for Sahana and me to let this go. We resolved that, one way or another, we would build a practical, realistic attack that held up under a comprehensive physical-world evaluation and could actually disrupt real Flock Safety cameras.

The first step toward realizing that vision was to build a v0 prototype. In his video, Benn Jordan attacked a single open-source license plate reader: fast-alpr. We decided this was a perfectly reasonable place for us to start as well. The initial attempt was incredibly simple: keep the rectangular patch insertion assumption from Zhang et al., collect a small dataset of images of my car and its license plate, and directly tune the pixels of the adversarial rim to disrupt the CNN-based license plate detector model. I trained it on my laptop over a few hours, then printed it out and tested. Despite decent success on the digital test set, these initial patches failed in the real world, even in ideal-case conditions.

Generally when something like this happens, you should go back to the literature, examine what others did that was successful, and try to first replicate another study. But that’s far too much work… why bother when you can just ask Claude how to fix it? It actually didn’t do too bad, and after some prodding to get around safety refusals, it came up with two pretty neat ideas. First, it recommended that we label the vertices of the license plate in all our training and test data, and from that compute a homography transformation to properly apply the border in the same perspective as the plate. That should help break the exploitation strategy based on pixel-perfect alignment of high-frequency noise, as opposed to genuine adversarial patterns. Second, it recommended the total variation loss as a secondary objective, which should hopefully further reduce the success of the high-frequency noise strategy.

We also decided to change our approach somewhat. Instead of just attacking the license plate detector, we also included the OCR model as a target. This way, there were multiple viable strategies to break the overall system, either by causing detection to fail or by causing the OCR to misread. It also enabled a second attack class: could we actually impersonate another license plate by biasing the induced misreads toward a specific attacker-defined plate number?

So we trained both the earlier disruption and new impersonation versions, printed them out, tested, and what do you know? They actually worked in the real world, even at different viewing angles!

It only took us a few months to get to that positive result. We spent the next few improving the system, cleaning up the codebase, and figuring out how to actually write, submit, and publish our result. The key novelty wasn’t in the success rate (other approaches had demonstrably higher metrics), but in the particular constraints we imposed on ourselves (street-legality, resource limitations, LLM-driven implementation) and the unique format of our attack (targeting both the detector and OCR models with the same border, instead of just one or the other like all prior literature). This phase was definitely where the majority of my learning happened. I had to get in touch with several of my current and prior professors to ask for advice and feedback. We fluctuated between submitting to a top-tier conference and mid-tier workshops, mainly because we didn’t have a good baseline for what kind of work could get accepted to each type of venue.

My biggest mistake was probably deciding not to pursue publication at VehicleSec 2026, a workshop under USENIX which really would’ve been a perfect fit for what we’d done. The deadline was in February, and if we’d committed to that, we absolutely could’ve gotten a paper done covering just this preliminary v0 result. But our broader goal was to disrupt a real Flock camera, not just a single open-source toy system. So I chose to not spend the time submitting the v0 and instead focus on building out a new approach for generalizing to black-box systems, without access to their model weights.

Another month of development later, and though I had made some real progress toward that goal, we were still very far from publication-ready on the new idea. By this point in March, most deadlines for conferences and workshops in 2026 had already passed, and I really wanted to show some deliverable result from this project before I graduated university. So I took a step back from the technical work and regrouped with my collaborator. We identified Security in Machine Learning Applications 2026 as a potential alternative venue for our v0 paper.

This time, I chose to set aside the cutting-edge work and take the time to flesh out our earlier idea. We conducted extensive physical-world evaluations of our single-target white-box attack at different distances, altitudes, and viewing angles. We applied a statistical method (distance correlation analysis) to quantify our approach’s dependence on viewing geometry, and established empirically that homography transformations and total variation loss really do improve attack performance at higher incident angles. We found the right citations to ground our contribution in what came before. And we cut, cut, cut, to remove everything excess or unnecessary and squeeze within the page limit.

The night we submitted, we had both travelled to Tulsa, OK, to participate in the Collegeiate Cyber Defense Competition, a high-stress event spanning two days in which we’d try to secure real infrastructure while actively under attack by a professional red team. It was a crazy time to be making final changes, but we had no choice; we’d be much too tired if we tried to make the submission any later. Well past midnight, we finally slipped below the 20 page limit and hit submit. You can see the full paper here:

To be honest, I expected this to be rejected, even from a workshop. We were just two undergraduates with hardly any guidance. I had no idea how to write a paper or what reviewers at this particular conference would be looking for. And the paper we submitted had very real limitations, not least of which was that it only attacked a single system with full access to its software and model weights. But there was enough of a chance that we had to try. And the way I saw it, even if we were rejected, it would give us a critical signal on what was and wasn’t important which could be invaluable to our future paper documenting true black-box transfer.

The one-month wait to receive our response was excruciating, but we finally got our decision: ACCEPTED! The reviewers had been pretty fair, and the criticisms they raised were ones that I had also shared at the time of submission. But they had indeed been willing to overlook some of the shortcomings of our work and instead focus on the bigger-picture contribution we would make to the workshop. Sahana and I were nothing short of ecstatic when we saw this response. We’d never imagined that our quirky side-project could eventually turn into a full publication, and certainly didn’t see ourselves being capable of conducting the entire research process entirely on our own.

I suppose the only way to learn is to try, fail, and repeat until you finally reach success.

Comments

All comments have a 300 character minimum to promote meaningful contributions and civil discussion. I look forward to reading your thoughts, feedback, and criticism!