LORE: Lagrangian-Optimized Robust Embeddings for Visual Encoders
Improving adversarial robustness of visual encoders via constrained Lagrangian optimization, without sacrificing clean accuracy.
Authors: Borna Khodabandeh*, Amirabbas Afzali*, Amirhossein Afsharrad, Shahabeddin Mousavi, Sanjay Lall, Sajjad Amini, Seyed-Mohsen Moosavi-Dezfooli (*equal contribution)
Venue: NeurIPS 2025
Visual encoders such as CLIP and DINOv2 are widely used as general-purpose feature extractors, but they are vulnerable to adversarial perturbations — small, imperceptible changes to inputs that cause large shifts in the embedding space. Improving robustness while maintaining clean accuracy is a constrained optimization problem with a natural Lagrangian formulation.
LORE introduces a training framework that directly optimizes this trade-off. By formulating robustness as a constraint rather than a regularizer, the Lagrangian approach allows the optimizer to find solutions that sit on the Pareto frontier between clean and robust performance, rather than collapsing to one extreme.
We apply LORE to several visual encoder families and demonstrate state-of-the-art adversarial robustness across standard benchmarks, with minimal degradation in clean accuracy.
This was conducted as my Bachelor’s project, in collaboration with Professors Sajjad Amini and Seyed-Mohsen Moosavi-Dezfooli.
