UnZipLoRA: Separating Content and Style from a Single Image

Chang Liu, Viraj Shah, Aiyu Cui, Svetlana Lazebnik
UIUC
arXiv Code (coming soon!)
Teaser Image

UnZipLoRA decomposes any image into its content and style!

Abstract

This paper introduces UnZipLoRA, a method for decomposing an image into its constituent subject and style, represented as two distinct LoRAs (Low-Rank Adaptations). Unlike existing personalization techniques that focus on either subject or style in isolation, or require separate training sets for each, UnZipLoRA disentangles these elements from a single image by training both the LoRAs simultaneously. UnZipLoRA ensures that the resulting LoRAs are compatible, i.e., they can be seamlessly combined using direct addition. UnZipLoRA enables independent manipulation and recontextualization of subject and style, including generating variations of each, applying the extracted style to new subjects, and recombining them to reconstruct the original image or create novel variations. To address the challenge of subject and style entanglement during training, UnZipLoRA employs a novel prompt separation technique, as well as column and block separation strategies to accurately preserve the characteristics of subject and style, and ensure compatibility between the learned LoRAs. Evaluation with human studies and automatic metrics demonstrates UnZipLoRA's effectiveness compared to other recent state-of-the-art methods, including DreamBooth-LoRA, Inspiration Tree, and B-LoRA.

Approach

UnZipLoRA decouples content and style from single input image by learning two distinct LoRAs simultaneously. It relies on three key components to ensure accurate disentanglement: prompt separation, column separation, and block separation.


Results

Subject and Style Generation and Recontextualization

UnZipLoRA decomposes style and subject from input image, effectively disentabling and preserving concepts and achieve superior subject, style fidelity.

Subject-Style Recontextualization

Using UnZipLoRA can recontextualize subject and style in input images in flexible contexts.


BibTeX

@misc{liu2024unziploraseparatingcontentstyle,
      title={UnZipLoRA: Separating Content and Style from a Single Image}, 
      author={Chang Liu and Viraj Shah and Aiyu Cui and Svetlana Lazebnik},
      year={2024},
      eprint={2412.04465},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2412.04465}, 
}