Image Style tuning w/ LoRa (Low-Rank Adaptation)

What this is

This project explores the power of using the LoRa (Low-Rank Adaptation) technique to fine-tune the outputs of an image model specifically Stable Diffusion XL and wrap it in a simple front-end experience. Although this project focuses on tuning an image model, this technique can be applied to any generative model (LLMs, etc.).

What this is not

This is not an exercise in the ability of modern coding agents or frontier models exploring their abilities with text-based prompts. No coding agents were used.

Why LoRa

The development of open-weight models has enabled people to begin tailoring generative models to specific use cases for individuals and organizations.

In the case of Stable Diffusion XL, it is beneficial to have access to tune 3.5 billion parameters. However, for many, this is an unrealistic number of paraeters to adjust due to hardware requirements, time, and the cost outweighing the benefits of the project. An extensive field of lightweight methods exists to customize models; this project focuses on LoRA.

With LoRA, we essentially freeze the weights of the large pre-trained model (Stable Diffusion XL) and introduce a smaller trainable matrix, which serves as the Low-Rank Adaptation. During training the weights of the LoRa adapters are the only ones being updated, while the larger model weights remain fixed and untouched.

Training

The Unstonio Pixel art data set was used to train the weights to achieve the pixel art styling. As mentioned before, this approach will work with any dataset of images to achieve results in a different style. 

In theory, I could have performed the training on my M3 Mac, however, to avoid unneeded headaches, a single NVIDIA RTX A6000 GPU was used for the training runs. Although not considered a consumer-level GPU, it is obtainable at a cost of ~$4.5k, an even more obtainable option is renting one for $0.35 an hour.

Inference / Generating A New image

Training results in a .safetensors file that contains the LoRA adapter weights learned during training. For the purpose of this demo, a local server was set up with an M3 Mac. The base XL Stable Diffusion model is downloaded, and the trained LoRA weights are attached. When a user enters a text prompt into the front-end experience, the model generates an image in the style of pixel art through the learned denoising process.

One might ask why go through this process if one can ask any image-generating model to “create an image in the style of pixel art.” If you only need to generate a single image, that would be a valid question, and all this work would be a waste of time. However, processes like this create a reusable component that can maintain visual consistency across many outputs.

Note: While the training dataset is crucial for refining the model's output to match the desired style, the base model performs the majority of the work during image generation. One must exercise due diligence in verifying the licensing and copyright clearance of the data used to train that base model for commercial use.