Efficient Domain Adaptation

Collaboration: Defence Science and Technology Group (DSTG)

Abstract

Few-shot learning is critical for data-scarce domains like remote sensing, but foundation models like CLIP struggle when novel concepts are poorly separated in pretrained latent spaces. To optimize performance with minimal parameters, this study evaluates parameter-efficient fine-tuning (PEFT) methods. Results show Low-rank Adaptation (LoRA) is highly effective, particularly for indistinguishable classes. Additionally, we propose a lightweight steering-vector-based mechanism for targeted representation modulation using significantly fewer parameters. Integrated with existing techniques, this approach consistently boosts downstream performance, demonstrating the value of structured, representation-aware adaptation in data-scarce environments like remote sensing.

In this study, we investigated different methods in zero-shot and few-shot settings:

---
- Model soup
- Model ensemble
- Cache model/ Tip-Adapter
- Clip-Adapter
- APE
- Task Residual
- LoRA
- Activate Steering Vector
---

Visualization of different images from remote sensing dataset x-View1.

Embedding visualization of dataset x-View1.

Class similarity visualization across 60 novel categories in the xView-1 dataset. Feature similarities are computed using text embeddings from both the pre-trained RemoteCLIP model and its LoRA fine-tuned counterpart. The comparison demonstrates that while numerous novel classes overlap significantly in the original pretrained latent space, LoRA fine-tuning successfully adapts the text encoder to expand the margins between these classes.

Method

We propose Adaptive Steering Vector (AdaSV), a learnable framework that embeds Trainable Steering Vectors (TSVs) across layers to generate gradual representation shifts. This approach aims to enhance class distinction using substantially fewer parameters while matching the performance of current few-shot adaptation methods.

Left figure shows how to construct a steering vector and add to a transformer layer. Right figure shows the results on CIFAR-10 and x-View 1.