Text-to-LoRA (T2L) is a novel hypernetwork designed to adapt large language models (LLMs) on the fly, solely based on natural language task descriptions, thereby addressing the limitations of expensive and lengthy traditional fine-tuning processes. The core hypothesis underpinning T2L is that different LoRA adapters share an underlying adaptation mechanism, allowing for their simultaneous optimization and compression into a single model, which can then generate new task-specific LoRA adapters zero-shot at inference time.

The methodology of T2L centers on its role as a hypernetwork, which generates the parameters for LoRA adapters, serving as the "base network." For each target transformer module ($m$) and layer index ($l$), T2L predicts the low-rank matrices A and B (comprising the $\Delta W$ of LoRA) based on a vector representation of a natural language task description $z_i$. Formally, the output for a task $i$ is given by $\Delta W^i_{m,l} = h_{\theta}(\phi^i_{m,l})$, where $h_{\theta}$ represents the hypernetwork with parameters $\theta$. The input descriptor $\phi^i_{m,l}$ is constructed by concatenating the embedding of the task description $z_i$, the learnable embedding for the module type $m$, and the learnable embedding for the layer index $l$: $\phi^i_{m,l} = \text{concat}[f(z_i), E[m], E[l]]$. Here, $f(z_i)$ is a vector representation of the text description (e.g., CLS token activation from a bidirectional transformer or last token activation from an LLM), and $E[m]$ and $E[l]$ are learnable embedding dictionaries indexed by module type and layer index, respectively. This architecture allows T2L to generate all LoRA matrices within a single forward pass by batching the input embeddings.

T2L introduces three architectural variants, imposing different output space constraints and inductive biases:

1. T2L (L - Large): The largest variant. Its final linear layer directly outputs both low-rank matrices A and B simultaneously. The number of weight connections to the output head is $|\theta_{\text{head}}| = d_{\text{out}} \times 2 \times r \times d$, where $d_{\text{out}}$ is the output size of the last MLP block in the hypernetwork, $r$ is the LoRA rank, and $d$ is the input/output dimension of the adapted linear layer.
2. T2L (M - Medium): A medium-sized model with a shared output layer for either the A or B matrix. The output head size is $|\theta_{\text{head}}| = d_{\text{out}} \times r \times d$.
3. T2L (S - Small): The most parameter-efficient model, with the strongest inductive biases. It outputs only one rank of a low-rank matrix (A or B) at a time. This makes its head size much smaller: $|\theta_{\text{head}}| = d_{\text{emb}} \times d$, where $d_{\text{emb}}$ is the dimension of the embedding.

T2L can be trained via two distinct regimes:

4. LoRA Reconstruction: This method trains T2L to reconstruct pre-trained task-specific LoRAs. The objective is to minimize the difference between the generated LoRA weights and a target library of existing LoRAs $\Omega$. The loss function is $L(\Omega, \theta) = E_{\Delta W^i \sim \Omega} |\Delta W^i - h_{\theta}(\phi^i)|$. While effective for compressing known LoRAs, this approach may struggle with zero-shot generalization to unseen tasks, as numerically distinct but functionally similar LoRAs for related tasks might reside in different minima.
5. Supervised Fine-Tuning (SFT): This regime directly optimizes T2L on fine-tuning datasets, sidestepping the need for intermediate target LoRA adapters. The training objective for T2L is $\theta = \text{argmin}_{\theta} E_{D_i \sim D, z_i \sim Z_i} \mathcal{L}_{\text{SFT}}(D_i, \Psi, h_{\theta}(\phi^i))$, where $\mathcal{L}_{\text{SFT}}$ is the supervised fine-tuning loss, $D_i$ is a dataset for task $i$, and $\Psi$ are the base LLM weights. SFT-trained T2L implicitly learns to cluster tasks, which enhances its zero-shot LoRA generation capabilities and generalization to unseen tasks.

Experiments were primarily conducted using Mistral-7B-Instruct as the base LLM, with gte-large-en-v1.5 for extracting task embeddings. All LoRA adapters were set to rank 8, targeting query and value projection modules. The SNI dataset (a subset of 479 tasks) was used for training, and evaluation was performed on 10 widely used benchmarks including Arc-challenge/easy, BoolQ, GSM8K, Hellaswag, OpenBookQA, PIQA, Winogrande, HumanEval, and MBPP.

Results show that T2L trained via reconstruction can fully recover the performance of task-specific "oracle" LoRAs for seen benchmark tasks (Table 1), and can even outperform them in some cases, suggesting that the lossy compression acts as a regularization. However, its performance drops as the reconstruction error increases with more training tasks. For zero-shot LoRA generation, SFT-trained T2L consistently improves over a multi-task LoRA baseline across benchmarks (Table 2), demonstrating its ability to generate useful LoRAs for previously unseen tasks using only natural language descriptions. The performance of T2L generally benefits from increasing the number of training tasks and scaling the computational budget proportionally (Table 3), with the larger L and M variants showing better scalability than the S variant. T2L also exhibits robustness to different task embedding models (e.g., gte-large-en-v1.5 vs. Mistral-7B-Instruct, Table 4) and varying task descriptions, performing best when task descriptions are semantically aligned with the target task, even if unseen during training (Table 5). This robustness and generalization capability highlight T2L's potential for democratizing LLM adaptation with minimal compute requirements.