#4708 Splitting LLMs into General Language and Specialize Model
#4708 Splitting LLMs into General Language and Specialize Model #4708 Describe how LLMs could be trained to produce to sub-models: one for general language interpretation and response, and one for specialty knowledge. The goal is to reduce the size of the specialized knowledge model. Decoupling general linguistic fluency from specialized domain knowledge involves splitting the traditional monolithic transformer into a shared General Interpreter (responsible for grammar, reasoning, dialog structure, and intent parsing) and a lightweight Specialized Knowledge Sub-Model (housing factual associations, domain-specific terminology, and relational logic). Core Architectural Approaches 1. Modular Parameter-Efficient Adapters (Rank-Constrained Knowledge Injection) Mechanics: The General Interpreter remains a frozen base model ( $B$ ). The specialty sub-model consists strictly of low-rank update matrices ( $\Delta W = A \times B$ ) or sparse Feed-Forward Network (FFN) layers inserted int...