검색 상세

Adapter-Only Bridging of Frozen Speech Encoder and Frozen LLM for ASR

자동음성인식을위한동결된음성인코더와동결된대규모언어 모델간어댑터만을이용한브릿징

  • 오준석 (Junseok Oh, 서강대학교 일반대학원)

초록(요약문)

Automatic speech recognition systems have achieved remarkable performance on general-domain benchmarks through end-to-end neural architectures trained on large-scale speech data. However, these systems exhibit significant performance degradation when deployed to specialized domains such as academic conferences, where domain-specific terminology and discourse patterns differ substantially from general speech. This dissertation argues that this limitation is not merely an acoustic modeling problem, but a knowledge- connection problem: the speech encoder often captures the acoustic evidence, while the decoder lacks sufficient access to specialized linguistic knowledge. Large language models, in contrast, possess extensive domain knowledge acquired from massive text corpora spanning diverse specialized fields, presenting an opportunity to enhance speech recognition through cross-modal integration. This dissertation proposes an adapter-only bridging approach that connects a frozen speech encoder with a frozen large language model for domain-adaptive automatic speech recognition. The approach explicitly freezes both components to preserve their pre-trained acoustic and linguistic knowledge while training only lightweight adapters for cross-modal alignment. The adapter architecture comprises a causal convolutional downsampler for temporal compression and local acoustic aggregation, followed by a multi-layer projection network for non-linear alignment to the LLM embedding space, totaling only 0.44% of the entire model parameters. Extensive experiments validate the proposed method using Whisper-large-v2 as the speech encoder and Gemma-3-4B-Instruct as the language model. Trained on 2,934 hours of general-domain speech, the approach achieves competitive performance on standard benchmarks while demonstrating substantial improvements on academic domains. Evaluation across four specialized areas—engineering, medical sciences, natural sciences, and social sciences—yields a 28.2% average relative word error rate reduction compared to the baseline encoder without any domain-specific training. Furthermore, inference-time domain prompting, enabled by the preserved language model knowledge, provides additional improvements in three of four domains and enhances domain-specific terminology recognition by 7.2 percentage points in F1 score while main- taining precision above 97%. This approach enables a single frozen model to adapt flexibly to specialized domains through natural language descriptions alone, eliminating the need for domain-specific training data or model retraining. Keywords Automatic Speech Recognition, Adapter-Based Speech-LLM Integration, Parameter- Efficient Learning, Inference-Time Domain Prompting, Cross-Modal Alignment

more

초록(요약문)

자동 음성 인식 시스템은 대규모 음성 데이터로 학습된 종단간 신경망 구조를 통해 일반 도메인 벤치마크에서 뛰어난 성능을 달성하였다. 그러나 이러한 시스템은 학술 대회와 같이 도메인 특화 전문 용어와 담화 패턴이 일반 음성과 크게 다른 전문 영역에 적용될 때 상당한 성능 저하를 보인다. 본 논문은 이러한 한계가 단순한 음향 모델링 문제가 아니라, 음성 인코더가 포착한 음향 단서와 전문 도메인 언어 지식 사이의 연결 문제라고 본다. 종단간 모델은 주로 일반 도메인 콘텐츠로 구성된 음성-텍스트 쌍 데이터로부터 제한된 언어 지식을 획득하는 반면, 대규모 언어 모델은 다양한 전문 분야를 아우르는 방대한 텍스트 말뭉치로부터 광범위한 도메인 지식을 보유하고 있어, 교차 모달 통합을 통한 음성 인식 향상의 가능성을 제시한다. 본 논문은 도메인 적응형 자동 음성 인식을 위해 동결된 음성 인코더와 동결된 대규모 언어 모델을 어댑터만으로 연결하는 방법을 제안한다. 제안하는 방법은 두 구성 요소의 사전 학습된 음향 지식과 언어 지식을 보존하기 위해 명시적으로 동결하고, 교차 모달 정렬을 위한 경량 어댑터만 학습한다. 어댑터 구조는 시간적 압축과 지역 음향 집계를 위한 인과적 합성곱 다운샘플러와, LLM 임베딩 공간으로의 비선형 정렬을 위한 다층 투영 네트워크로 구성되며, 전체 모델 매개변수의 0.44%만을 차지한다. 광범위한 실험을 통해 제안하는 방법을 검증하였으며, Whisper-large-v2를 음성 인코더로, Gemma-3-4B-Instruct를 언어 모델로 사용하였다. 2,934시간의 일반 도메인 음성으로 학습한 결과, 표준 벤치마크에서 경쟁력 있는 성능을 달성하는 동시에 학술 도메인에서 상당한 개선을 보였다. 공학, 의학, 자연과학, 사회과학의 네 가지 전문 영역에 대한 평가에서, 도메인별 학습 없이 기준 인코더 대비 평균 28.2%의 상대적 단어 오류율 감소를 달성하였다. 또한 보존된 언어 모델 지식을 활용한 추론 시 도메인 프롬프팅은 네 개 도메인 중 세 개에서 추가적인 개선을 제공하고, 도메인 특화 전문 용어 인식에서 F1 점수 7.2 퍼센트 포인트 향상을 달성하면서도 97% 이상의 정밀도를 유지하였다. 이 접근법은 단일 동결 모델이 자연어 설명만으로 전문 도메인에 유연하게 적응할 수 있게 하여, 도메인별 학습 데이터 수집이나 모델 재학습의 필요성을 제거한다.

more

목차

List of Tables vii
List of Figures ix
Notation x
Abstract xiii
Chapter 1. Introduction 1
1.1 Modern End-to-End Speech Recognition and Its Limitations 2
1.2 The Internal Language Model Problem 5
1.3 Large Language Models as Complementary Knowledge Sources 8
1.4 The Bridging Challenge: From Speech Encoder to LLM 9
1.4.1 The Modality Gap 9
1.4.2 Why Freeze Both Components 9
1.4.3 Trade-offs in Existing Speech-LLM Integration 10
1.5 Research Objectives 13
1.6 Research Approach 14
1.7 Contributions 15
1.8 Dissertation Organization 17
Chapter 2. Background and Related Work 19
2.1 Prior Approaches to Language Model Integration in ASR 19
2.1.1 Statistical Language Models 19
2.1.2 Neural Language Models 20
2.1.3 End-to-End ASR and Internal Language Models 22
2.1.4 Transformer-Based Speech Recognition and Whisper 23
2.2 Language Model Fusion Methods 24
2.2.1 Shallow Fusion 25
2.2.2 Deep Fusion 25
2.2.3 Cold Fusion 26
2.2.4 Internal LM Estimation （ILME） 26
2.2.5 Internal LM Training （ILMT） 27
2.2.6 ILM Estimation Variants 28
2.2.7 Hybrid Autoregressive Transducer （HAT） 28
2.3 Speech-LLM Integration 29
2.3.1 Speech Representation Types for LLM Integration 30
2.3.2 The Modality Gap in Speech-Text Integration 31
2.3.3 Transition from Language Model Fusion to Speech-LLM Alignment 32
2.3.4 Evolution of Speech-LLM Integration 34
2.3.5 Linear Projection Approaches 34
2.3.6 Joint Fine-tuning Approaches 35
2.3.7 Other Integration Methods 36
2.3.8 Recent Advances in Frozen LLM Strategies 37
2.4 Domain Adaptation for ASR 41
2.4.1 The Internal Language Model Problem in End-to-End ASR 41
2.4.2 Prior Adaptation Methods 43
2.4.3 Recent Adaptation Approaches 44
2.5 Multimodal Alignment 45
2.5.1 Vision-Language Alignment 45
2.5.2 Speech-Text Alignment 46
2.6 Parameter-Efficient Fine-Tuning 47
2.6.1 Low-Rank Adaptation 48
2.6.2 LLM Parameter Freezing Strategies 49
2.6.3 Adapter Modules 49
2.6.4 Knowledge Preservation 50
2.7 Research Gap Analysis 51
2.7.1 Component Selection Criteria 54
2.8 Summary 55
Chapter 3. Adapter-Only Bridging of Frozen Speech Encoder and Frozen
LLM 57
3.1 Dual-Frozen Architecture with Lightweight Adapters 57
3.1.1 Design Considerations 58
3.1.2 Component Integration 59
3.2 Causal Convolutional Adapter for Modality Bridging 60
3.2.1 Frozen Audio Encoder 60
3.2.2 Trainable Modality Adapters 61
3.2.3 Frozen Large Language Model 70
3.3 Training Methodology 72
3.3.1 Training Objective 73
3.3.2 General-Domain Training Strategy 78
3.4 Inference-Time Domain Prompting 81
3.4.1 Generic Prompting for General-Purpose Transcription 81
3.4.2 Domain-Specific Prompting for Specialized Domains 82
3.5 Comparison with Related Approaches 84
3.5.1 Comparison with Frozen LLM Approaches 84
3.5.2 Comparison with Integration Strategies 85
3.6 Summary 86
Chapter 4. Experimental Evaluation 89
4.1 Experimental Design 90
4.1.1 Experimental Protocol 90
4.1.2 Reproducibility and Deterministic Inference 91
4.1.3 Evaluation Metrics 91
4.2 Datasets 92
4.2.1 Training Data 92
4.2.2 Evaluation Data 92
4.3 Comparison Methods 94
4.3.1 Baseline Methods 94
4.3.2 Proposed Method Configurations 95
4.4 Results on General-Domain Benchmarks 96
4.4.1 Encoder-LLM Combination Analysis 96
4.4.2 Comparison with Baseline Methods 97
4.4.3 LibriSpeech Performance 98
4.4.4 Common Voice Performance 99
4.4.5 TEDLIUM Performance 99
4.4.6 Analysis of General-Domain Results 100
4.5 Domain Adaptation Results 101
4.5.1 Baseline Comparisons 101
4.5.2 Analysis of Minimal Projection Performance 102
4.5.3 Analysis of Joint Fine-tuning Performance 104
4.5.4 Domain Prompting Effectiveness 104
4.5.5 Analysis by Domain 106
4.6 Domain-Specific Terminology Recognition 107
4.6.1 Analysis of F1 Scores 107
4.6.2 Analysis of Precision-Recall Trade-off 108
4.6.3 Analysis of Cross-Domain Patterns 109
4.7 Ablation Study 110
4.8 Case Studies on Domain Adaptation 112
4.8.1 Analysis of Prompt Sensitivity 114
4.8.2 Analysis of Domain Prompting Effects 115
4.9 Parameter Efficiency 122
4.10 Computational Considerations 123
4.11 Analysis and Discussion 123
4.11.1 Analysis of Architectural Design Trade-offs 124
4.11.2 Analysis of Domain Prompting Effectiveness 126
4.11.3 Analysis of Generalization and Robustness 128
4.12 Summary 129
Chapter 5. Conclusion 132
5.1 Summary of Contributions 132
5.2 Principal Findings 133
5.3 Limitations 134
5.4 Future Research Directions 136
5.5 Implications 139
5.6 Concluding Remarks 140
References 142
초록 （국문） xv

more
반출 Meta View 목록