Spectral Adaptive Dropout: Frequency-Based Regularization for Improved Generalization
Deep neural networks are often susceptible to overfitting, necessitating effective regularization techniques. This paper introduces Spectral Adaptive Dropout, a novel frequency-based regularization technique that dynamically adjusts dropout rates based on the spectral characteristics of network grad...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-06-01
|
| Series: | Information |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2078-2489/16/6/475 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1839653778470469632 |
|---|---|
| author | Zhigao Huang Musheng Chen Shiyan Zheng |
| author_facet | Zhigao Huang Musheng Chen Shiyan Zheng |
| author_sort | Zhigao Huang |
| collection | DOAJ |
| description | Deep neural networks are often susceptible to overfitting, necessitating effective regularization techniques. This paper introduces Spectral Adaptive Dropout, a novel frequency-based regularization technique that dynamically adjusts dropout rates based on the spectral characteristics of network gradients. The proposed approach addresses the limitations of traditional dropout methods by adaptively targeting high-frequency components that typically contribute to overfitting while preserving essential low-frequency information. Through extensive experimentation on character-level language modeling tasks, the study demonstrates that the method achieves a 1.10% improvement in validation loss while maintaining competitive inference speeds. Thise research explores several implementations including FFT-based analysis, wavelet decomposition, and per-attention-head adaptation, culminating in an optimized approach that balances computational efficiency with regularization effectiveness. Our results highlight the significant potential of incorporating frequency-domain information into regularization strategies for deep neural networks. |
| format | Article |
| id | doaj-art-a5f5c61d2ea14b5d9c383bb6e37f916d |
| institution | Matheson Library |
| issn | 2078-2489 |
| language | English |
| publishDate | 2025-06-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Information |
| spelling | doaj-art-a5f5c61d2ea14b5d9c383bb6e37f916d2025-06-25T13:57:39ZengMDPI AGInformation2078-24892025-06-0116647510.3390/info16060475Spectral Adaptive Dropout: Frequency-Based Regularization for Improved GeneralizationZhigao Huang0Musheng Chen1Shiyan Zheng2Department of Physics and Information Engineering, Quanzhou Normal University, Quanzhou 362000, ChinaDepartment of Physics and Information Engineering, Quanzhou Normal University, Quanzhou 362000, ChinaDepartment of Physics and Information Engineering, Quanzhou Normal University, Quanzhou 362000, ChinaDeep neural networks are often susceptible to overfitting, necessitating effective regularization techniques. This paper introduces Spectral Adaptive Dropout, a novel frequency-based regularization technique that dynamically adjusts dropout rates based on the spectral characteristics of network gradients. The proposed approach addresses the limitations of traditional dropout methods by adaptively targeting high-frequency components that typically contribute to overfitting while preserving essential low-frequency information. Through extensive experimentation on character-level language modeling tasks, the study demonstrates that the method achieves a 1.10% improvement in validation loss while maintaining competitive inference speeds. Thise research explores several implementations including FFT-based analysis, wavelet decomposition, and per-attention-head adaptation, culminating in an optimized approach that balances computational efficiency with regularization effectiveness. Our results highlight the significant potential of incorporating frequency-domain information into regularization strategies for deep neural networks.https://www.mdpi.com/2078-2489/16/6/475spectral adaptive dropoutfrequency-based regularizationneural networksoverfittinggradient analysisdeep learning |
| spellingShingle | Zhigao Huang Musheng Chen Shiyan Zheng Spectral Adaptive Dropout: Frequency-Based Regularization for Improved Generalization Information spectral adaptive dropout frequency-based regularization neural networks overfitting gradient analysis deep learning |
| title | Spectral Adaptive Dropout: Frequency-Based Regularization for Improved Generalization |
| title_full | Spectral Adaptive Dropout: Frequency-Based Regularization for Improved Generalization |
| title_fullStr | Spectral Adaptive Dropout: Frequency-Based Regularization for Improved Generalization |
| title_full_unstemmed | Spectral Adaptive Dropout: Frequency-Based Regularization for Improved Generalization |
| title_short | Spectral Adaptive Dropout: Frequency-Based Regularization for Improved Generalization |
| title_sort | spectral adaptive dropout frequency based regularization for improved generalization |
| topic | spectral adaptive dropout frequency-based regularization neural networks overfitting gradient analysis deep learning |
| url | https://www.mdpi.com/2078-2489/16/6/475 |
| work_keys_str_mv | AT zhigaohuang spectraladaptivedropoutfrequencybasedregularizationforimprovedgeneralization AT mushengchen spectraladaptivedropoutfrequencybasedregularizationforimprovedgeneralization AT shiyanzheng spectraladaptivedropoutfrequencybasedregularizationforimprovedgeneralization |