Memristor-only LSTM Acceleration with Non-linear Activation Functions
Abstract
Long Short-Term Memories (LSTMs) applied to Speech Recognition are an essential application of modern embedded devices. Computing Matrix-Vector Multiplications (MVMs) with Resistive Random Access Memory (ReRAM) crossbars has paved the way for solving the memory bottleneck issues related to LSTM processing. However, mixed-signal and fully-analog accelerators still lack in developing energy-efficient and versatile devices for the calculus of activation functions between MVM operations. This paper proposes a design methodology and circuitry that achieves both energy efficiency and versatility by introducing a programmable memristor array for computing nonlinearities. We exploit the inherent capability of ReRAM crossbars in computing MVM to perform piecewise linear interpolation (PWL) of non-linear activation functions, achieving a programmable device with a smaller cost. Experiments show that our approach outperforms state-of-the-art LSTM accelerators being 4.85x more efficient using representative speech recognition datasets.