Aiming at handling the problem caused by the lack of prior spectral knowledge of anomalous pixels for hyperspectral anomaly detection (HAD). In this paper, we propose a background augmentation with transformer-based autoencoder for hyperspectral remote sensing image anomaly detection. The representative background pixels are selected based on sparse representation for obtaining typical background pixels as training samples of the transformer-based autoencoder. The selected typical background pixels can be used for training the transformer-based autoencoder to realize background pixel reconstruction. Thereafter, the pseudo background samples can be reconstructed from the transformer-based autoencoder, which is used to subtract the original image to obtain the residual image. Finally, Reed-Xiaoli (RX) is used to detect the anomalous pixels from residual image. Experiments results demonstrate that the proposed transformer-based autoencoder which can present competitive hyperspectral image anomaly detection results than other traditional algorithms.