Unmanned aerial vehicle (UAV) aerial image interpretation plays an important role in the military and civilian files. The latest semantic segmentation methods are based on deep learning with different structure to encoder spatial feature. However, they are larger networks which are not effective for UAV with limited resources. Thus, a real-time adaptive spatial structure semantic segmentation network, ASRNet, is proposed for UAV aerial image. Firstly, ASRNet is based on an encoder-decoder structure with a module called local structure feature descriptor in the middle. Secondly, the descriptor utilizes features at different abstraction levels from both the encoder and decoder to describe different target with higher spatial resolution adaptively. Lastly, the local structure feature descriptor enables a better gradient flow from deeper layers to shallower layers by adding short paths for the back-propagation. The experiments validate the effectiveness of the proposed method from the accuracy and computation time.