Painting is an important witness of the development of human civilization. In the communication and collision of Chinese and Western culture and art, because of the differences in political, geographical, historical and cultural backgrounds of the two countries, we find that there are great differences in the process of creating Chinese and Western art. As an expressive form of painting language, it truly and accurately reflects the painter’s personality and unique psychological activities. The innovation of this article is intended for distinguishing Chinese from Western paintings by leveraging the brushstroke characteristics of paintings carefully. In particular, we run edge detection method and Sobel operator to extract the characteristics of brushstroke; meanwhile, this research uses a 3 * 3 filter of image filtering to obtain image edge line. Considering the continuity of the painting brushstroke, we use morphological operation to remove noise and track to correction, connect and filter the edge of the line that are detected, aiming to extract the brushstroke features of painting. On this basis, combined with the deep learning model, we propose a new Chinese and Western painting classification framework, which helps to describe the style of painting works and improve the accuracy of Chinese and Western painting classification. Regarding Chinese and Western painting database constructed in the article, SVM shows its unique advantages compared with four commonly used classifier methods. In addition, this paper compares the classification based on brushstroke features to that without, the results show that the accuracy of classification based on brushstroke is nearly 10% better.