In the event of a natural disaster, Japanese local governments investigate the level of damage of the buildings and issue damage certificates to the victims. The damage certificate is used to determine the contents of support provided to the victims; hence, they must be issued rapidly and accurately. However, in the past, the investigation of damage was time consuming, thus delaying the support provided to the victims. Additionally, while investigating the roof of the damaged building, it was difficult for the investigators to look at the entire roof and calculate the damage rate accurately. To address this issue, we have developed an image processing model to automatically calculate the rate of damage on a roof through image recognition from aerial photos. To circumvent the problem of lack of training data reported in our previous study [1], in this study, roof images were divided into roof surfaces based on image segmentation by deep learning, and the number of training data was increased. Our model calculated the rate of damage for up to 80% of roof data more accurately than the conventional assessment by a field investigator.