There has been a considerable amount of research aimed at automating the documentation of accessibility in the built environment. Yet so far, there has been no fully automatic system that has been shown to reliably document surface quality barriers in the built environment in real-time. This is a mixed problem of HCI and applied machine learning, requiring the careful use of applied machine learning to address the real-world concern of practical documentation. To address this challenge, we offer a framework for designing applied machine learning approaches aimed at documenting the (in)accessibility of the built environment. This framework is designed to take into account the real-world picture, recognizing that the design of any accessibility documentation system has to take into account a range of factors that are not usually considered in machine learning research. We then apply this framework in a case study, illustrating an approach which can obtain a f-ratio of 0.952 in the best-case scenario.