The recent advances in the field of embedded hardware and computer vision have made autonomous vehicles a tangible reality. The primary requirement of such an autonomous vehicle is an intelligent system that can process sensor inputs such as camera or lidar to have a perception of the surroundings. The vision algorithms are the core of a camera-based Advanced Driver Assistance Systems (ADAS). However, most of the available vision algorithms are x86 architecture based and hence, they cannot be directly ported to embedded platforms. Texas Instrument’s (TI) embedded platforms provide Block Accelerator Manager (BAM) framework for porting vision algorithms on embedded hardware. However, the BAM framework has notable drawbacks which result in higher stack usage, execution time and redundant code-base. We propose a novel lightweight framework for TI embedded platforms which addresses the current drawbacks of the BAM framework. We achieve an average reduction of 15.2% in execution time and 90% reduction in stack usage compared to the BAM framework.