The use of consumer wearable health device (CWHD) for fitness tracing has seen an upward trend worldwide. CWHDs support individuals in taking ownership of their personal well-being and keeping track of their fitness goals. However, there are genuine concerns over the accuracy of the data collected by these devices. In this study, we investigated the factors that influence the accuracy of the data collected by CWHDs for heart rate measurement, physical activity (PA), and sleep monitoring using a systematic literature review. Forty-seven papers were analyzed from five electronic databases based on specific inclusion and exclusion criteria. All 47 papers that we analyzed were published by authors from developed countries. Using thematic analysis, we classified the factors that influence the accuracy of the data collected by CWHDs into three main groups, namely (i) the tracker and sensor type, (ii) the algorithm used in the device, and (iii) the limitation in the design, energy consumption, and processing capability of the device. The research results point to a dearth of studies that focus on the accuracy of the data collected by CWHDs by researchers from developing countries.