Artificial Intelligence (AI) usage is rapidly expanding in our society. Private sector has already taken the leap of faith in using AI for efficiency and for generating better value for the customers and shareholders. The promise of AI is quite alluring for the governments as well. It promises to be the breakthrough technology which can catapult public sector to hitherto unseen efficiency and productivity. It has the potential to truly transform the public service delivery and the way government interfaces with citizens – from a demand driven model to a predictive model of public service delivery. However, there are a large number of pitfalls and blind-spots associated with AI, which make its adoption in government particularly challenging. For successful adoption of AI in public sector, governments must understand these challenges clearly and lay down regulatory public policies to ensure that the possible adverse impacts (such as exclusion, bias etc.) of AI are mitigated. This paper attempts to systematically explore these challenges with a view to enable public policy makers to respond to them.