Visually impaired persons often experience limited user accessibility regarding smart devices. Particularly, it is challenging for such persons to precisely touch a button or icon to operate smart devices. Existing studies have reported a lack of step-by-step guidance for visually impaired users to perform tasks on smart devices. Although prototypes and non-visual interface systems were implemented to aid blind people in using mobile phones, a step-by-step guidance system was not developed. Moreover, devices other than smartphones were not emphasized. To address these limitations, we propose a novel guidance system that employs an object detection method for recognizing the control panels of smart devices and offers appropriate guidelines based on the user’s targeted task. A user study conducted with 20 participants indicated that users were more satisfied with our guidance system than with existing voice assistants.