Security is a major concern in Electronic Payment (E-Payment) systems. Usually, these systems are protected against illegal users, so-called hackers, by different means, such as Personal identification numbers (PINs), passwords, cards, etc. However, these hackers may manage to bypass this protection by having recourse to different strategies. Many techniques have been proposed to counter hacking attempts; however, there are still situations where an illegal user may succeed to access the E-payment system easily by stealing from a legal user its payment card. The use of Artificial Intelligence methods for face authentication, like deep learning, has made facial biometry a highly developing and accurate technology, especially in the past decade. In this paper, we propose the joint use of deep learning-based facial biometry and RFID cards to reinforce the security of an E-Payment system. By doing so, we ensure that a user should be physically present carrying his RFID card to be able to access the E-Payment system. We have tested three deep learning-based face authentication models and validated them on MUCT and CASIA Face-V5 datasets, to choose the most suitable one for our proposed secured E-Payment system, obtaining top verification rates of 99.90% and 99.26%, respectively. Two versions of this system are proposed; in the first version, which is based on a Personnel Computer (PC) and a Raspberry card, face authentication is implemented in a PC and the control of the RFID reader is performed by a Raspberry Pi 3, whereas in the second version, which may be considered as an embedded system, all the job is accomplished by the Raspberry Pi.