The present study aimed to develop a deep reinforcement learning-based model using neural networks to detect fraudulent and suspicious transactions in banking payment systems, with a focus on POS-based financial transactions. This applied case-study research was conducted using real transaction data from Bank Pasargad and the benchmark CCFD dataset. The research dataset included more than 250,000 real credit card transactions and 284,807 transactions from the benchmark dataset. A hybrid framework consisting of deep reinforcement learning, artificial neural networks, and autoencoder algorithms was employed for fraud detection. After preprocessing, normalization, and dimensionality reduction using the bottleneck method, the data were divided into training and testing subsets. Both supervised and unsupervised learning strategies were simultaneously implemented to improve anomaly detection and fraud classification performance. The proposed algorithms were implemented using R and Python machine learning libraries. The results obtained from the real transaction dataset demonstrated that the proposed model achieved excellent fraud detection performance with an AUC value of 0.999 and a Gini coefficient of 0.999. The final model correctly identified approximately 83% of fraudulent transactions and nearly 100% of legitimate transactions. Furthermore, evaluation on the CCFD benchmark dataset yielded an accuracy rate of 0.95 and a precision score of 0.97, indicating the model’s strong capability in handling highly imbalanced datasets and identifying anomalous financial behaviors. The findings indicated that integrating deep reinforcement learning, neural networks, and autoencoder techniques provides an effective approach for detecting fraud in financial and banking transactions. The proposed framework not only improved fraud detection rates but also reduced classification errors and effectively managed imbalanced transaction data. Therefore, the model can serve as a reliable infrastructure for intelligent banking transaction monitoring and real-time fraud detection systems.