Comparison of two machine learning-based algorithms for two parts of pairs trading strategy

Articles in Press

Document Type : Research Paper

Authors

1 Faculty of Economics, Management and Accounting, Yazd University

2 Associate professor Faculty of Economics, Management and Accounting, Yazd University

3 Associate Professor Yazd University, Faculty of Economics, Management and Accounting, Yazd University, Yazd, Iran.

Abstract

Objectives: Market participants are increasingly interested in quantitative trading models and the smart application of data science. These approaches can enhance the effectiveness of trading strategies, such as pairs trading—a valuable investment method even during bear market.



Method: This strategy involves two key steps: selecting two securities (a pair), and detecting an anomaly in price gap between them (trading). We evaluate two machine learning approaches to determine the most effective algorithm for each stage. To efficiently group related securities and uncover high-potential pairs—tasks that even skilled investors may find challenging—we employed two density-based clustering methods: DBSCAN (Density-Based Spatial Clustering of Applications with Noise) and OPTICS (Ordering Points To Identify the Clustering Structure). These algorithms help define the search space by identifying meaningful clusters of securities. To reduce the portfolio drawdowns while maximizing returns after detecting price divergences, we implemented two time series forecasting models: Long Short-Term Memory (LSTM) and LSTM Encoder-Decoder.



Results: OPTICS outperforms DBSCAN, demonstrating greater efficiency with fewer variables, a higher Sharpe ratio, and an increased proportion of profitable pairs. The LSTM encoder-decoder outperformed the LSTM model, delivering higher returns, an improved Sharpe ratio, and fewer days of portfolio decline.



Key Innovations: This study introduces a novel approach by leveraging machine learning models for both phases of the strategy while optimizing model selection. Additional distinctive features include the use of high-frequency intraday data (5-minute intervals) (all market stocks, not a specific stock or industry), and the focus on net returns after accounting for transaction costs.

Keywords

Main Subjects

Journal of Asset Management and Financing

Articles in Press, Accepted Manuscript
Available Online from 28 January 2026

Files

Share

How to cite

Statistics