دانشگاه اصفهان مدیریت دارایی و تامین مالی 2383-1189 14 4 2026 12 22 Comparison of Two Machine Learning–Based Algorithms for the Two Stages of a Pairs Trading Strategy مقایسه دو الگوریتم مبتنی بر یادگیری ماشینی برای دو بخش راهبرد معاملات زوجی 109 132 30255 10.22108/amf.2026.146240.2006 FA سمیرا اسدیان دانشجوی دکتری، گروه حسابداری، دانشکده اقتصاد مدیریت و حسابداری، دانشگاه یزد، یزد، ایران حجت الله صادقی دانشیار، گروه مدیریت صنعتی، دانشکده اقتصاد مدیریت و حسابداری، دانشگاه یزد، یزد، ایران داریوش دموری دانشیار، گروه حسابداری و مالی، دانشکده اقتصاد مدیریت و حسابداری، دانشگاه یزد، یزد، ایران Journal Article 2025 08 06 Market participants have shown increasing interest in quantitative trading models and the intelligent application of data science. These approaches can enhance the effectiveness of trading strategies such as pairs trading—a valuable investment method even during bear markets. This strategy comprises two key steps: selecting two securities (pair formation) and detecting anomalies in the price spread between them (trading execution). We evaluate two machine learning approaches to identify the most effective algorithm for each stage. To efficiently group related securities and uncover high-potential pairs—a task that even skilled investors may find challenging—we employ two density-based clustering methods: DBSCAN (Density-Based Spatial Clustering of Applications with Noise) and OPTICS (Ordering Points to Identify the Clustering Structure). These algorithms define the search space by identifying meaningful clusters of securities. To reduce portfolio drawdowns while maximizing returns after detecting price divergences, we implement two time series forecasting models: Long Short-Term Memory (LSTM) and LSTM encoder-decoder. OPTICS outperforms DBSCAN, demonstrating greater efficiency with fewer variables, a higher Sharpe ratio, and a larger proportion of profitable pairs. The LSTM encoder-decoder outperforms the standard LSTM model, delivering higher returns, an improved Sharpe ratio, and fewer days of portfolio decline. This study introduces a novel approach by leveraging machine learning models for both phases of the pairs trading strategy while optimizing model selection. Other distinctive features include the use of high-frequency intraday data (5‑minute intervals) across all market stocks (rather than a specific stock or industry), and a focus on net returns after accounting for transaction costs. Keywords Pairs Trading Strategy, Machine Learning, Density-Based Clustering, Forecasting-Based Model Introduction Pairs trading was first introduced to the financial literature by Gatev et al. (2006) and involves two steps: identifying two securities (a pair) and detecting an irregularity in the price gap between them (taking a trade). Given its proven robustness during market downturns and broad adoption (Do & Faff, 2010; Jacobs & Weber, 2015; Clegg & Krauss, 2018), the application of intelligent data science methods offers a promising path to enhance pairs trading performance. In the pair formation phase, we use two density-based clustering algorithms—DBSCAN and OPTICS—to determine the search space and identify clusters among a large number of securities, a task that is otherwise difficult to perform (Ester et al., 1996; Ankerst et al., 1999). In the trading phase, we employ two time-series prediction models—Long Short-Term Memory (LSTM) and an LSTM encoder-decoder network—to shorten portfolio drawdown periods and boost returns. The key contribution of this research lies in its systematic application of multiple machine learning algorithms across every phase of the strategy, aiming to identify the optimal models and enhance overall performance through predictive analytics. Accounting for transaction costs enables the reporting of realistic net profitability. Furthermore, the use of 5-minute intraday data significantly improves the accuracy of our findings. Materials & Methods The data consist of 111 stocks traded on the Tehran Stock Exchange and the IranFaraBourse over the period 2014 to 2019. To ensure liquidity conditions consistent with the criteria adopted by Gatev et al. (2006) and Do & Faff (2012), data imputation was performed using the tsrobprep open-source package in R software. To create meaningful clusters, we first performed dimensionality reduction using principal component analysis (PCA), followed by the application of DBSCAN and OPTICS clustering algorithms. Subsequently, four conditions were applied: (1) the components of each pair are co-integrated; (2) the Hurst component of the pair's price gap exhibits mean-reverting properties; (3) the price gap of each pair shows convergence and divergence over appropriate periods; and (4) the mean reversion of the pair's price gap occurs with sufficient frequency. The data were divided into a formation period and a trading period. The work includes three different test portfolios of securities: Portfolio 1 includes all pairs identified during the formation period; Portfolio 2 includes pairs with positive results; and Portfolio 3 is limited to investing in the 10 pairs with the highest returns in the validation set. To compare the robustness of the standard threshold-based model with the proposed prediction-based model, simulations were carried out using the LSTM and LSTM encoder-decoder models. All pairs in the portfolio were equally weighted. The code developed for this work was implemented in Python. Findings The first step in applying the proposed approach to stock clustering involves defining an appropriate number of principal components to reduce the dimensionality of the original data, which was set to five. OPTICS offers two distinct advantages over DBSCAN: (1) it effectively handles data clusters of varying densities, and (2) it eliminates the challenge of determining the optimal ε parameter. Analysis of the selected pairs shows that this method yields a promising portfolio with a satisfactory profit margin and a high average Sharpe ratio of 10.57. The results of the prediction-based trading models (LSTM and LSTM encoder-decoder) using the OPTICS algorithm for the period from April 2017 to March 2019 are presented in the table 1, alongside the results for the standard threshold-based model. In the standard trading model, the portfolio drawdown is significantly shorter; however, the drawdown in the LSTM encoder-decoder model is adequate and substantially shorter than that of the other model. The longer drawdown period for the standard LSTM is 190 days. The portfolio performs considerably better in terms of total return on investment (21,993.8%) and Sharpe ratio (35.50). The superior results from the LSTM encoder-decoder suggest that multi-stage prediction is effective in these settings. Table 1. Comparing trading results using a 3-year formation period LSTM encoder-decoder-based model LSTM-based model standard Trading model parameter validation 12 12 2 Drawdown numbers -0.51 2.18 6.7 SR 616.28 29817.4 7911.1 ROI(%) 170.24 33.97 8.43 MDD(%) 9829(9749-80) 9044(8983-62) 46(12-34) Trading numbers(+, -) 9.1 22.7 100 Profitable pairs (%) test 29 190 10 Drawdown numbers 50.35 2.18 0.53 SR 21993.8 -90.12 1918.9 ROI(%) 79.17 99.80 23.13 MDD(%) 150(143-7) 1196(1162-34) 25(12-13) Trading numbers(+, -) 100.0 0 86.4 Profitable pairs (%) Discussion & conclusion In comparing the results of this study with those of others, the work of Sarmento and Horta (2020) introduced advanced predictive modeling techniques—namely LSTM and its encoder-decoder variant—to the field of US ETF trading analysis. While these models successfully shortened portfolio drawdown periods, they also reduced overall returns. In their study, the traditional securities portfolio significantly outperformed the models in both total return and Sharpe ratio. Therefore, key features of the Iranian capital market's structure—such as its price fluctuation limits, base volume trading rules, and the surge in retail investor participation during 2019–2020—may be the primary reasons for the differing outcomes observed in the present study. Scientifically, however, this outcome is better explained by the limited adoption of machine learning models in the Iranian capital market. Even the use of classical algorithmic trading remains minimal, making the application of advanced models exceedingly rare. Flori and Regoli (2021) employed LSTM to analyze nonlinear interdependencies within and between financial time series for forecasting S&P 500 stock market performance. Their LSTM framework's predictive accuracy was benchmarked against several common machine learning models, including logistic regression, gradient boosted trees, and feedforward neural networks. The purpose of the LSTM application was to forecast the probability of a stock generating excess returns compared to similar stocks in the short term. Their analysis indicates that investment strategies incorporating such forecasts are effective for improving portfolio performance, as they generate predictive signals with exploitable information beyond basic price-yield gaps. باتوجه‌به علاقۀ مشارکت‌کنندگان بازار به مدل‌های معاملاتی کمّی و استفاده از کاربردهای هوشمندانۀ علم داده می‌توان عملکرد راهبرد‌های معاملاتی ازجمله معاملات زوجی را بهبود بخشید که ابزار سرمایه‌گذاری خوبی حتی در بازارهای نزولی است. با در نظر گرفتن دو مرحلۀ این راهبرد شامل شناسایی دو ورقۀ بهادار (یک زوج) و کشف بی‎‌نظمی در شکاف قیمتی بین آنها (انجام معامله)،‎‌ دو روش یادگیری ماشینی برای انتخاب الگوریتم بهتر در هر مرحله مقایسه شد. برای تعیین فضای جست‎‌وجو که سعی بر گروه‎‌بندی اوراق‌بهادار مرتبط در خوشه‎‌ها و شناسایی زوج‎‌های برتر دارد، از دو الگوریتم خوشه‌بندی مبتنی بر چگالی شامل خوشه‎‌بندی فاصله‎‌ای مبتنی بر چگالیِ کاربردها با نوفه (دی‌بی‌اسکن) و مرتب‌سازی نقاط برای شناسایی ساختار خوشه‌بندی (آپتیکز) استفاده شد که در غیر این‎‌صورت شناسایی آن حتی برای سرمایه‎‌گذار باتجربه هم دشوار است. از طرفی برای به‌ حداقل رساندن تعداد روزهای کاهش ارزش سبد به‌همراه افزایش بازده پس از یافتن شکاف قیمتی، از دو مدل معاملاتی مبتنی بر پیش‎‌بینی سری‎‌زمانی، حافظۀ کوتاه‎‌مدت طولانی (ال‌اس‌تی‌ام) و حافظۀ کوتاه‎‌مدت طولانی رمزگذار _ رمزگشا استفاده شد. برتری آپتیکز به دی‌بی‌اسکن در تعداد متغیرهای کمتر و نسبت شارپ بالا و درصد زوج‌های سودآور بالاتر مشهود است. مدل ال‌اس‌تی‌ام رمزگذار _ رمزگشا در کل در کسب بازدهی و نسبت شارپ، عملکرد بهتر و دربارۀ کاهش تعداد روزهای رکودی سبد بالاتر از مدل ال‌اس‌تی‌ام بود. استفاده از داده‌های با تکرار زیاد (5دقیقه‌ای) (تمامی سهام ‌بازار و نه سهم یا صنعت خاصی) به همراه استفاده از مدل‌های یادگیری ماشینی برای افزایش سرعت در تصمیم‌گیری و ارائۀ بازده‌های خالص به‌منظور بررسی مدل‌ها از موارد مهم این کار است.

https://amf.ui.ac.ir/article_30255_06b8e3fcd2384a38e218c31548bce28e.pdf