Introduction

Investment in capital markets has become a crucial driver of modern economic growth, offering significant profit potential while simultaneously supporting capital formation and financial system stability. In recent years, there has been a notable increase in retail and novice investors, particularly among younger generations who increasingly recognize the importance of financial literacy for achieving long-term economic security. This growing participation reflects broader digital access to financial markets and heightened awareness of investment as a strategic tool for wealth accumulation. However, the expansion of investor participation also amplifies exposure to financial risk, especially in equity markets where potential returns are inherently proportional to potential losses.

Stock market investments are widely acknowledged as high-risk financial instruments due to the stochastic, non-linear, and highly volatile nature of stock price movements. Price dynamics are influenced by a complex interaction of internal factors, such as corporate fundamentals and managerial performance, as well as external factors including regulatory changes, macroeconomic conditions, geopolitical events, and investor sentiment. These intertwined influences result in chaotic and multifractal behaviors within financial time series, making stock price forecasting a computationally challenging yet essential task for informed investment decision-making [1], [2].

The inherent complexity and non-linearity of financial markets significantly limit the effectiveness of traditional forecasting approaches. Conventional techniques such as fundamental and technical analysis, along with classical statistical models, have long been employed to analyze market movements. Models like Autoregressive Integrated Moving Average (ARIMA) and Generalized Autoregressive Conditional Heteroskedasticity (GARCH) rely on assumptions of stationarity and normality that are frequently violated in real-world financial data. Empirical evidence shows that stock price series often exhibit non-stationarity, volatility clustering, fat-tailed distributions, and asymmetric behaviors, causing traditional models to underperform, particularly during periods of market turbulence [3], [4], [5].

To mitigate these limitations, hybrid statistical models such as ARIMA–GARCH have been proposed to jointly model linear trends and conditional variance. While these hybrid approaches offer improved volatility estimation compared to standalone models, they remain constrained by sensitivity to parameter specification, limited adaptability to rapidly changing market conditions, and difficulties in capturing complex non-linear dependencies. Consequently, their predictive accuracy and robustness remain insufficient for highly dynamic and chaotic financial environments, highlighting the need for more flexible and expressive modeling paradigms [6], [7], [8].

In response to these challenges, deep learning has emerged as a dominant framework for financial time-series forecasting. Architectures such as Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, and Convolutional Neural Networks (CNNs) have demonstrated superior capability in modeling non-linear relationships and extracting hierarchical features from historical data. LSTM networks, in particular, are well-suited for sequential data due to their ability to capture long-term dependencies. Recent studies further demonstrate that hybrid and multi-architecture deep learning models outperform both traditional statistical methods and single-model deep learning approaches in highly volatile financial contexts [9], [10], [11].

Despite these advancements, accurate modeling of long-term dependencies and temporal dynamics remains a persistent challenge. Financial time series often exhibit long-range dependencies across multiple time horizons, requiring models that can simultaneously capture global contextual relationships and localized temporal patterns. Transformer architectures, with their self-attention mechanisms, have shown remarkable success in learning long-term dependencies by modeling relationships across entire sequences. However, Transformers alone may struggle with fine-grained temporal dynamics in noisy financial data, motivating the development of hybrid architectures that integrate attention-based models with recurrent networks such as LSTM to leverage complementary strengths [12], [13], [14].

Beyond predictive accuracy, a critical limitation of existing stock price prediction studies lies in their predominant focus on error minimization while neglecting explicit risk quantification. For investors, understanding potential downside risk is as important as forecasting expected returns. The absence of integrated risk analysis can result in overly optimistic or speculative investment decisions. Addressing this research gap, this study proposes a Hybrid Transformer–LSTM model integrated with Monte Carlo simulation to estimate Value at Risk (VaR) probabilistically. The model is evaluated using historical stock data of PT Bank Mandiri (Persero) Tbk (BMRI), a blue-chip stock characterized by high market capitalization and liquidity. This integrated framework aims to provide a more comprehensive investment decision-support system by jointly delivering accurate price predictions and measurable risk assessments.

Figure 1 shows the Hybrid Transformer-LSTM model research methodology for predicting stock Log Returns, which is then validated with VaR Monte Carlo risk analysis.

Figure 1. Research Method

In this approach, LSTM is used to extract temporal trends [15], which are then refined by the Multi-Head Attention mechanism in Transformer to capture data anomalies [16]. These features are combined through Global Average Pooling and Dense layers to produce an accurate Log Return prediction model. And the final stage is performance evaluation using the MAE and MAPE indicator metrics, which is followed by Value at Risk (VaR) risk estimation through Monte Carlo Simulation to map the maximum potential loss realistically [17].

1. Data Collection

The data used in this research is daily historical data on shares of PT Bank Mandiri Persero Tbk (BMRI) obtained from the web investing.com in the time period 01/03/2013 to 12/03/2025. The historical data consists of 3131 rows and consists of the variables "Date, Price, Open, High, Low, Vol."

Table 1. Bank Mandiri Historical Dataset

2. Data Preprocessing and Feature Engineering

Data Stage Data preprocessing is a fundamental step to transform raw data into a numerical format that can be studied optimally by Deep Learning architecture [16]. The quality of data representation at this stage plays a crucial role in accelerating model convergence and preventing bias in prediction results. The initial steps start with Feature Selection and Data Cleaning. This research utilizes the main attributes of daily trading activity, namely Open, High, Low, Close, Volume as multivariate input variables [15]. Data integration is ensured by checking for missing values to ensure the validity of the model input. Next, a normalization process was carried out using the Min-Max Scaler technique. This step is very important because the Gradient Descent based algorithm is very sensitive to data scales that have large variance, so equalization of the scale is needed so that the training process runs stably [18]. Normalization transforms all feature values ​​into an interval range of [0,1] using the following equation:

(1)

Where x is the original value and x' is the normalized value.

After the data is normalized, the next stage is the formation of a sequential data structure using the Sliding Window method. This process changes the data dimensions from a 2-dimensional table format (rows, columns) to a 3-dimensional tensor format (Samples, Time Steps, Features) so that it can be processed by the LSTM layer [19]. This study sets the window size at 60 days. Finally, the dataset is divided into a train set and a test set with a ratio of 80:20 to ensure that model evaluation is carried out objectively on data that has never been seen before [20].

3. Data Pattern Analysis

Prior to the hybrid modeling process, historical stock data is analyzed to understand its volatility characteristics. This identification, based on technical analysis theory, serves as the basis for the LSTM layer to capture long-term trends and for the Transformer to assign attention weights to the most relevant technical signals to generate accurate predictions [21].

4. Model Architecture

Figure 2. Architecture Hybrid LSTM-Transformer

The modeling stage is carried out by building a Hybrid Deep Learning architecture that integrates the Long Short-Term Memory (LSTM) mechanism sequentially with the Transformer block, as in Figure 2. This Hybrid approach is designed to acquire two characteristics of long-term temporal dependencies from time series data [15] while the transformer is tasked with identifying global relationships between features through an attention mechanism that is capable of capturing long-range correlations [16]. The model architecture is composed with dimensional tensor input (Samples, Time Steps, Features) the main computational process occurs in two core components as follows: information.

a. Long Short-Term Memory (LSTM)

This layer acts as an initial encoder that processes the data sequence using a gating mechanism to regulate the flow of relevant information [18]. Each LSTM cell has three main gates Input Gate (i_t), Forget Gate (f_t), and Output Gate (o_t). This mechanism allows the model to discard irrelevant information (such as market noise) and retain important trends to overcome the vanishing gradient problem. The update process of the hidden state (h_t) and cell state (c_t) at time step t is calculated based on the following equation:

(2)

(3)

(4)

(5)

Where σ is the sigmoid activation function, W is the weight matrix, b is the bias, and ⨀ denotes the element-wise product. The output of this layer is set with return_sequences=True to pass the temporal features to the transformer block [19].

b. Transformer (Multi-Head Attention)

The temporal features of the LSTM are then processed using the Multi-Head Self-Attention mechanism. This process allows the model to assign different weights (attention scores) to each time step within the input window, allowing the model to focus on the most relevant market signals. The attention score is calculated using the Scaled Dot-Product Attention function, defined in the following equation:

(6)

Where Q (Query), K (Key), and V (Velue) are the projection matrices of the input, and is a scaling factor to maintain gradient stability. The output of this block is then compacted through Global Average [22]. Pooling before entering the final training and evaluation stag

5. Performance Evaluation

After the training phase is completed, performance measurements are conducted to validate the performance of the Hybrid model in predicting new data. Evaluation is carried out by comparing the predicted value () with the actual value () in the testing set. To comprehensively measure the level of accuracy and error magnitude, this study uses two evaluation metrics, Mean Absolute Error (MAE) and Mean Absolute Percentage Error (MAPE).

a. Mean Absolute Error (MAE)

MAE is used to measure the average absolute difference between predicted and actual values. This metric describes how much the model's predictions differ. The smaller the MAE value, the more accurate the model's performance in stock prediction. MAE is defined by the equation

(7)

Where n is the number of data, the predicted result value with actual value [23].

b. Mean Absolute Percentage Error (MAPE)

In addition to nominal errors, evaluation is also carried out using Mean Absolute Percentage Error (MAPE) to measure relative accuracy in percentage form. MAPE is very effective in knowing how much the prediction deviates from the actual value, thus facilitating the interpretation of model performance regardless of the stock price scale [23]. The mathematical formulation for calculating MAPE is shown in the following equation:

(8)

The integration of these two metrics ensures the validity of the model both on a nominal and percentage scale.

c. Value at Risk (VaR) Monte Carlo Simulation- Based

The evaluation is then complemented by investment risk estimation using the Value at Risk (VaR) method based on Monte Carlo simulation. This simulation was run for 20,000 iterations to project the probability of future extreme losses based on the trend (drift) characteristics studied by the Hybrid model. The risk estimation formula is defined as follows:

(9)

Where P₀ is the initial stock price, S_t is the distribution of simulated stock prices at the end of the period (T), a is the confidence level value [24]. This evaluation provides a statistically valid loss tolerance limit for investors.

Results and Discussion

A. Results

1. Data Analysis and Pre-processing

The initial discussion focused on analyzing the characteristics of the historical stock data used and the results of the feature engineering. The historical data used in this study is the stock price data of PT Bank Mandiri (Persero) Tbk (BMRI), obtained from investing.com. The data covers the daily trading period from March 1, 2013 to March 12, 2025, with a total of 3,131 rows of observations.

The main variables used include Date, Price, Open, High, Low, and Vol. Given the highly volatile nature of stock price movements and the influence of temporal trends, this study integrates additional features into the feature engineering process. Technical indicators such as the Relative Strength Index (RSI) and Moving Average are used to enrich the information learned by the model.

Figure 3. Visualization of BMRI stock price and RSI indicator

Based on Figure 3, it can be identified that the stock price data (top graph) is continuous and a time series with a dynamic trend. The RSI indicator (bottom graph) shows momentum alignment with price fluctuations, with areas above 70 indicating overbought conditions and areas below 30 indicating oversold conditions. This ensures that the technical features used are relevant to the prediction targets. To better understand the data structure, a seasonal decomposition was performed to separate the trend, seasonal, and residual (noise) components, as shown in Figure 4.

Figure 4. Decomposition of Trend, Seasonal, and Residual Components

Figure 4 shows that the BMRI stock price data contains a fluctuating long-term trend component and a periodically recurring seasonal component. The high fluctuation in the residual component indicates significant noise, making the use of a nonlinear model such as the Hybrid Transformer-LSTM more appropriate than a conventional linear modell. Next, a correlation analysis was performed between variables to examine the relationship between input features and the prediction targets, as shown in Figure 5.

Figure 5. Correlation Matrix Between Features

Figure 5 shows that trend-based features such as the SMA and EMA have a very strong positive correlation (>0.98) with the closing price, indicating multicollinearity between price variables. Meanwhile, the RSI feature has a low correlation (0.09) with price. This value does not indicate the feature is useless, but rather indicates that the RSI carries independent information not represented by price data, thus enriching the model.

Next, the data is transformed using a scaling technique using a Min-Max Scaler to adjust the values ​​of each feature to a range of 0 to 1. The goal is to reduce the impact of scale differences between features (for example, between a price in the thousands of rupiah and an RSI value in the tens of thousands). The final process at this stage is data splitting with an 80:20 ratio, with 80% as training data and 20% as test data.

Table 2. Data Splitting Output

2. Modeling dan Evaluasi

The modeling phase is performed using the Hybrid Transformer-LSTM architecture.

Table 3. Hybrid Transformer-LSTM Model Architecture

Based on Table 3, the model is designed with a complex sequential structure. The input data (shape: 60, 8) is first processed by an LSTM layer (32 units) to capture long-term dependencies in the time series data. The LSTM output is then passed to the Multi-Head Attention layer to focus weights on the most relevant features. The uniqueness of this architecture lies in the ADD layer, which functions as a residual connection (connecting the LSTM output directly with the Attention output). This mechanism aims to maintain the flow of information so that it doesn't get lost as the network deepens. Next, the data is normalized using Layer Normalization before finally entering the Dense layer to produce a single price prediction output. After the training phase is complete, model performance is measured using the MAE and MAPE evaluation metrics. The evaluation results on the test data are presented in:

Table 4. MAE and MAPE Evaluation Results

Based on Table 4, the model produced a MAPE value of 4.13%. Referring to Lewis's (1982) criteria, a value below 10% is categorized as Highly Accurate Forecasting. This indicates the model's predictive ability is very accurate. Furthermore, the MAE value yielded 246.35. This value represents the average absolute deviation in rupiah, indicating that the model's predictions were only 246.35 Rupiah off the actual price. Considering that the average BMRI share price during the study period was around 5,500 Rupiah, this nominal error is considered very small and tolerable.

Figure 6. Performance Evaluation of Hybrid Transformer-LSTM

The validation results are confirmed through the visualization of the results presented in Figure 6. The graph shows that the prediction line is able to follow the actual price fluctuation pattern of BMRI shares very closely, which confirms that the model successfully captures the movement trend without experiencing significant overfitting.

3. Value at Risk (VaR) Monte Carlo

After the accuracy evaluation was completed, the research continued with an investment risk analysis using the Value at Risk (VaR) method based on Monte Carlo simulation. The simulation was run for 20,000 iterations to project the distribution of possible future prices.

Figure 7. Value at Risk ( VaR ) Monte Carlo

The visualization results of the Monte Carlo simulation are presented in Figure 7. The left panel visualizes thousands of stock price movement scenario trajectories (transparent blue lines) for the next 20 days, while the right panel displays a histogram of the loss frequency distribution. To provide a more detailed picture of the resulting risk values, a quantitative summary of the simulation results is presented in table 5.

Table 5. Value at Risk (VaR)

Referring to Table 5 and the distribution visualization above, the Value at Risk (VaR) calculation at a 99% confidence level yields a risk estimate of 5.35%, visually indicated by the red dotted line. This figure indicates that within the projection period, the probability of investors experiencing a loss greater than 5.35% is only 1%. This information provides valid quantitative risk limits for investors to use as a basis for loss mitigation in investment decision-making.

B. Discussion

The results of the data analysis and pre-processing stage confirm that BMRI stock prices exhibit strong non-linear, volatile, and time-dependent characteristics, which are typical of financial time series. The visualization of price movements and RSI indicators demonstrates that while price data capture long-term trends, momentum-based indicators such as RSI provide complementary information related to market sentiment and overbought or oversold conditions. This finding aligns with previous studies that emphasize the importance of incorporating technical indicators to enrich feature representation and improve predictive performance in stock forecasting models [10], [25]. Furthermore, the seasonal decomposition analysis reveals substantial noise in the residual component, reinforcing the argument that linear models are insufficient for capturing the complex dynamics of stock prices, as also noted by Zhang and Wen [2].

The correlation analysis further provides important insights into feature relationships. Trend-based indicators such as SMA and EMA show extremely high correlations with the closing price, indicating strong multicollinearity. While this confirms their relevance, it also suggests redundancy if used without careful modeling strategies. In contrast, RSI exhibits a weak linear correlation with price, yet this does not imply irrelevance. Instead, it indicates that RSI contributes orthogonal, non-linear information not captured by raw price features. This observation is consistent with findings by Chen et al, who argue that low-correlation features can still significantly enhance deep learning models by introducing diverse representations, particularly when attention-based mechanisms are employed [26].

The performance of the Hybrid Transformer–LSTM model demonstrates that integrating sequential learning and global attention mechanisms is highly effective for stock price prediction. The LSTM layer successfully captures temporal dependencies, while the Multi-Head Attention layer selectively emphasizes informative time steps and features. The inclusion of a residual connection (ADD layer) plays a crucial role in preserving information flow and mitigating gradient degradation, which is a known challenge in deep architectures. Similar architectural benefits have been reported in prior hybrid models combining recurrent networks and attention mechanisms, where residual learning improves both convergence stability and predictive accuracy [13], [27]. This confirms that the proposed architecture is well-suited to modeling complex financial time series.

Quantitatively, the model achieves a MAPE of 4.13%, which falls into the “highly accurate forecasting” category based on Lewis’s benchmark [28]. This level of accuracy is competitive with, and in some cases superior to, results reported in previous studies using standalone LSTM, CNN-LSTM, or ARIMA-based hybrid models, which typically report MAPE values ranging between 5% and 10% for stock price prediction [9], [29]. The low MAE value of 246.35 Rupiah further indicates that the model’s prediction errors are economically small relative to the average BMRI stock price. The close alignment between predicted and actual price trajectories observed in the validation visualization suggests that the model generalizes well and avoids overfitting, addressing a common limitation highlighted in earlier deep learning studies [30].

Beyond predictive accuracy, the integration of Monte Carlo–based Value at Risk (VaR) analysis represents a key contribution of this study. While many prior works focus solely on minimizing prediction error, they often neglect explicit risk quantification, limiting their practical relevance for investors. The VaR result of 5.35% at a 99% confidence level indicates a relatively low probability of extreme losses, providing a clear and interpretable risk boundary. This finding is consistent with studies that emphasize the effectiveness of Monte Carlo simulations for capturing uncertainty and tail risk in financial forecasting [31], [32]. By combining accurate price prediction with probabilistic risk estimation, this study advances existing literature and offers a more comprehensive decision-support framework for investment analysis

Conclusion

This study demonstrates that the Hybrid Transformer–LSTM model is highly effective for predicting stock prices in a complex and volatile financial environment. By integrating sequential learning through LSTM with global dependency modeling via Multi-Head Attention and residual connections, the proposed architecture successfully captures both temporal patterns and contextual relationships in BMRI stock price data. The empirical results show strong predictive performance, as evidenced by a Mean Absolute Percentage Error (MAPE) of 4.13% and a low Mean Absolute Error (MAE) of 246.35 Rupiah, indicating that the model produces highly accurate and economically meaningful forecasts. These findings confirm that hybrid deep learning architectures outperform conventional and single-model approaches when applied to non-linear financial time series.

Beyond accuracy, this research highlights the importance of incorporating risk quantification into stock price prediction frameworks. The integration of Monte Carlo simulation for Value at Risk (VaR) estimation provides a probabilistic assessment of potential losses, offering practical insights for investment decision-making. The VaR result of 5.35% at the 99% confidence level indicates a relatively low probability of extreme downside risk within the forecast horizon. By combining precise price prediction with measurable risk evaluation, this study contributes to the development of more robust and investor-oriented forecasting systems, and provides a foundation for future research to extend the framework to multi-asset portfolios, longer forecasting horizons, or additional sources of market information.