Abstract

General Background: In the era of digital transformation, the success of public service applications depends significantly on user experience (UX). Specific Background: The M-Paspor application by the Directorate General of Immigration Indonesia was developed to streamline passport services but suffers from low public ratings and recurring user complaints. Knowledge Gap: Despite its potential, comprehensive UX metrics that consider both functional and emotional dimensions have not systematically evaluated the application. Aims: This study aims to assess the usability of the M-Paspor application using the HEART Framework, which evaluates five key dimensions: Happiness, Engagement, Adoption, Retention, and Task Success. Results: From 100 user responses, the application scored high in Engagement (0.75), Retention (0.71), Happiness (0.66), and Adoption (0.64), but only medium in Task Success (0.57). Three dimensions did not meet the 70% usability target, revealing critical shortcomings in interface design, onboarding guidance, and payment or scheduling features. Novelty: This research applies the HEART Framework as a stand-alone evaluative tool to systematically diagnose user experience issues in a government mobile application. Implications: The findings offer actionable recommendations to improve M-Paspor’s usability and provide a replicable model for evaluating other public digital services.

Highlights:

1. Engagement scored highest at 0.75 usability level.

2. Task Success dimension did not meet the target.

3. Design, guidance, and access features require improvement.

Keywords: HEART Framework, M-Paspor, Usability, User Experience, Public Service App

Introduction

In the digital era driven by technological advancements, the success and acceptance rate of an application is highly dependent on the quality of the user experience (UX) [1]. Based on ISO FDIS 9241-210, user experience is defined as individual perceptions and responses arising from the use or expectations of a product, system, or service [2]. UX covers a wide range of aspects, from the emotional experience, meaning, and value of human-computer interaction, to perceptions of functional aspects such as usability, ease of use, and system efficiency [3]. An application can be said to have good UX if it is able to provide satisfaction and comfort to its users, especially through an intuitive and easy-to-understand interface. Users who feel helped will tend to give positive reviews of the application [4]. Conversely, poor UX can not only decrease the efficiency of the service, but also hinder the wider adoption of the technology [5].

The M-Paspor application is a digital service developed by the Directorate General of Immigration of Indonesia with the aim of simplifying the passport application process online. This innovation is presented as a solution to the problem of long queues and bureaucratic complexity that is often found in conventional services [6]. This application was officially implemented since January 18, 2022 by the Directorate General of Immigration. However, since its launch, M-Paspor is still getting low ratings on official download platforms, at 2.2 out of 5 on Google PlayStore and 1.3 out of 5 on AppStore [7]. Negative assessments from the public conveyed through social media include various complaints, such as system glitches in the application, ID card numbers that are not detected, payment statuses that are not updated, verification codes that do not go to email, and immigration office locations that are not detected so that they cannot be selected [8]. Therefore, evaluation of user experience is an important step to measure and understand the level of satisfaction and effectiveness of the application in meeting the needs of its users [9].

The HEART Framework is a framework developed by Kerry Rodden from Google's research team to measure usability and user experience (UX). The framework uses a user-centric approach that places the user as the main focus in the evaluation of usability and UX [10]. HEART can be applied to both small and large scale UX evaluations [11]. There are five main dimensions measured in HEART, namely Happiness (satisfaction), Engagement (involvement), Adoption (new use), Retention (user activeness), and Task Success (task success). Evaluation is done through the stages of goals, signals, and metrics [12]. This method allows for more specific and thorough problem identification. In addition to measuring the effectiveness of the application, HEART also evaluates the emotional impact as well as the quality of interaction felt by users [13]. Thus, HEART is not only an evaluation tool, but also a strategic guide in improving the quality of user experience [14].

Several studies have shown that the HEART Framework is effective in evaluating user experience because it covers emotional, behavioral and performance aspects. [15] successfully implemented HEART to thoroughly evaluate the Information System for Rehabilitation of Victims of Drug Abuse. [16] also used HEART Metrics to measure the user experience of the SATUSEHAT application, showing the HEART dimensions were able to capture important aspects of user interaction. [17] proved that HEART can be effectively applied to digital entertainment services such as TIX.ID, without the need for a combination of other methods. [18] used the HEART Framework to evaluate the Seedling investment application and found that the majority of HEART dimensions showed satisfactory scores, although some aspects such as navigation and efficiency still need to be improved. [19] on BC HNI Cilacap 1 Web Commerce, with results that provide a comprehensive overview of the level of satisfaction and effectiveness of the site.

With the various problems previously described, it can be concluded that evaluating the user experience of the M-Passport application is a crucial step to improve the quality of digital services in the public sector. The use of the HEART Framework method as an evaluative approach offers advantages in identifying problems faced by users in a systematic and measurable manner. Thus, the results of this evaluation are expected to provide recommendations for improvements that are relevant and have a direct impact on increasing public satisfaction and trust in digital-based immigration services.

Method

This research has several stages as shown below:

Figure 1.Flowchart

A. Data Collection

Data collection in this study is to conduct observation techniques through direct observation of the M-Passport application and continued with the distribution of questionnaires to 100 respondents. The preparation of the list of question items is based on user-centered metrics from the HEART Framework with a Likert scale value range ranging from a value of 1 which is indicated by a statement of Strongly Disagree (STS) to a value of 5 which is indicated by a statement of Strongly Agree (SS). The list of questionnaire statements can be seen in Table 1.

B. HEART Framework Testing

1. Determination of Assessment Metric

The determination of this metric is based on the HEART Framework which consists of five variables to be assessed namely Happiness, measuring how satisfied and happy users are with their experience, this includes how visually appealing it is, whether they would recommend it to others and how easy it is to use the product. Engagement focuses on the frequency of user interaction with the product, measuring the user's level of involvement and connection to the product or service. Adoption relates to how quickly and easily new users accept and use the new product. Retention, measures how often users use the product within a certain period of time. Task Success, focuses on the user's ability to complete tasks efficiently, effectively, and with minimal errors.

2. Determination of Goals, Signals and Metrics

The first step in the HEART Framework analysis process is to set Goals, Signals (indicators of change) and Metrics (quantitative measures). Goals are the main objectives of each dimension to be achieved in the context of user experience. Signals are indicators that reflect changes or results from user interactions, while Metrics are units of measurement used to assess the success of each of these indicators. This determination is important so that the user experience evaluation has a clear direction and benchmark.

3. Maximum Score Calculation

Calculating the maximum value in each dimension. Calculation of N_max,, where the maximum value determined from the statement (NP_max) is multiplied by the number of statements on each criterion (∑P) and the number of respondents involved for a criterion (Res), as shown in the following equation:

N_max=〖NP〗_max×∑P×Res(1)

4. Total Value Calculation

Calculating the total value of each criterion. This calculation is done to display the results of the value obtained from the total value of the statements on the questionnaire.

5. Determination of Criteria Values

Calculate the criteria value (N_s) by dividing the value of each criterion obtained (N_total) by the maximum value (N_max) and multiplying by 100% as in the following equation:

N_s=N_total/N_max ×100% % (2)

Then calculate the average of each variable and continue by determining the level of usability based on Table 2.

Result and Discussion

A. Determintaion of Goals, Signals and Metrics

B. Validity and Reliability Test

The results of validity testing are shown in the following table:

Based on the results of the validity test calculation, the r-table value is obtained with n = 100 and a significance level of 0.05, so the r-table is 0.1654. The r-count value on each variable item is greater than the r-table, so the question items used can be declared valid. Thus, it can be concluded that each indicator has an r-count value greater than the r-table value, so that all indicators are declared valid.

The results of the reliability test are shown in the following table:

Based on the results of the reliability test calculation, all dimensions of the HEART Framework are declared reliable. This is in accordance with the criteria of the reliability test, namely a questionnaire is declared reliable if the Cronbach's Alpha value ≥ 0,7.

C. HEART Framework Testing

Based on the results of the HEART Framework testing, the usability level is obtained. The Happiness dimension, obtained a maximum value of 3,000 and a total value of 1,982, resulting in a criterion value of 0.66 or 66%, which indicates that the usability level is in the high category. The Engagement dimension, obtained a maximum value of 2,000 and a total value of 1,503, resulting in a criterion value of 0.75 or 75%, which indicates that the level of usability is in the high category. The Adoption dimension, obtained a maximum value of 2,000 and a total value of 1,285, resulting in a criterion value of 0.64 or 64%, which indicates that the level of usability is in the high category. The Retention dimension, obtained a maximum value of 2,000 and a total value of 1,422, resulting in a criterion value of 0.71 or 71%, which indicates that the usability level is in the high category. Task Success, obtained a maximum value of 3,000 and a total value of 1,701, resulting in a criterion value of 0.57 or 57%, which indicates that the usability level is in the medium category. These results are obtained based on the usability level in Table 2.

D. Evaluation of Goals Achievement

Based on the results of the evaluation of the achievement of goals, the dimensions of the HEART Framework that have not yet achieved goals are obtained, including

1. Happiness

The Happiness dimension, getting a criterion value of 66%, where this achievement has not met the predetermined target of 70%. There are two attributes that get a low percentage, so it is necessary to provide recommendations for improvement, namely H1 and H5.

2. Adoption

The Adoption dimension, getting a criterion value of 64%, where this achievement has not met the predetermined target of 70%. There is one attribute that gets a low percentage, so it needs to be given recommendations for improvement, namely A3.

3. Task Success

The Task Success dimension, getting a criterion value of 57%, where this achievement has not met the predetermined target of 70%. There are three attributes that get a low percentage, so recommendations for improvement need to be given, namely TS2, TS4 and TS6.

E. Recommendation for Improvement

Conclusion

This study evaluates the user experience of the M-Passport application using the HEART Framework method which consists of five dimensions: Happiness, Engagement, Adoption, Retention, and Task Success. The results showed that all dimensions were in the high usability category, except for the Task Success dimension which was in the medium category. The usability values obtained are: Happiness (0.66), Engagement (0.75), Adoption (0.64), Retention (0.71), and Task Success (0.57). However, there are three dimensions that have not reached the target value of 70%, namely Happiness, Adoption, and Task Success. This shows that although the level of usability is generally quite good, there are still shortcomings that need to be improved. Recommendations for improvement include improving interface design, adding onboarding guides, diversifying payment methods, providing real-time scheduling features, and adding FAQ features. This research shows that the HEART Framework is effective in identifying UX weaknesses in public digital services and can be a strong basis for future development of the M-Passport application in a more user-centered manner. This research is expected to be a reference for developers in improving the UX quality of the M-Passport application and a reference for future evaluations of public digital services.

References

1. R. Firdaus and D. Wahab, “Rancangan usulan aplikasi melalui pendekatan perancangan alat ukur kepuasan pengguna menggunakan metode EUCS, Green and Pearson, dan skala psikologi (kasus: otorisasi call center Cititrans),” J. Tata Kelola dan Kerangka Kerja Teknol. Inf., vol. 4, no. 1, pp. 48–53, 2018, doi: 10.34010/jtk3ti.v4i1.1399.
2. S. R. Henim and R. P. Sari, “Evaluasi user experience sistem informasi akademik mahasiswa pada perguruan tinggi menggunakan User Experience Questionnaire,” J. Komput. Terap., vol. 6, no. 1, pp. 69–78, 2020, doi: 10.35143/jkt.v6i1.3582.
3. P. S. E. Prima, “Evaluasi user experience pada aplikasi e-wallet dengan metode User Experience Questionnaire,” J. Teknol. dan Sist. Komput., vol. 10, no. 2, pp. 1–2, 2021.
4. N. Tri et al., “Analisis user experience pada layanan telekomunikasi operator seluler menggunakan metode System Usability Scale (SUS),” Digit. Transform. Technol., vol. 3, no. 1, pp. 49–57, 2023, [Online]. Available: https://doi.org/10.47709/digitech.v3i1.2391
5. T. Pustaka, “Analisis user experience terhadap kepuasan pengguna aplikasi Shopee menggunakan metode HEART Framework,” J. Teknol. dan Informasi, vol. 9, no. 1, pp. 1–8, 2025.
6. S. B. Helpiastuti, I. Syaifana, and H. Rohman, “Kualitas pelayanan M-Paspor di Kantor Imigrasi,” J. Ilm. Manaj. Publik dan Kebijak. Sos., vol. 7, no. 1, pp. 15–30, 2023.
7. V. Oktavianti, M. Hutahaean, and A. L. Tobing, “Efektivitas pelaksanaan program M-Paspor dalam pelayanan pengurusan paspor kepada masyarakat di Kantor Imigrasi Kelas I Khusus TPI Medan,” Gov. J. Ilm. Kaji. Polit. Lokal dan Pembang., vol. 9, no. 4, pp. 100–104, 2023.
8. M. Widiansyah, F. F. Az-zahra, and A. Pambudi, “Fine-tuning model IndoBERT untuk analisis sentimen berbasis aspek pada aplikasi M-Paspor,” Joutica J. Inform. Unisla, 2024.
9. A. Prameswari, S. Zakaria, and S. Centia, “Pelayanan publik berbasis electronic government melalui penerapan aplikasi mobile paspor (M-Paspor) di Kantor Imigrasi Kelas I TPI Bandung tahun 2022,” J. Adm. Pemerintah., vol. 3, no. 2, pp. 256–262, 2022.
10. S. Sidabutar and A. Ichwani, “Evaluasi user experience aplikasi mobile SATUSEHAT menggunakan HEART Metrics,” J. Pepadun, vol. 5, no. 1, pp. 72–82, 2024, doi: 10.23960/pepadun.v5i1.184.
11. B. W. Trenggono, A. Faroqi, and A. Wulansari, “Penerapan metode HEART Metrics dalam menganalisis user experience aplikasi e-learning,” Jutisi J. Ilm. Tek. Inform. dan Sist. Inf., vol. 11, no. 2, p. 471, 2022, doi: 10.35889/jutisi.v11i2.876.
12. A. L. Hanum, T. K. Miranti, D. Fatmawati, M. F. Diyon, and C. J. Prawiro, “Analisis user experience aplikasi mobile Peduli Lindungi menggunakan HEART Metrics,” Braz. Dent. J., vol. 33, no. 1, pp. 1–12, 2022.
13. T. Sistem, “Informasi,” J. Teknol. dan Sist. Informasi, vol. 5, no. 4, pp. 199–205, 2024, doi: 10.62527/jitsi.5.
14. N. N. Putri, A. Faroqi, T. Lathif, and M. Suryanto, “Analisis user experience aplikasi PLN Mobile berdasarkan alat ukur HEART Metrics,” Jutisi J. Ilm. Tek. Inform. dan Sist. Inf., vol. 11, no. 3, pp. 1–9, 2023.
15. O. V. T. Utami, C. Wiguna, and D. M. Kusumawardani, “Implementasi dan pengukuran pengalaman pengguna sistem informasi rehabilitasi korban penyalahgunaan napza menggunakan HEART Framework,” Sistemasi, vol. 10, no. 2, p. 460, 2021, doi: 10.32520/stmsi.v10i2.1304.
16. F. A. Alijoyo, S. Suhaerudin, and S. Meilia, “Measuring the user experience of the SATUSEHAT application with the HEART Metrics method approach,” SIBATIK J. J. Ilm. Bid. Sos. Ekon. Budaya, Teknol. dan Pendidik., vol. 3, no. 4, pp. 515–534, 2024, [Online]. Available: https://www.publish.ojs-indonesia.com/index.php/SIBATIK/article/view/1854
17. R. A. Kristi et al., “Analisis user experience aplikasi Tix.ID menggunakan HEART Framework,” Pros. Semnas Sistem Informasi, no. September, pp. 10–11, 2022.
18. D. S. Perdana and L. Y. Banowosari, “Evaluating the Bibit app: The HEART Framework approach in UX design,” J. RESTI (Rekayasa Sist. dan Teknol. Informasi), vol. 8, no. 2, pp. 250–257, 2024, doi: 10.29207/resti.v8i2.5714.
19. A. Faisal, M. Taufiqurrochman, A. Aziz, and N. Agustin, “Analisis user experience pada web commerce BC HNI Cilacap 1 dengan HEART Framework,” Seminar Nasional Teknologi Informasi dan Komputer (SENTIKA), pp. 872–880, 2024.