<bold id="bold-877bd9aeb9fa2a0b27377f369b5a72ab">Introduction </bold>

HEART Framework Evaluation of M-Paspor User Experience

Evaluasi UX Aplikasi M-Paspor dengan Metode HEART Framework

Novita Ramadayanti

novitaramadayanti3@gmail.com Sumiati

Sumiati

novitaramadayanti3@gmail.com

Indonesia Indonesia 27 05 2025

<bold id="bold-877bd9aeb9fa2a0b27377f369b5a72ab">Introduction </bold>

In the digital era driven by technological advancements, the success and acceptance rate of an application is highly dependent on the quality of the user experience (UX) [1]. Based on ISO FDIS 9241-210, user experience is defined as individual perceptions and responses arising from the use or expectations of a product, system, or service [2]. UX covers a wide range of aspects, from the emotional experience, meaning, and value of human-computer interaction, to perceptions of functional aspects such as usability, ease of use, and system efficiency [3]. An application can be said to have good UX if it is able to provide satisfaction and comfort to its users, especially through an intuitive and easy-to-understand interface. Users who feel helped will tend to give positive reviews of the application [4]. Conversely, poor UX can not only decrease the efficiency of the service, but also hinder the wider adoption of the technology [5].

The M-Paspor application is a digital service developed by the Directorate General of Immigration of Indonesia with the aim of simplifying the passport application process online. This innovation is presented as a solution to the problem of long queues and bureaucratic complexity that is often found in conventional services [6]. This application was officially implemented since January 18, 2022 by the Directorate General of Immigration. However, since its launch, M-Paspor is still getting low ratings on official download platforms, at 2.2 out of 5 on Google PlayStore and 1.3 out of 5 on AppStore [7]. Negative assessments from the public conveyed through social media include various complaints, such as system glitches in the application, ID card numbers that are not detected, payment statuses that are not updated, verification codes that do not go to email, and immigration office locations that are not detected so that they cannot be selected [8]. Therefore, evaluation of user experience is an important step to measure and understand the level of satisfaction and effectiveness of the application in meeting the needs of its users [9].

The HEART Framework is a framework developed by Kerry Rodden from Google's research team to measure usability and user experience (UX). The framework uses a user-centric approach that places the user as the main focus in the evaluation of usability and UX [10]. HEART can be applied to both small and large scale UX evaluations [11]. There are five main dimensions measured in HEART, namely Happiness (satisfaction), Engagement (involvement), Adoption (new use), Retention (user activeness), and Task Success (task success). Evaluation is done through the stages of goals, signals, and metrics [12]. This method allows for more specific and thorough problem identification. In addition to measuring the effectiveness of the application, HEART also evaluates the emotional impact as well as the quality of interaction felt by users [13]. Thus, HEART is not only an evaluation tool, but also a strategic guide in improving the quality of user experience [14].

Several studies have shown that the HEART Framework is effective in evaluating user experience because it covers emotional, behavioral and performance aspects. [15] successfully implemented HEART to thoroughly evaluate the Information System for Rehabilitation of Victims of Drug Abuse. [16] also used HEART Metrics to measure the user experience of the SATUSEHAT application, showing the HEART dimensions were able to capture important aspects of user interaction. [17] proved that HEART can be effectively applied to digital entertainment services such as TIX.ID, without the need for a combination of other methods. [18] used the HEART Framework to evaluate the Seedling investment application and found that the majority of HEART dimensions showed satisfactory scores, although some aspects such as navigation and efficiency still need to be improved. [19] on BC HNI Cilacap 1 Web Commerce, with results that provide a comprehensive overview of the level of satisfaction and effectiveness of the site.

With the various problems previously described, it can be concluded that evaluating the user experience of the M-Passport application is a crucial step to improve the quality of digital services in the public sector. The use of the HEART Framework method as an evaluative approach offers advantages in identifying problems faced by users in a systematic and measurable manner. Thus, the results of this evaluation are expected to provide recommendations for improvements that are relevant and have a direct impact on increasing public satisfaction and trust in digital-based immigration services.

<bold id="bold-8f4cbc72c667addf84e7c245cad57bf1">Method</bold>

This research has several stages as shown below:

Figure 1 Flowchart

<bold id="bold-faeccf60fb619ae208f392562c2f2bed">A.</bold> <bold id="bold-1">Data Collection</bold>

Data collection in this study is to conduct observation techniques through direct observation of the M-Passport application and continued with the distribution of questionnaires to 100 respondents. The preparation of the list of question items is based on user-centered metrics from the HEART Framework with a Likert scale value range ranging from a value of 1 which is indicated by a statement of Strongly Disagree (STS) to a value of 5 which is indicated by a statement of Strongly Agree (SS). The list of questionnaire statements can be seen in Table 1.

Table 1 Research Questionnaire

Dimensi Pernyataan

Happiness H1 I am interested in the appearance of the M-Paspor application

H2 I feel that the response speed of the M-Paspor application system is as expected

H3 The menu structure and navigation in the M-Paspor application are easy to understand

H4 The M-Paspor application helps me apply for a passport more efficiently

H5 I find it easy to get a passport queue schedule through the application

H6 I would recommend the M-Paspor application to others

Engagement E1 The M-Paspor application can be used every time

E2 I use the M-Paspor application every time I need passport processing services

E3 I need to use the M-Paspor application to see information about the status of applying for or renewing a passport periodically

E4 I intend to use the M-Paspor application in the long term

Adoption A1 I feel enthusiastic about trying the M-Paspor application for the first time

A2 I feel that the registration process in the M-Paspor application is easy and fast to do on first use

A3 The guide available in the M-Paspor application helps me understand the initial steps of use

A4 Features on the M-Paspor application work well when used for the first time

Retention R1 I continue to monitor notifications or alerts from the M-Paspor app after passport application or renewal

R2 I will save the M-Paspor application on my phone for future use

R3 The quality of the M-Paspor application is good

R4 I prefer to use the M-Paspor application for passport processing even though there are other alternatives

Task Success TS1 I can complete the entire passport application or extension process smoothly

TS2 I can make payments through the M-Paspor application

TS3 I can operate the M-Paspor application in a short time

TS4 I can change the schedule on the M-Paspor application

TS5 Rarely crashes occur in the M-Paspor application

TS6 I can access all the information and process of applying for a passport without the need for help from other parties

<bold id="bold-15cc1d3fcd2c733af5585834b64c7f21">B.</bold> <bold id="_bold-13">HEART Framework Testing</bold>

Dimensi	Pernyataan
Happiness	H1	I am interested in the appearance of the M-Paspor application
H2	I feel that the response speed of the M-Paspor application system is as expected
H3	The menu structure and navigation in the M-Paspor application are easy to understand
H4	The M-Paspor application helps me apply for a passport more efficiently
H5	I find it easy to get a passport queue schedule through the application
H6	I would recommend the M-Paspor application to others
Engagement	E1	The M-Paspor application can be used every time
E2	I use the M-Paspor application every time I need passport processing services
E3	I need to use the M-Paspor application to see information about the status of applying for or renewing a passport periodically
E4	I intend to use the M-Paspor application in the long term
Adoption	A1	I feel enthusiastic about trying the M-Paspor application for the first time
A2	I feel that the registration process in the M-Paspor application is easy and fast to do on first use
A3	The guide available in the M-Paspor application helps me understand the initial steps of use
A4	Features on the M-Paspor application work well when used for the first time
Retention	R1	I continue to monitor notifications or alerts from the M-Paspor app after passport application or renewal
R2	I will save the M-Paspor application on my phone for future use
R3	The quality of the M-Paspor application is good
R4	I prefer to use the M-Paspor application for passport processing even though there are other alternatives
Task Success	TS1	I can complete the entire passport application or extension process smoothly
TS2	I can make payments through the M-Paspor application
TS3	I can operate the M-Paspor application in a short time
TS4	I can change the schedule on the M-Paspor application
TS5	Rarely crashes occur in the M-Paspor application
TS6	I can access all the information and process of applying for a passport without the need for help from other parties

1. Determination of Assessment Metric

The determination of this metric is based on the HEART Framework which consists of five variables to be assessed namely Happiness, measuring how satisfied and happy users are with their experience, this includes how visually appealing it is, whether they would recommend it to others and how easy it is to use the product. Engagement focuses on the frequency of user interaction with the product, measuring the user's level of involvement and connection to the product or service. Adoption relates to how quickly and easily new users accept and use the new product. Retention, measures how often users use the product within a certain period of time. Task Success, focuses on the user's ability to complete tasks efficiently, effectively, and with minimal errors.

2. Determination of Goals, Signals and Metrics

The first step in the HEART Framework analysis process is to set Goals, Signals (indicators of change) and Metrics (quantitative measures). Goals are the main objectives of each dimension to be achieved in the context of user experience. Signals are indicators that reflect changes or results from user interactions, while Metrics are units of measurement used to assess the success of each of these indicators. This determination is important so that the user experience evaluation has a clear direction and benchmark.

3. Maximum Score Calculation

Calculating the maximum value in each dimension. Calculation of N_max_,, where the maximum value determined from the statement (NP_max) is multiplied by the number of statements on each criterion (∑P) and the number of respondents involved for a criterion (Res), as shown in the following equation:

(1)

4. Total Value Calculation

Calculating the total value of each criterion. This calculation is done to display the results of the value obtained from the total value of the statements on the questionnaire.

5. Determination of Criteria Values

Calculate the criteria value (N_s) by dividing the value of each criterion obtained (N_total) by the maximum value (N_max) and multiplying by 100% as in the following equation:

% (2)

Then calculate the average of each variable and continue by determining the level of usability based on Table 2.

Table 2 Usability Level

Correlation Coefficient Level

0,81 < r < 1,00 Very High

0,61 < r < 0,80 High

0,41 < r < 0,60 Medium

0,21 < r < 0,40 Low

0,00 < r < 0,20 Very Low

<bold id="bold-b2155a4fc148df01d77f851b8010a192">Result and Discussion</bold> <bold id="bold-8152199645351877be37c8718edc7241">A. </bold> <bold id="bold-2">Determintaion of Goals, Signals and Metrics<bold id="bold-3"/></bold> Table 3 Goals-Signals- Metrics

Correlation Coefficient	Level
0,81 < r < 1,00	Very High
0,61 < r < 0,80	High
0,41 < r < 0,60	Medium
0,21 < r < 0,40	Low
0,00 < r < 0,20	Very Low

Dimensions Goals Signals Metrics

Happiness 70% of users are satisfied with the appearance, navigation, speed and usefulness of the application Conducting questionnaires by providing statements that are oriented or represent aspects of Happiness The form of questionnaire answers will use a Likert scale of 1-5 and researchers will calculate the percentage of users who answer at points 4 and 5 (agree and strongly agree).

Engagement 70% of users actively use the app for various needs and for a long time Conducting the distribution of questionnaires by providing statements oriented or representing aspects of Engagement

Adaption 70% of new users find it easy, fast and clear the first time they use the app Conducting the distribution of questionnaires by providing statements oriented or representing aspects of Adaption

Retention 70% of users return to use the app after initial use Conducting the distribution of questionnaires by giving statements oriented or representing the Retention aspect

Task Success 70% of users successfully complete the service process through the application without problems Conducting the distribution of questionnaires by providing statements oriented or representing the Task Success aspect

<bold id="bold-8c5335e97a2b5c105e20d92bb3c79459">B. </bold> <bold id="bold-c4e8bda3538256b418c5ca88a8c792db">Validity and Reliability Test<bold id="bold-23d9ae777b2f001039cfbdbd54bf69eb"/></bold>

Dimensions	Goals	Signals	Metrics
Happiness	70% of users are satisfied with the appearance, navigation, speed and usefulness of the application	Conducting questionnaires by providing statements that are oriented or represent aspects of Happiness	The form of questionnaire answers will use a Likert scale of 1-5 and researchers will calculate the percentage of users who answer at points 4 and 5 (agree and strongly agree).
Engagement	70% of users actively use the app for various needs and for a long time	Conducting the distribution of questionnaires by providing statements oriented or representing aspects of Engagement
Adaption	70% of new users find it easy, fast and clear the first time they use the app	Conducting the distribution of questionnaires by providing statements oriented or representing aspects of Adaption
Retention	70% of users return to use the app after initial use	Conducting the distribution of questionnaires by giving statements oriented or representing the Retention aspect
Task Success	70% of users successfully complete the service process through the application without problems	Conducting the distribution of questionnaires by providing statements oriented or representing the Task Success aspect

The results of validity testing are shown in the following table:

Table 4 Validity Test

Indicator r hitung r tabel Description

H1 0,655 0,1654 Valid

H2 0,721 0,1654 Valid

H3 0,644 0,1654 Valid

H4 0,625 0, 1654 Valid

H5 0,613 0, 1654 Valid

H6 0,693 0, 1654 Valid

E1 0,816 0, 1654 Valid

E2 0,714 0, 1654 Valid

E3 0,727 0, 1654 Valid

E4 0,646 0, 1654 Valid

A1 0,641 0, 1654 Valid

A2 0,721 0, 1654 Valid

A3 0,694 0, 1654 Valid

A4 0,683 0, 1654 Valid

R1 0,434 0, 1654 Valid

R2 0,455 0, 1654 Valid

R3 0,898 0, 1654 Valid

R4 0,375 0, 1654 Valid

TS1 0,558 0, 1654 Valid

TS2 0,745 0, 1654 Valid

TS3 0,422 0, 1654 Valid

TS4 0,741 0, 1654 Valid

TS5 0,447 0, 1654 Valid

TS6 0,788 0, 1654 Valid

Indicator	r hitung	r tabel	Description
H1	0,655	0,1654	Valid
H2	0,721	0,1654	Valid
H3	0,644	0,1654	Valid
H4	0,625	0, 1654	Valid
H5	0,613	0, 1654	Valid
H6	0,693	0, 1654	Valid
E1	0,816	0, 1654	Valid
E2	0,714	0, 1654	Valid
E3	0,727	0, 1654	Valid
E4	0,646	0, 1654	Valid
A1	0,641	0, 1654	Valid
A2	0,721	0, 1654	Valid
A3	0,694	0, 1654	Valid
A4	0,683	0, 1654	Valid
R1	0,434	0, 1654	Valid
R2	0,455	0, 1654	Valid
R3	0,898	0, 1654	Valid
R4	0,375	0, 1654	Valid
TS1	0,558	0, 1654	Valid
TS2	0,745	0, 1654	Valid
TS3	0,422	0, 1654	Valid
TS4	0,741	0, 1654	Valid
TS5	0,447	0, 1654	Valid
TS6	0,788	0, 1654	Valid

Based on the results of the validity test calculation, the r-table value is obtained with n = 100 and a significance level of 0.05, so the r-table is 0.1654. The r-count value on each variable item is greater than the r-table, so the question items used can be declared valid. Thus, it can be concluded that each indicator has an r-count value greater than the r-table value, so that all indicators are declared valid.

The results of the reliability test are shown in the following table:

Table 5 Reliability Test

Dimensions Cronbach’s Alpha Description

Happiness 0,757 Reliabel

Engagement 0,790 Reliabel

Adaption 0,759 Reliabel

Retention 0,704 Reliabel

Task Success 0,756 Reliabel

Dimensions	Cronbach’s Alpha	Description
Happiness	0,757	Reliabel
Engagement	0,790	Reliabel
Adaption	0,759	Reliabel
Retention	0,704	Reliabel
Task Success	0,756	Reliabel

Based on the results of the reliability test calculation, all dimensions of the HEART Framework are declared reliable. This is in accordance with the criteria of the reliability test, namely a questionnaire is declared reliable if the Cronbach's Alpha value ≥ 0,7.

<bold id="bold-91a96f587c86bef42a16bad9a16bb173">C. </bold> <bold id="bold-29f259d64a140238278c43a57e39e597">HEART Framework Testing<bold id="bold-b0166a958967790e5ea296ea88952833"/></bold>

Table 6 HEART Framework Testing Results

Dimenssion Number of Questions N max N total N Kriteria Level of Usability

Happiness 6 3000 1982 0,66 High

Engagement 4 2000 1503 0,75 High

Adoption 4 2000 1285 0,64 High

Retention 4 2000 1422 0,71 High

Task Success 6 3000 1701 0,57 Medium

Dimenssion	Number of Questions	N max	N total	N Kriteria	Level of Usability
Happiness	6	3000	1982	0,66	High
Engagement	4	2000	1503	0,75	High
Adoption	4	2000	1285	0,64	High
Retention	4	2000	1422	0,71	High
Task Success	6	3000	1701	0,57	Medium

Based on the results of the HEART Framework testing, the usability level is obtained. The Happiness dimension, obtained a maximum value of 3,000 and a total value of 1,982, resulting in a criterion value of 0.66 or 66%, which indicates that the usability level is in the high category. The Engagement dimension, obtained a maximum value of 2,000 and a total value of 1,503, resulting in a criterion value of 0.75 or 75%, which indicates that the level of usability is in the high category. The Adoption dimension, obtained a maximum value of 2,000 and a total value of 1,285, resulting in a criterion value of 0.64 or 64%, which indicates that the level of usability is in the high category. The Retention dimension, obtained a maximum value of 2,000 and a total value of 1,422, resulting in a criterion value of 0.71 or 71%, which indicates that the usability level is in the high category. Task Success, obtained a maximum value of 3,000 and a total value of 1,701, resulting in a criterion value of 0.57 or 57%, which indicates that the usability level is in the medium category. These results are obtained based on the usability level in Table 2.

<bold id="bold-f6d16756723074b21fda4a3e648cd73b">D. </bold> <bold id="bold-dd05ea22e7f95190339e07bfac90639f">Evaluation of Goals Achievement<bold id="bold-4e9a65daa8549fa594421ebe0ab1ab5f"/></bold>

Table 7 Evaluation of Goals Achievement

Dimenssion Indicator NTotal (Answer 4 and 5) NKriteria Average Description

Happiness H1 172 34% 66% Goals not achieved

H2 389 78%

H3 364 73%

H4 376 75%

H5 299 60%

H6 382 76%

Engagement E1 383 77% 75% Goals achieved

E2 394 79%

E3 359 72%

E4 367 73%

Adaption A1 370 74% 64% Goals not achieved

A2 364 73%

A3 156 31%

A4 395 79%

Retention R1 398 80% 71% Goals achieved

R2 390 78%

R3 201 40%

R4 433 87%

Task Success TS1 398 80% 57% Goals not achieved

TS2 186 37%

TS3 397 79%

TS4 173 35%

TS5 377 75%

TS6 170 34%

Dimenssion	Indicator	NTotal (Answer 4 and 5)	NKriteria	Average	Description
Happiness	H1	172	34%	66%	Goals not achieved
H2	389	78%
H3	364	73%
H4	376	75%
H5	299	60%
H6	382	76%
Engagement	E1	383	77%	75%	Goals achieved
E2	394	79%
E3	359	72%
E4	367	73%
Adaption	A1	370	74%	64%	Goals not achieved
A2	364	73%
A3	156	31%
A4	395	79%
Retention	R1	398	80%	71%	Goals achieved
R2	390	78%
R3	201	40%
R4	433	87%
Task Success	TS1	398	80%	57%	Goals not achieved
TS2	186	37%
TS3	397	79%
TS4	173	35%
TS5	377	75%
TS6	170	34%

Based on the results of the evaluation of the achievement of goals, the dimensions of the HEART Framework that have not yet achieved goals are obtained, including

1. Happiness

The Happiness dimension, getting a criterion value of 66%, where this achievement has not met the predetermined target of 70%. There are two attributes that get a low percentage, so it is necessary to provide recommendations for improvement, namely H1 and H5.

2. Adoption

The Adoption dimension, getting a criterion value of 64%, where this achievement has not met the predetermined target of 70%. There is one attribute that gets a low percentage, so it needs to be given recommendations for improvement, namely A3.

3. Task Success

The Task Success dimension, getting a criterion value of 57%, where this achievement has not met the predetermined target of 70%. There are three attributes that get a low percentage, so recommendations for improvement need to be given, namely TS2, TS4 and TS6.

<bold id="bold-5975e73bc48d40d1c733a12f363d64d1">E. </bold> <bold id="bold-ca071532524963f50eadbee452dd7841">Recommendation for Improvement<bold id="bold-2b54ecfa6f727b80fc047b9b523f3c0c"/></bold>

Table 8 Recommendation for Improvement

Dimenssion Indicator Recommendation for Improvement

Happiness H1 I am interested in the appearance of the M-Paspor application Improve interface design through the selection of matching colors and more modern visual elements

H5 I find it easy to get a passport queue schedule through the application Real-time quota information and alternative schedules or locations should be provided

Adoption A3 The guides available in the M-Paspor application help me understand the initial steps of use Add onboarding features such as interactive tutorials or illustrated guides to make it easier for new users

Task Success TS2 I can make payments through the M-Paspor application Improve the payment system and add more varied methods such as digital wallets (GoPay, OVO), virtual account transfers or QRIS

TS4 Can change the schedule in the M-Paspor application Add a schedule change feature in the application, so that users can easily change the interview schedule without having to contact the immigration office

TS6 I can access all the information and process of applying for a passport without the need for help from other parties Add an FAQ feature to help answer common user questions independently

<bold id="bold-04f09e8ec9456f4985aef0f7337d8029">Conclusion</bold>

Dimenssion	Indicator	Recommendation for Improvement
Happiness	H1	I am interested in the appearance of the M-Paspor application	Improve interface design through the selection of matching colors and more modern visual elements
H5	I find it easy to get a passport queue schedule through the application	Real-time quota information and alternative schedules or locations should be provided
Adoption	A3	The guides available in the M-Paspor application help me understand the initial steps of use	Add onboarding features such as interactive tutorials or illustrated guides to make it easier for new users
Task Success	TS2	I can make payments through the M-Paspor application	Improve the payment system and add more varied methods such as digital wallets (GoPay, OVO), virtual account transfers or QRIS
TS4	Can change the schedule in the M-Paspor application	Add a schedule change feature in the application, so that users can easily change the interview schedule without having to contact the immigration office
TS6	I can access all the information and process of applying for a passport without the need for help from other parties	Add an FAQ feature to help answer common user questions independently

This study evaluates the user experience of the M-Passport application using the HEART Framework method which consists of five dimensions: Happiness, Engagement, Adoption, Retention, and Task Success. The results showed that all dimensions were in the high usability category, except for the Task Success dimension which was in the medium category. The usability values obtained are: Happiness (0.66), Engagement (0.75), Adoption (0.64), Retention (0.71), and Task Success (0.57). However, there are three dimensions that have not reached the target value of 70%, namely Happiness, Adoption, and Task Success. This shows that although the level of usability is generally quite good, there are still shortcomings that need to be improved. Recommendations for improvement include improving interface design, adding onboarding guides, diversifying payment methods, providing real-time scheduling features, and adding FAQ features. This research shows that the HEART Framework is effective in identifying UX weaknesses in public digital services and can be a strong basis for future development of the M-Passport application in a more user-centered manner. This research is expected to be a reference for developers in improving the UX quality of the M-Passport application and a reference for future evaluations of public digital services.