ABSTRACT

This article aims to explore the quality of the information provided by public authorities to citizens, businesses, and other stakeholders as part of the implementation of e-services of “A State in a Smartphone” in Ukraine. The article presents the structure of the authors’ model of information quality assessment, which includes three levels of characteristics and allows calculating the integral indicator of information quality. The model involves the use of expert research methods. The results of the study indicate that the information provided to users by public authorities has a fairly high level of quality, but there are reserves for improvement.

Modern public administration in Ukraine is a system aimed at providing administrative services to the population. The introduction of e-government standards together with indicators and tools for measuring the degree of implementation of the standard is a practical realization of the constitutional right of citizens to receive publicly available administrative services of sufficient level of quality.

The subjects of the provision of electronic public services are both state and local governments and any other bodies to which the state delegates their implementation. The legitimacy of state institutions is related to the effectiveness of identifying, modeling, and realizing individual and group interests and needs. This model of the state emphasizes the orientation of social communication “from the bottom up.” An indicator of the effectiveness of public administration is customer satisfaction. Recent research on the benefits of information and communications technology (ICT) in public service delivery confirms the relevance of this issue.1,2,3,4

The principles and directions of further reforming the activity of public authorities in the field of administrative services are defined in the concept of development of the system of providing administrative services by the executive authorities, approved by the decree of the Cabinet of Ministers of Ukraine on February 15, 2006, N 90-p. According to the concept, to improve the quality of administrative services, it is necessary to maximize decentralization of their provision; ensure decent salary in administrative bodies, motivate employees to achieve the end result, rather than formal compliance with the rules; develop administrative service standards; introduce modern forms of administrative services provision, which will allow organizing all or most common administrative services provided at a certain administrative and territorial level in one room; and arrange payment for services directly on site.

The Law of Ukraine “On Administrative Services” No 5203-VI Revision of April 4, 2018, reflects the specifics of introducing the idea of a Service State in Ukraine, which, in particular, refers to the implementation of the idea of serving the state to citizens, ensuring the realization of the rights of citizens and economic entities through the provision of administrative services, the establishment of centers of administrative services on the basis of local self-government bodies and district state administrations, which should create more convenient and accessible conditions for communication of citizens with the authorities.

In the Order of the Cabinet of Ministers of Ukraine “On Approval of the Concept of Development of the Electronic Services System in Ukraine” No. 918-2016-r dated November 16, 2016, e-service is defined as an administrative and another public service, which is provided to the subject of the appeal in electronic form by means of information, telecommunication, information, and telecommunication systems. In Art 1 of the Law of Ukraine “On Electronic Confidential Services” № 2155-VIII, Revision dated February 13, 2020, the term “electronic service” is formulated as any service provided through the information and telecommunication system.

The Concept of Development of Electronic Democracy in Ukraine and the Plan of Measures for its implementation provides for measures to increase the readiness of public authorities, businesses, and citizens to use the opportunities of electronic democracy. The concept proposes a transformational way that emphasizes the enhancement of e-government functionalities and the reduction of the authorities’ expenditures on the exercise of power by applying modern innovative approaches, methodologies, and technologies. The concept envisages the implementation of complex measures in the following areas: modernization of public services and development of interaction between government, citizens, and business through ICTs; modernization of public administration through ICTs; and e-government development management. The main measures for ensuring the development of electronic services in Ukraine are introduction of electronic services in all spheres of public life, as well as providing integrated electronic services for life and business situations; implementation of the one-stop-shop principle by ensuring the development and functioning of the Single State Portal of Administrative Services; development of electronic public procurement, electronic contracts and invoices, and electronic auctions; and promoting the use of electronic services by individuals and legal entities.

In October 2019, the Information and Communication Department of the Cabinet of Ministers of Ukraine presented the first e-services of “A State in a Smartphone” (https://www.kmu.gov.ua). These services are part of the new concept of the Digital State.5 According to the concept, Ukraine is implementing the “Diia” project (The State and I). The “Diia” application is being developed by the Ministry of Digital Transformation in cooperation with the Ministry of Internal Affairs, the largest Ukrainian bank, PrivatBank and the international IT Company EPAM. Beta testing of a mobile application in which more than 58,000 citizens take part started on December 16, 2019.6

December 4, 2019, the Government of Ukraine approved the Resolution “Issues of the Unified State Web Portal of Electronic Services and the Unified State Portal of Administrative Services” and the corresponding Regulations. The Regulation governs the functioning of the “Diia” portal as the Unified State Web Portal of Electronic Services. This portal has an official Internet address diia.gov.ua. Use of the “Diia” portal is carried out free of charge, around the clock, 7 days a week.

The project supports digital operators in the deployment of 4G networks, and in cooperation with the Ukrainian Library Association is developing digital literacy. On January 21, 2020, the national digital literacy platform “Digital Education” was launched to provide access to free education (https://osvita.diia.gov.ua).

The application of the concept of “Digital State” has several significant advantages: the country becomes more competitive on the world stage and the authorities and local self-government increase the speed of data exchange between state structures and speed of providing public services to the population and business. This concept makes it possible to make services more accessible, reduce corruption in state bodies, the burden on civil servants and the state apparatus, and reduce the costs of providing services.

An important component of providing these services should be a quality system that meets the needs of both service users and all stakeholders. Among the functions of this system, the most important is the assessment of the quality of information. In recent years, a number of studies have appeared on the quality of information.7,8,9,10,11,12

There are many approaches and models to the theory and practice of information quality assessment. The possibility of their application depends on many factors. Expert evaluations are the most convenient to assess the quality of information for the general public, where different categories of users can serve as experts, such as citizens, civil servants, people with disabilities, business representatives, and so on.

The key research question of this article is to improve the system of quality assessment of the information provided by government agencies to citizens, businesses, and other stakeholders as part of the implementation of e-services of “A State in a Smartphone.”

The main contributions of this work are as follows:

  • Development of a model for assessing the quality of information, which differs from existing models by the presence of three hierarchical levels of information characteristics and allows you to calculate the integral indicator of information quality based on expert assessments and to determine quality characteristics at all levels of the model.

  • Development of the information quality assessment framework that establishes the procedure for conducting studies by the expert method.

  • Conducting assessment and analysis of information in beta testing of the “Diia” mobile app, determining the most important quality characteristics.

  • Providing practical recommendations for improving the quality of the information in public authorities and improving the electronic services system in Ukraine.

The sections of the paper are organized as follows: section “Literature Review” deals with the literature on the topic of research. The next two sections provide a methodology for the study and the information quality assessment model. Section “Assessment and Analysis of Information Quality” explains how assessment and analysis of information quality are put into practice. The last section presents conclusions and a description of future work.

Literature Review

There is a great deal of research into the quality of the information in recent years. Dai et al. examined data-profiling technology to control data quality,13 Demoulin and Coussement explored an advanced technology adoption model that integrates information quality and senior management support,14 and Gharib and Giorgini, offered a framework for modeling and analysis of information quality requirements for sociotechnical systems.15

Hao et al. analyzed the fiscal stimulus program’s effect on accounting information quality,16 Kang and Namkung examined the quality of information when evaluating customers for product trade,17 and Klassen et al. examined the relationship between information quality and risk in decision-making.18 McDowall’s book Data Integrity and Data Governance is dedicated to helping implement data integrity and data management in the pharmaceutical industry,19 Xiaofeng and Xiang reviewed the concept of big data, explore its current state and summarize new challenges in the future,20 and Yagci and Das examined the amount and quality of product design information available in online reviews.21

A large number of studies on information quality conducted in different years required reviews and the creation of a framework for classifying literature and identifying the main topics and themes that distinguish this area.22 Some authors have taken as a basis the comparison of data quality with the quality of physical objects,23 whereas others focused on the assessment and contextual aspects of data quality.24 Some authors considered usability factors.25 Zhu et al. provided an overview of the evolution and current landscape of data research.26

Among the studies, there are works that examine the dimensions of quality. Dimensions allow evaluating information quality. Researchers of information quality use models in which they take into account relationships between dimensions and overall information quality. Among the most commonly used dimensions are Relevance, Reliability, and Timeliness,27 Accessibility, Interpretability, Accuracy, Completeness, and Timeliness.28 Wixom and Todd consider information quality in four dimensions: Completeness, Accuracy, Format, and Currency.29 Lee et al. provided a classification of dimensions suggested by various authors, including Value-added, Transaction availability, Timeliness, Usefulness, Appearance, Conciseness, Uniqueness, Comparability, Comprehensiveness, Essentialness, Attribute granularity, and others.30 Francisco et al. proposed a classification of dimensions according to data quality problems. Kenett and Shmueli described the dimensions of information quality used by various international organizations.31 Batini and Scannapieco provided a detailed analysis of many dimensions of information quality.32

A number of studies have been devoted to digital transformations. The problem of administrative burden, which can be solved using digital government, is relevant.33 Digital government identifies societal adoption, organizational readiness, and other conditions under which it can be an effective tool for Administrative Burden Reduction. Nielsen et al. investigated the Burden reduction as a key issue in modern public administrations’ and businesses’ agendas.34 The purpose of this study is to review the current state of the art of reducing administrative workload and to identify current gaps for better planning of future research. The authors examined 122 articles from a pool of 742 documents in four dimensions: methodology, type, focus, and target stakeholders. The paper by Lopes et al. presents the research landscape for ICT-enabled public service delivery scientific and policy literature.35 The paper outlines the research gaps on Public Service Delivery and a series of policy recommendations to enhance public service delivery.

There is a difference between data quality and online information quality. In a study by Wang et al., the authors focused on the study of websites, namely, in determining two characteristics—the perceived quality of information and the perceived value of online communities.36 It was found that the value of online communities is positively related to consumer confidence. In addition, loyalty to a website is highly dependent on consumer confidence in that website.37,38,39

The issue of improving the efficiency of value creation through user-oriented segmentation and the integration of relevant content, personal data, and transactions were explored by Morten Meyerhoff Nielsen and Robert Krimmer.40

The quality of online information and the quality of data are important components in an e-government system. This research focuses on the quality of data that determines the value of documents that the user receives as a result of public services.

Jakob Nielsen proposed the methods of Usability Inspection as a set of cost-effective ways of evaluating user interfaces to find usability problems.41 User experience guidelines for designing applications and websites for mobile devices have also been developed, which is based on 17 in-person and 10 remote usability studies with users in eight countries.42

The development of information technologies, in particular, artificial intelligence (AI), plays an important role in the provision of electronic public services. Ben Shneiderman explored the possibilities of using human-centered artificial intelligence (HCAI), proposing ideas for a two-dimensional structure of HCAI, which shows how to have both a high level of human management and a high level of automation; the transition from imitating people to empowering people; a three-tier management structure that includes teams of software engineers, managers who can emphasize a safety culture and industry certification.43 His research also looks at emulation to understand a person’s ability to build systems that perform tasks as well or better than humans, and the use of AI techniques to build widely used products and services.44 An important issue in the field of Service Delivery is ensuring a minimum level of usability in public service offers online. Morten Meyerhoff Nielsen’s tutorial focuses on usability guidelines and standards as a directive tool with the aim of assisting governments in its implementation, and streamlining the governance model conceived to ensure compliance with the usability criteria as outlined in the usability guidelines.45

This section has considered the issue of assessing the quality of information as a subtopic of the literature review. This issue is also addressed in the section “Model of Information Qulaity Assessment.”

Research Methodology

The research methodology involves the use of expert methods for assessing the quality of information. For this purpose, a three-level hierarchical model was developed, which allows experts to evaluate first-level dimensions of information quality and calculate quality indicators for all levels of the model (Table 1). The result of the calculations is the integral indicator of information quality. The study was carried out in the sequence shown in Figure 1.

TABLE 1

Model of Information Quality Assessment with Three Levels of Characteristics

QualityLevel 3 characteristicsK3jG3j %Level 2 characteristicsK2jG2j %Level 1 characteristics (dimensions)K1jRating, QjWeight, G1j %
Integral indicator of information quality Ki1 Managerial characteristics K31 G31 Consumer characteristics K21 G21 Purpose K11 Q1 G11 
Addressability K12 Q2 G12 
Value K13 Q3 G13 
Suitability for decision- making K22 G22 Completeness K14 Q4 G14 
Accuracy K15 Q5 G15 
Reliability K16 Q6 G16 
Necessity K17 Q7 G17 
Systemic characteristics K32 G32 Structural characteristics K23 G23 Transmission method K18 Q8 G18 
Format K19 Q9 G19 
Independence of form K110 Q10 G110 
Synergistic characteristics K24 G24 Nonadditivity K111 Q11 G111 
Noncommutativeness K112 Q12 G112 
Nonassociativity K113 Q13 G113 
Performance characteristics K33 G33 Time characteristics K25 G25 Speed K114 Q14 G114 
Timeliness K115 Q15 G115 
Regularity K116 Q16 G116 
Relevancу K117 Q17 G117 
Efficiency characteristics K26 G26 Usefulness K118 Q18 G118 
Suitability K119 Q19 G119 
QualityLevel 3 characteristicsK3jG3j %Level 2 characteristicsK2jG2j %Level 1 characteristics (dimensions)K1jRating, QjWeight, G1j %
Integral indicator of information quality Ki1 Managerial characteristics K31 G31 Consumer characteristics K21 G21 Purpose K11 Q1 G11 
Addressability K12 Q2 G12 
Value K13 Q3 G13 
Suitability for decision- making K22 G22 Completeness K14 Q4 G14 
Accuracy K15 Q5 G15 
Reliability K16 Q6 G16 
Necessity K17 Q7 G17 
Systemic characteristics K32 G32 Structural characteristics K23 G23 Transmission method K18 Q8 G18 
Format K19 Q9 G19 
Independence of form K110 Q10 G110 
Synergistic characteristics K24 G24 Nonadditivity K111 Q11 G111 
Noncommutativeness K112 Q12 G112 
Nonassociativity K113 Q13 G113 
Performance characteristics K33 G33 Time characteristics K25 G25 Speed K114 Q14 G114 
Timeliness K115 Q15 G115 
Regularity K116 Q16 G116 
Relevancу K117 Q17 G117 
Efficiency characteristics K26 G26 Usefulness K118 Q18 G118 
Suitability K119 Q19 G119 
FIGURE 1

Information quality assessment framework.

FIGURE 1

Information quality assessment framework.

Close modal

At the first stage, a preliminary assessment of information was carried out by determining characteristics: purpose, volume, methods and means of dissemination, presentation, sources, and users of information. A preliminary assessment made it possible to select experts who are knowledgeable in this field of research, who determined the weight coefficients G1, G2, and G3 for each characteristic of the model. The suggestions of each expert allowed obtaining consistent (average) values of weighting coefficients.

The next stage of the study was the calculation of relative quality indicators (Kij) for each level of the model and the integral indicator of information quality (Kki).

The chosen method is based on the procedure for the selection of information quality characteristics by competent experts from among civil servants of various government bodies. The selection of experts was carried out during vocational training courses. As part of the study, the experts assessed the quality of the information they use in their activities and which fills the information databases of sites within the e-government system. They were offered a list of information quality characteristics listed in the “Information Quality Assessment Model” section and obtained from a literature review. The task of the experts was to select the most representative from their point of view characteristics for the evaluation of information. The selection was carried out in several rounds with an analysis of the results after each round. The experts came to a consensus in terms of determining the dimensions of the first level of the model presented in Table 1, which are the most convenient for assessing the quality of the information in the e-government system.

Further, the experts agreed on the classification of these characteristics, which made it possible to develop a special questionnaire. The questionnaire was proposed to determine the weight coefficients in three stages: in the first stage, the experts assessed the coefficients individually; in the second stage, the collective assessment was determined by discussion in small groups; and in the third stage, they made a collective decision on the coefficients. The model was tested in a number of departments of the regional state administration and local governments, which provide information to the portal “Diia.” The results of these studies were ratings that satisfied the experts. The model was used to evaluate information from the Diia portal.

There are several Public Service Portals in Ukraine, but the most promising and discussed in society and the scientific community is the portal “Diia.” It is a mobile application with digital documents and a portal with public services.

Information Quality Assessment Model

The International Quality Standard ISO 9000:2015 defines quality as “the degree to which a set of inherent characteristics of an object meets the requirements.” There are a number of scientific papers in which the authors determine the characteristics of information quality (dimensions). Lee et al. propose 15 dimensions46. Pipino, Lee, and Wang use 16 dimensions in their studies.47 Eppler considered 70 dimensions in his study.48Britchenko proposed 19 dimensions.49

There are also international standards that apply to assess the quality of information systems. An International Standard ISO/IEC 25012:2008, “Software Engineering—Data Quality Model Requirements and Evaluation (SQuaRE)” offers a number of dimensions that are used to evaluate information quality. These dimensions are divided into two categories: Inherent Data Quality and System-Dependent Data Quality. Inherent dimensions are Accuracy, Completeness, Consistency, Credibility, and Currentness. System-Dependent features include Availability, Portability, and Recoverability. Several features are both Inherent and System-Dependent dimensions: Accessibility, Compliance, Confidentiality, Efficiency, Precision, Traceability, and Understandability. This division of information into categories does not make it possible to assess the factors that most affect the quality of information and the causes of deficiencies in the provision of electronic services. The Inherent Data Quality category includes general information dimensions, whereas the System-Dependent Data Quality category focuses on technical issues of information systems development. An accurate assessment of the information quality can be made using a model that shows the relationship of dimensions with certain forces (political, social, organizational, and technological) that influence the development of public electronic services.50

Various authors of scientific papers have offered a wide variety of dimensions of information quality, but some of them are often repeated: Completeness, Relevancy, Timeliness, Reputation, Security, Objectivity, and Understandability.51 Eppler points to the dimensions of Accuracy, Clarity, Correctness, Related Terms, and Timeliness.52 Ehling and Körner provide the following dimensions: Relevance, Accuracy, Timeliness and Punctuality, Comparability, Coherence, Accessibility, and Clarity.53

One of the dimensions mentioned in information quality research is readability, which is associated with ease of reading.54 Ghose and Ipeirotis took this dimension as a basis and proposed an approach that involved writing-style assessment to determine the quality of information.55 The readability included the length of the review in characters, the length of the review in words, the length of the review in sentences, and the number of spelling errors in the review. This approach is used to automate text analysis. The Flesch–Kincaid grade-level formula in the form of a linear function,56 the Flesch reading ease formula,57 and the formula for calculating the LIX readability index58 were proposed for calculating readability. The LIX index calculation formula is as follows:
(1)
where
(2)
(3)

Long words mean that the word has more than six characters. The advantages of LIX are that this method gives an unbiased assessment, and it is simple and easy to use. However, LIX has a number of disadvantages. The evaluation results do not indicate the reasons for poor text quality,59 do not take into account the content and organization of the text,60 and the use of these formulas encourages authors to break long sentences into short ones, which can interfere with text comprehension.61 In addition, in the case of using models with a large number of dimensions, it leads to the use of various methodological approaches: some dimensions are determined by calculation methods and others by expert assessments. Therefore, our research model does not use the LIX readability index and the information quality is assessed only by expert methods.

Basic quality dimensions for statistics are given in the book Understanding Economic Statistics: An OECD Perspective.62 These include Relevance, Accuracy, Timeliness and Punctuality, Accessibility, Interpretability, Coherence, and Credibility. The European statistical system uses the following quality dimensions: Relevance, Accuracy, Timeliness and Punctuality, Comparability, Coherence, Accessibility, and Clarity.63 Broydo and Ilyina offer the following dimensions of quality: Accuracy, Completeness, Interpretability, Relevance, Timeliness, Objectivity, Credibility, Precision, and Currentness.64 Britchenko offers the following dimensions: Purpose, Addressability, Value, Reliability, Necessity, Transmission Method, Format, Independence of Form, Nonadditivity, Noncommutativeness, Nonassociativity, Speed, Regularity, Usefulness, and Suitability.65 The analysis of the literature shows that among the dimensions of the quality of information, the most important and most commonly used in scientific works and standards are Completeness, Accuracy, Timeliness, and Relevancy.

Table 1 shows a three-level model for assessing the quality of information, which was developed based on the analysis of the literature.

The model was built using deductive logic when choosing characteristics from the total number of existing ones. The model development began with the third level. The quality of information depends on the system in which it is created. In our case, it is an e-government system. E-government is a management structure, so one aspect of the study should be public management. Accordingly, evaluating the information generated in this system should bea managerial characteristics. The second aspect should be a systems approach that considers e-government as a system that is a group of interacting and interrelated elements that act according to a set of rules to form a unified whole.66 Based on this, system characteristic were proposed. The third aspect is the process approach, which is the basis of the quality management system and is formulated in the ISO 9000:2015 standard. The implementation of processes in the system and, accordingly, the achievement of quality can be assessed using performance characteristics.

At the second level of the model, there should be characteristics that are part of the characteristics of the third level.

Management characteristics were divided into consumer characteristics and suitability for decision-making. Consumer characteristics were chosen because the system of public administration must be human-oriented and the main object of public services should be people. Consumer characteristics were divided into Purpose (each information has a purpose), Addressability (the service has its addressee), and Value (value of the service to the user). Suitability for decision-making has two aspects: the first Suitability for civil servants who provide services through the system and the second Suitability for the user, who must be able to make independent decisions about information retrieval, analysis, and forms of service. Based on these considerations, Suitability for decision-making was divided into Completeness, Accuracy (information should reflect correct data), Reliability (user can trust the information received), and Necessity (user gets the information he needs).

Systemic characteristics were divided into Structural characteristics and synergistic characteristics. The structure is the way in which the parts of a system are arranged or organized. The transmission method was included here as it is important to organize information flows to the user and to find ways of transition from one array of information to another array. Format as a structural characteristic describes the form of presentation of information to the user, and independence of form refers to the semantic component and ease of perception of information. Synergistic characteristics are related to the application of a synergistic approach. The synergetic approach is to identify opportunities for combination, cooperation, and self-organization of system elements. It incorporates different elements or groups to work together. Synergy means the interaction of elements that produces an effect greater than the effect that would result from simply adding up the effects of each individual element.67 Nonadditivity is the dimension that characterizes synergy in the system. The user must receive a comprehensive service that includes all the necessary interconnected and interdependent components on a “single window” basis. Noncommutativeness in this approach makes it possible to assess the logic of the procedure for providing information that is part of the public service. Nonassociativity refers to streamlining the process of providing information to achieve the greatest effect.

Performance characteristics were divided into time characteristics and efficiency characteristics. Time is a critical characteristic of any process. It includes Speed (dimension important for the user who seeks to receive the service as soon as possible), Timeliness (this dimension becomes critical if the user does not have time to use the received service), Regularity (data should be received regularly in databases), and Relevance (data must be updated; the user should not receive outdated information). Efficiency consists of usefulness, which describes the effect of the use of information (costs should be less than the effect obtained) and suitability, which is a general dimension of the efficiency of the system.

Considering the large number of characteristics that can describe the quality of information, it is very difficult to develop an “ideal” model. However, the proposed logic allows us to create a sufficiently adequate model in which the properties of the model and the corresponding properties of the modeled object coincide.

The model uses quality dimensions at the first level, which are most convenient for conducting research by interviewing users of information.

  1. Purpose. The transmitted information has different purposes depending on the type of activity. The main goals of the information provided by the electronic services system are informing, evaluation, planning, control, search, and so on.

  2. Addressability. It complements the characteristics of the purposes and indicates the presence of a consumer of information. Consumers of electronic services are citizens, businesses, public authorities, and others. Each user has specific information requirements.

  3. Value. It does not obey exact laws and is a subjective factor. There is no single criterion for the value of information, but the main one can be considered the degree of influence on the effectiveness of solving problems.

  4. Completeness (sufficiency). This dimension of information means that messages about a managed object should cover all states of the object by all managed parameters. Adding any other messages does not give new knowledge about the object. Information is considered complete if it is possible to make the right decision on its basis.

  5. Accuracy. This is a dimension that quantifies the parameters that determine the state of a managed object. Decreasing these parameters is unacceptable, and the increase may be unnecessary.

  6. Reliability. This dimension means that there is complete confidence that the received messages are not distorted. This characteristic of the e-service must be confirmed by the necessary regulatory support.

  7. Necessity. This characteristic of information is that messages about an object should give only the necessary knowledge. Shortening the information contained in a message reduces the knowledge of the object.

  8. Transmission method. The methods of transmitting information to people are based on feelings (vision, hearing, taste, touch, and smell). Computers process information using signals. The perception of information by a person depends on the way the signals are transmitted.

  9. Format. The format involves providing information in the form of documentation, magnetic, laser disks, and so on. The information contained in the electronic service should be provided in a user-friendly form.

  10. Independence of form. It is a property of information to be independent of the form in which it is stored or delivered to the consumer. The user should not feel inconvenienced by the use of different technologies for processing, transmitting, and storing information.

  11. Nonadditivity. This dimension consists in the fact that the result of the joint action of two information pieces on a system is not equal to the sum of the results of the impact of these two pieces of information on the same system.

  12. Noncommutativeness. It is a dimension that consists of the fact that the impact consistently in time on any system of information pieces A, B, C, D, and so on, will differ from the result of the impact on the same system of the same information pieces A, B, C, D, and so on, but if they come one after another in time in a different sequence.

  13. Nonassociativity. This dimension of information lies in the fact that the impact consistently in time on any system of information pieces A, B, and C will differ from the result of exposure to the same system of information pieces A and D, where D is information obtained as a result of the joint action of information pieces B and C.

  14. Speed. The speed of transmission and reception of information depends on the organization of information flows. It can be expressed in time. Computer performance is determined by the number of pieces of information per unit of time. The user of information seeks to reduce the time of receiving electronic services.

  15. Timeliness. This dimension of information is characterized by the endurance of time intervals during which the necessary messages arrive. Delaying information disrupts service delivery and renders messages useless.

  16. Regularity. If the messages about a managed object are received with the necessary frequency determined by the control mode, then the requirement of the regularity of information is observed.

  17. Relevancy. The aging of information, that is, the loss of its value over time, means that the new information received changes the previous one. The novelty of information refers to a characteristic that affects the interaction of the source and the recipient of the information and reflects the time from the onset of the event to the receipt of the message.

  18. Usefulness. Useful information is associated with a problem being solved at any determined moment in time. The costs of obtaining and using the information should not be more than the effect of its use.

  19. Suitability. It is a property of information that includes characteristics of value, completeness (sufficiency), necessity and redundancy, accuracy, reliability (probability), speed, timeliness, regularity, and usefulness.

Assessment and Analysis of Information Quality

The model shown in the previous section of the article was used to assess the quality of information on the portal “Diia.” The fourth level of the model is the integral indicator of information quality.68

The integral indicator was calculated using the relative quality indicators (Kij), which are determined by the following formula:
(4)

Qij—is the expert rating in points (from 1 to 5),

qil—the lowest rating of all possible (equal 1), and

qih —the highest rating of all possible (equal 5).

The integral indicator of information quality (Kki) is determined on the basis of the hierarchy of characteristics, weight coefficients, and relative quality indicators by the following formula:
(5)

Kij—the relative quality indicator, and

Gi—the weight coefficient of the characteristic for each level of the model.

To measure simple properties, the method of expert assessment was used, after which the relative quality indicators Kij and the integral indicator of the quality of information Kkj were calculated. Experts in assessing the quality of information were civil servants working at regional-level government bodies. A group of experts determined the weight coefficients (importance) of the characteristics at all levels of the model (G1, G2, and G3). Each respondent assessed the information using the first-level quality dimensions on the Likert scale (from 1 to 5). At the next stage, the relative quality indicators were calculated taking into account the weight coefficient of each characteristic in percent.

Specialists from various public authorities took part in the study, such as the Office of the Regional Administration, district departments of the Pension Fund of Ukraine, the Department of Social Protection of the Regional State Administration, the Department of Labour and Social Protection of the District Council, and the Department of State Migration Service in the Region and district councils. The experts had at least 10 years of experience. They researched the pages of the portal site in accordance with their professional competencies, namely, “Pensions, benefits and assistance” (https://diia.gov.ua/services/categories/gromadyanam/pensiyi-pilgi-ta-dopomoga), “Passport of a citizen of Ukraine for travel abroad with a contactless electronic medium” (https://guide.diia.gov.ua/register/00027/), and “Licenses and permits” (https://diia.gov.ua/services/categories/gromadyanam/licenziyi-ta-permission), and others. The study took into account that end users have secondary or higher education, as well as basic skills in working with information systems. About 97.1% of the population of Ukraine has this level of education (http://www.futureskills.org.ua/ua/map).

A total of seven categories of services were studied (https://guide.diia.gov.ua/).

  • Citizenship and migration

  • Social protection

  • Life safety

  • Professional activity

  • Finance and taxes

  • Education, sports and tourism, and culture and religion

  • Protection and security

The literature contains different opinions about the size of the sample of participants. For example, Sim et al. recommend from 4 to 35, other scientists recommend from 12 to 60.69,70

A total of 35 civil servants took part, some of whom are internal experts for conducting internal audits of public authorities (Resolution of the Cabinet of Ministers of Ukraine, 2011). The experts were divided into groups of five people for each of the seven categories of services. The number of documents in each category was estimated by viewing the site. Using a stratified sample for each document and calculating its size using a sample size calculator (https://www.surveysystem.com/sscalc.htm), the number of documents required for a qualitative study was determined.

The confidence level—95 and the confidence interval—5 were used in the calculation. On average, one expert processed 346 documents. By document, we mean information on the site in the form of text, sound, images, and (or) a combination thereof, which contains details that allow it to be identified. Sample sizes by document type are shown in Table 2.

TABLE 2

Sample Sizes by Document Type

Types of informationPopulationSample size
Citizenship and migration 483 214 
Social protection 665 244 
Life safety 206 137 
Professional activity 127 96 
Finance and taxes 369 188 
Education, sports and tourism, and culture and religion 351 184 
Protection and security 242 149 
Types of informationPopulationSample size
Citizenship and migration 483 214 
Social protection 665 244 
Life safety 206 137 
Professional activity 127 96 
Finance and taxes 369 188 
Education, sports and tourism, and culture and religion 351 184 
Protection and security 242 149 

The Delphi method was used to determine the weight coefficients of the characteristics, also known as Estimate-Talk-Estimate or ETE.71 It is a systematic interactive forecasting method used by a group of experts. The main idea of the method is that the decisions of a structured group of experts are more accurate than those of an unstructured group.72

The peculiarity of this method is that the head of the study after each round makes an anonymous summary of the previous round. It also indicates the reasons why the experts expressed their opinion. Next, the experts review the preliminary responses based on the responses of other members of their group. This reduces the range of answers, and the group converges to the “correct” answer. The process took place in three rounds during which weight coefficients of characteristics were set for all levels of the model. The criteria for stopping the process were consensus building and stability of results. Each group of experts determined the coefficients for their type of information. The evaluation results were determined by calculating the average values of the coefficients for the last two rounds.73 The weight coefficients in the models developed by different groups of experts were similar in their values. Therefore, to further assess the quality of information, it is advisable to use the same coefficients for the entire site. That is, one model is enough to conduct the entire study.

The evaluation of the information quality was carried out by the same groups of experts who were involved in determining the weight coefficients. Each group investigated their type of information according to Table 2. A 1 to 5 Likert scale was defined in terms of very bad (1), bad (2), neutral (3), good (4), and very good (5).

  1. Very bad means lack of information on the portal.

  2. Bad means that there is only reference information about where you can get the service without using the portal.

  3. Neutral means that the service is already on the site, but not in full.

  4. Good means that the public service is available, but the user has some difficulty using the system (requires special skills).

  5. Very good means that the service is easy to get, the user does not need special skills, and the system is user-friendly.

The “cutoff” point in the Likert scale separates values 1 and 2 from 3, 4, and 5. A value of 1 means no information on request, and 2 only reference information, which does not allow receiving the service through the portal. As the portal was created for the purpose of providing services on the principle of “single window,” such information does not satisfy the user and can be considered negative.

Tables 3 to 6 show the results of a study of the quality of information from the portal “Diia.” Table 3 shows average weight coefficients for the first level of the model. Table 4 shows average weight coefficients for the second and third levels of the model. Table 5 shows the results of calculating the relative quality indicators for the first level of the model. Table 6 shows the results of calculating the relative quality indicators for the second and third levels of the model.

TABLE 3

Average Weight Coefficients for the First Level of the Model

Types of documentsPurposeAddressabilityValueCompletenessAccuracyReliabilityNecessityTransmission methodFormatIndependence of formNonadditivityNoncommutativenessNon associativitySpeedTimelinessRegularityRelevancyUsefulnessSuitability
G11G12G13G14G15G16G17G18G19G110G111G112G113G114G115G116G117G118G119
Citizenship and migration 56 31 13 15 35 24 26 34 52 14 35 41 24 24 38 11 27 63 37 
Social protection 32 45 23 25 29 24 22 44 41 15 26 38 36 29 29 21 21 46 54 
Life safety 43 36 21 27 31 22 20 34 31 26 36 33 31 22 33 21 24 61 39 
Professional activity 24 43 33 18 23 31 28 24 45 31 25 37 38 18 24 27 31 54 46 
Finance and taxes 22 33 45 25 23 31 21 35 36 29 34 30 36 25 18 23 34 42 58 
Education, sports and tourism, and culture and religion 41 31 28 24 26 27 23 19 40 41 33 27 40 20 31 22 27 37 63 
Protection and security 31 35 34 25 25 23 27 22 38 40 31 31 38 18 27 26 29 51 49 
Average rating by types of documents 36 36 28 23 27 26 24 30 40 28 31 34 35 22 29 22 28 51 49 
Types of documentsPurposeAddressabilityValueCompletenessAccuracyReliabilityNecessityTransmission methodFormatIndependence of formNonadditivityNoncommutativenessNon associativitySpeedTimelinessRegularityRelevancyUsefulnessSuitability
G11G12G13G14G15G16G17G18G19G110G111G112G113G114G115G116G117G118G119
Citizenship and migration 56 31 13 15 35 24 26 34 52 14 35 41 24 24 38 11 27 63 37 
Social protection 32 45 23 25 29 24 22 44 41 15 26 38 36 29 29 21 21 46 54 
Life safety 43 36 21 27 31 22 20 34 31 26 36 33 31 22 33 21 24 61 39 
Professional activity 24 43 33 18 23 31 28 24 45 31 25 37 38 18 24 27 31 54 46 
Finance and taxes 22 33 45 25 23 31 21 35 36 29 34 30 36 25 18 23 34 42 58 
Education, sports and tourism, and culture and religion 41 31 28 24 26 27 23 19 40 41 33 27 40 20 31 22 27 37 63 
Protection and security 31 35 34 25 25 23 27 22 38 40 31 31 38 18 27 26 29 51 49 
Average rating by types of documents 36 36 28 23 27 26 24 30 40 28 31 34 35 22 29 22 28 51 49 
TABLE 4

Average Weight Coefficients for the Second and Third Levels of the Model

Types of documentsManagerial characteristicsSystemic characteristicsPerformance characteristicsConsumer characteristicsSuitability for decision-makingStructural characteristicsSynergistic characteristicsTime characteristicsEfficiency characteristics
G31G32G33G21G22G23G24G25G26
Citizenship and migration 20 23 57 67 33 73 27 81 19 
Social protection 32 18 50 43 57 73 27 46 54 
Life safety 33 29 38 56 44 44 56 35 65 
Professional activity 49 20 31 62 38 51 49 58 42 
Finance and taxes 40 33 27 52 48 34 66 47 53 
Education, sports and tourism, and culture and religion 52 27 21 47 53 51 49 58 42 
Protection and security 44 31 25 35 65 61 39 62 38 
Average rating by types of documents 39 26 36 52 48 0.55 45 55 39 
Types of documentsManagerial characteristicsSystemic characteristicsPerformance characteristicsConsumer characteristicsSuitability for decision-makingStructural characteristicsSynergistic characteristicsTime characteristicsEfficiency characteristics
G31G32G33G21G22G23G24G25G26
Citizenship and migration 20 23 57 67 33 73 27 81 19 
Social protection 32 18 50 43 57 73 27 46 54 
Life safety 33 29 38 56 44 44 56 35 65 
Professional activity 49 20 31 62 38 51 49 58 42 
Finance and taxes 40 33 27 52 48 34 66 47 53 
Education, sports and tourism, and culture and religion 52 27 21 47 53 51 49 58 42 
Protection and security 44 31 25 35 65 61 39 62 38 
Average rating by types of documents 39 26 36 52 48 0.55 45 55 39 
TABLE 5

The Results of Calculating the Relative Quality Indicators for the First Level of the Model

Types of documentsPurposeAddressabilityValueCompletenessAccuracyReliabilityNecessityTransmission methodFormatIndependence of formNonadditivityNoncommutativenessNon associativitySpeedTimelinessRegularityRelevancyUsefulnessSuitability
K11K12K13K14K15K16K17K18K19K110K111K112K113K114K115K116K117K118K119
Citizenship and migration 0.75 0.5 0.5 0.75 0.75 0.75 0.5 0.5 0.75 0.75 0,5 0,75 0,75 0,75 0,5 0,75 0,5 
Social protection 0.75 0.5 0.5 0.75 0.75 0.75 0.5 0.5 0.75 0.75 0,5 0,75 0,75 0,75 0,5 0,75 0,5 
Life safety 0.75 0.75 0.5 0.75 0.5 0.75 0.75 0,5 0,5 0,75 0,5 0,75 0,75 
Professional activity 0.5 0.75 0.75 0.5 0.5 0.75 0.5 0.75 0.75 0,75 0,75 0,5 0,5 0,75 0,75 0,5 
Finance and taxes 0.75 0.75 0.5 0.5 0.75 0.75 0.75 0.5 0.75 0.5 0.5 0,75 0,25 0,75 0,5 0,5 0,75 
Education, sports and tourism, and culture and religion 0.75 0.75 0.5 0.75 0.5 0.75 0.5 0.75 0.75 0,5 0,5 0,5 0,75 0,5 0,25 0,75 0,75 
Protection and security 0.5 0.75 0.75 0.75 0.75 0.75 0.5 0.5 0.5 0,75 0,75 0,75 0,75 0,75 0,5 0,5 
Average rating by types of documents 0.75 0.71 0.68 0.71 0.68 0.75 0.61 0.64 0.79 0.71 0.71 0,64 0,68 0,71 0,68 0,61 0,71 0,79 0,61 
Types of documentsPurposeAddressabilityValueCompletenessAccuracyReliabilityNecessityTransmission methodFormatIndependence of formNonadditivityNoncommutativenessNon associativitySpeedTimelinessRegularityRelevancyUsefulnessSuitability
K11K12K13K14K15K16K17K18K19K110K111K112K113K114K115K116K117K118K119
Citizenship and migration 0.75 0.5 0.5 0.75 0.75 0.75 0.5 0.5 0.75 0.75 0,5 0,75 0,75 0,75 0,5 0,75 0,5 
Social protection 0.75 0.5 0.5 0.75 0.75 0.75 0.5 0.5 0.75 0.75 0,5 0,75 0,75 0,75 0,5 0,75 0,5 
Life safety 0.75 0.75 0.5 0.75 0.5 0.75 0.75 0,5 0,5 0,75 0,5 0,75 0,75 
Professional activity 0.5 0.75 0.75 0.5 0.5 0.75 0.5 0.75 0.75 0,75 0,75 0,5 0,5 0,75 0,75 0,5 
Finance and taxes 0.75 0.75 0.5 0.5 0.75 0.75 0.75 0.5 0.75 0.5 0.5 0,75 0,25 0,75 0,5 0,5 0,75 
Education, sports and tourism, and culture and religion 0.75 0.75 0.5 0.75 0.5 0.75 0.5 0.75 0.75 0,5 0,5 0,5 0,75 0,5 0,25 0,75 0,75 
Protection and security 0.5 0.75 0.75 0.75 0.75 0.75 0.5 0.5 0.5 0,75 0,75 0,75 0,75 0,75 0,5 0,5 
Average rating by types of documents 0.75 0.71 0.68 0.71 0.68 0.75 0.61 0.64 0.79 0.71 0.71 0,64 0,68 0,71 0,68 0,61 0,71 0,79 0,61 
TABLE 6

The Results of Calculating the Relative Quality Indicators for the Second and Third Levels of the Model

Types of documentsIntegral quality indicatorManagerial characteristicsSystemic characteristicsPerformance characteristicsConsumer characteristicsSuitability for decision-makingStructural characteristicsSynergistic characteristicsTime characteristicsEfficiency characteristics
Kk1K31K32K33K21K22K23K24K25K26
Citizenship and migration 0.74 0.65 0.76 0.76 0.64 0.69 0.80 0.65 0.79 0,66 
Social protection 0.67 0.65 0.72 0.68 0.58 0.70 0.74 0.66 0.75 0,62 
Life safety 0.78 0.76 0.69 0.86 0.86 0.63 0.70 0.68 0.78 0,90 
Professional activity 0.72 0.74 0.69 0.69 0.77 0.70 0.64 0.75 0.72 0,64 
Finance and taxes 0.67 0.66 0.71 0.64 0.64 0.69 0.66 0.74 0.64 0,65 
Education, sports and tourism, and culture and religion 0.68 0.77 0.56 0.61 0.85 0.69 0.55 0.58 0.51 0,75 
Protection and security 0.74 0.75 0.74 0.71 0.76 0.75 0.79 0.67 0.68 0,76 
Average rating by types of documents 0.71 0.71 0.70 0.71 0.73 0.69 0.70 0.68 0.70 0,71 
Types of documentsIntegral quality indicatorManagerial characteristicsSystemic characteristicsPerformance characteristicsConsumer characteristicsSuitability for decision-makingStructural characteristicsSynergistic characteristicsTime characteristicsEfficiency characteristics
Kk1K31K32K33K21K22K23K24K25K26
Citizenship and migration 0.74 0.65 0.76 0.76 0.64 0.69 0.80 0.65 0.79 0,66 
Social protection 0.67 0.65 0.72 0.68 0.58 0.70 0.74 0.66 0.75 0,62 
Life safety 0.78 0.76 0.69 0.86 0.86 0.63 0.70 0.68 0.78 0,90 
Professional activity 0.72 0.74 0.69 0.69 0.77 0.70 0.64 0.75 0.72 0,64 
Finance and taxes 0.67 0.66 0.71 0.64 0.64 0.69 0.66 0.74 0.64 0,65 
Education, sports and tourism, and culture and religion 0.68 0.77 0.56 0.61 0.85 0.69 0.55 0.58 0.51 0,75 
Protection and security 0.74 0.75 0.74 0.71 0.76 0.75 0.79 0.67 0.68 0,76 
Average rating by types of documents 0.71 0.71 0.70 0.71 0.73 0.69 0.70 0.68 0.70 0,71 

The tables also contain the average values of the studied indicators. These results indicate that at the second level, Structural characteristics (55%) and Time characteristics (55%) have the greatest weight on average. Managerial characteristics have the greatest weight (39%) at the third level. Consumer characteristics have the highest average score of the relative quality indicator at the second level (0.73). At the third level, Managerial and Performance characteristics received the highest score −0.71. Thus, the analysis shows that the most important quality attributes of electronic services “State in the smartphone” is Structural characteristics, Time characteristics, Managerial and Performance characteristics, and Consumer characteristics. Research also shows that improving the quality of information depends to a large extent on enhancing Consumer and Managerial characteristics.

As a result of the study, the average integrated indicator of information quality was 0.71. This indicator may take values from zero to one, so the result indicates that there are reserves to improve the quality of information.

The fact that at the second level, Structural characteristics and Time characteristics received the greatest weight coefficients from experts indicates the importance of technological factors and time factors for the user of electronic services. At the same time, planning, organization, motivation, control, and decision-making in the development processes of the service delivery system are important, confirming the high weight of managerial characteristics at the third level.

The last rows of Tables 3 and 4 show the values of the weight coefficients calculated as the arithmetic means of these coefficients by type of information. The absolute weight coefficients for the first level were also calculated in order to determine the characteristics that most affect the quality of information. If in the model (Table 1), the sum of weight coefficients is 100% within the characteristics of the second level, then to obtain the absolute weight, it is necessary to multiply the value of the weight coefficient of the first level by the weight coefficient of the second level and the coefficient of the third level. Then the sum of the values of all nineteen absolute coefficients of the first level is 100% (Table 7).

TABLE 7

Absolute Values of Weight Coefficients

Level 3 characteristicsGav3
%
Level 2 characteristicsGav2,
%
Level 1 characteristicsGav1,
%
Gabs
%
Managerial characteristics 39 Consumer characteristics 52 Purpose 36 7.30 
Addressability 36 7.30 
Value 28 5.68 
Suitability for decision-making 48 Completeness 23 4.31 
Accuracy 27 5.05 
Reliability 26 4.87 
Necessity 24 4.49 
Systemic characteristics 26 Structural characteristics 55 Transmission method 30 4.29 
Format 40 5.72 
Independence of form 28 4.00 
Synergistic characteristics 45 Nonadditivity 31 3.63 
Noncommutativeness 34 3.98 
Nonassociativity 35 4.10 
Performance characteristics 36 Time characteristics 55 Speed 22 4.36 
Timeliness 29 5.74 
Regularity 22 4.36 
Relevancу 28 5.54 
Efficiency characteristics 39 Usefulness 51 7.16 
Suitability 49 6.88 
      100 
Level 3 characteristicsGav3
%
Level 2 characteristicsGav2,
%
Level 1 characteristicsGav1,
%
Gabs
%
Managerial characteristics 39 Consumer characteristics 52 Purpose 36 7.30 
Addressability 36 7.30 
Value 28 5.68 
Suitability for decision-making 48 Completeness 23 4.31 
Accuracy 27 5.05 
Reliability 26 4.87 
Necessity 24 4.49 
Systemic characteristics 26 Structural characteristics 55 Transmission method 30 4.29 
Format 40 5.72 
Independence of form 28 4.00 
Synergistic characteristics 45 Nonadditivity 31 3.63 
Noncommutativeness 34 3.98 
Nonassociativity 35 4.10 
Performance characteristics 36 Time characteristics 55 Speed 22 4.36 
Timeliness 29 5.74 
Regularity 22 4.36 
Relevancу 28 5.54 
Efficiency characteristics 39 Usefulness 51 7.16 
Suitability 49 6.88 
      100 

The analysis of the obtained absolute weight coefficients was performed using a Pareto diagram, which shows their decreasing values. Twenty percent of the characteristics with the highest weight coefficients of the first level include Purpose, Addressability, Usefulness, and Suitability. This means that these characteristics have the greatest impact on the quality of information (Figure 2).

FIGURE 2

Pareto Diagram.

Studies have shown that the indicators at the second and third levels have similar values, which indicates a fairly balanced system of service delivery. Among the practical measures to improve the system of electronic services in Ukraine, the most appropriate areas follow:

  • Maximize the list of electronic services.

  • Improving the quality of citizens’ advice on administrative services.

  • Conducting monitoring and evaluation of the quality of electronic services, involving public organizations.

  • Standardization of information that underlies public services.

  • Ensuring the quality of electronic services and the efficiency of their provision.

One of the tools for improving the efficiency of providing high-quality electronic services in Ukraine is quality management systems in public authorities. To create a quality management system in the executive branch, it is necessary to define:

Circle of consumers of services (citizens or public groups, enterprises and organizations, state authorities, local self-government, etc.).

A list of services provided to each category of consumers.

  • Customer requirements for each type of service (contained in regulatory documents) and their level of satisfaction, as well as create permanent mechanisms for tracking requirements and satisfaction.

  • The quality criteria for each type of service and the quality standards of the services under these criteria.

  • Mechanisms for informing consumers about the obligations undertaken to provide services (standards for their quality).

  • A scheme for verifying that the services provided meet certain standards and identifying noncompliant services.

  • Actions to identify inappropriate services provided to consumers.

In order to maintain effective feedback from e-services consumers, it is also advisable to introduce online surveys of e-service users on the quality of administrative service delivery. It is also necessary to determine the procedure for establishing the paid nature of services, the implementation of tariff-pricing policy on transparent economic principles for the formation of lists of paid services.

Conclusions

The article discusses the improvement of the system for assessing the quality of the information provided by public authorities to citizens, businesses, and other interested parties as part of the implementation of e-services of “A State in a Smartphone.” The results of this study were the creation of a hierarchical model that allows studying the characteristics of information and determining an integral indicator of information quality based on expert assessments, development of an information quality assessment framework that establishes the procedure for conducting studies, conducting assessment and analysis of information of the “Diia” mobile app, and providing practical recommendations for improving the quality of the information.

The hierarchical model allows evaluating the quality of information by involving experts from different categories of information users. The model has three levels: the first contains nineteen characteristics, the second six, and the third has three characteristics. The study considered information for such categories of services: citizenship and migration, social protection, life safety, professional activity, finance and taxes, education, sports and tourism, culture and religion, protection and security.

The system for assessing the quality of information uses the information quality assessment framework, which provides a selection of experts, determination of information quality weighting coefficients, evaluation of quality characteristics, and calculation of relative quality indicators and the integral indicator of information quality. Using a sample size calculator, the number of documents for each category of services required for a qualitative study was determined.

Selected experts evaluated the characteristics of information (dimensions) to determine the weight coefficient of each dimension, the value of quality indicators for all levels of the model, and the integral indicator of information quality. The obtained average integral indicator of information quality is 0.71, which indicates a sufficiently high level of quality of information circulating in public.

Pareto analysis for absolute weights of first-level information quality characteristics showed that 20% of the characteristics with the highest weight include Purpose, Addressability, Usefulness, and Suitability. These characteristics have the greatest impact on the quality of information.

Studies have shown the importance of factors influencing the development of electronic services in Ukraine. Among them, the most important are Political, Social, and Organizational factors. To improve the functioning of the system of providing electronic services within the project “Country in a smartphone,” the following ways of improvement are proposed: increasing the list of electronic services, improving the system of consulting citizens, monitoring the quality of services, and standardization of information and ensuring the quality of services.

Improving the quality of electronic services in Ukraine is associated with the need to implement such measures as determining the range of consumers of services, developing a list of relevant services, defining quality criteria, and developing quality standards for services; development of mechanisms for informing consumers about the provision of services, determining the method of checking the compliance of services with standards, and determining actions in identifying inappropriate services. Feedback from consumers of electronic services should be an important component of the system.

In further studies, the authors plan to investigate the quality of the information in public administration at national, regional, and local levels, as well as in local governments. This requires increasing the sample of the information provided by the relevant authorities and involving more experts for evaluation. The research is planned to be carried out within the framework of the current system of advanced training of civil servants in Ukraine on the basis of specialized educational institutions.

Ukraine is on the path of integration with European countries, so the interaction of its e-government system and these countries is an important task. The level of digitalization in European countries is much higher than in Ukraine, as evidenced by research conducted within the framework of the United Nations (UN). The UN releases E-Government Surveys every two years. A 2020 study shows that leading European countries in e-government development 2020 are Denmark, Estonia, Finland, Sweden, the United Kingdom of Great Britain and Northern Ireland, and Norway. The most important experience in the formation and development of electronic government for Ukraine is possessed by the United Kingdom of Great Britain, as a country that has come a long way in the development of e-government, and Estonia, as a post-Soviet country. In the future, our plans include a study of the quality of information on government portals in these countries.

Acknowledgment

The article is based on research within the scientific topic “The modernization foundations of sustainable development of the regions of Ukraine in the context of decentralization of power,” conducted at Chernihiv National University of Technology. The state registration number of the topic is 0117U004541. The authors thank the experts who participated in the evaluation of the quality of information during the study and all anonymous reviewers for valuable comments and suggestions.

FOOTNOTES

1.

Acharya et al.

2.

Grossman et al.

3.

Pontones-Rosa et al.

4.

Uyar et al.

5.

The First Political.

6.

Ministry and Committee for Digital Transformation of Ukraine.

7.

Liu et al.

8.

Demoulin and Coussement.

9.

Hao et al.

10.

Kang and Namkung.

11.

Gharib and Giorgini.

12.

Yagci and Das.

13.

Dai et al.

14.

Demoulin and Coussement.

15.

Gharib and Giorgini.

16.

Hao et al.

17.

Kang and Namkung

18.

Klassen et al.

19.

McDowall.

20.

Xiaofeng and Xiang.

21.

Yagci and Das.

22.

Ramos-Lima et al.; Shankaranarayanan and Blake.

23.

Wang, Storey, and Firth.

24.

Ge and Helfert.

25.

Neely and Cook; Madnick et al.

26.

Zhu et al.

27.

Bovee.

28.

Wang and Strong.

29.

Wixom and Todd

30.

Lee et al., “AIMQ: A Methodology for Information Quality Assessment.”

31.

Kenett and Shmueli.

32.

Batini and Scannapieco.

33.

Veiga et al.

34.

Nielsen et al., 2017

35.

Lopes et al.

36.

Wang et al.

37.

Corritore et al.

38.

Lee et al., “What Makes Internet Users Visit Cyber Stores Again? Key Design Factors for Customer Loyalty.”

39.

Floh and Treiblmaier.

40.

Nielsen and Krimmer.

41.

Nielsen, “Usability Inspection Methods.”

42.

Budiu and Nielsen.

43.

Shneiderman, “Human-Centered Artificial Intelligence: Three Fresh Ideas.”

44.

Shneiderman, “Design Lessons from AI’s Two Grand Goals: Human Emulation and Useful Applications.”

45.

Nielsen, “Tutorial: How to Create an Ecosystem and Framework for Digital Public Sector Service Delivery.”

46.

Lee et al.

47.

Pipino, Lee, and Wang.

48.

Eppler.

49.

Britchenko.

50.

Zhao et al.; Rafique et al.; El-Haddadeh et al.

51.

Lee et al.; Pipino et al.

52.

Eppler.

53.

Ehling and Körner.

54.

Mertz et al.

55.

Ghose and Ipeirotis.

56.

Kincaid et al.

57.

Flesch.

58.

Anderson.

59.

Redish.

60.

Redish and Selzer.

61.

Hargis.

62.

Giovanni.

63.

Ehling and Körner.

64.

Broydo and Ilyina.

65.

Britchenko.

66.

Definition of system.

67.

Reverso Dictionary.

68.

Mirzoyan.

69.

Sim et al.

70.

Adler, P and Peter Adler.

71.

Sackman.

72.

Rowe and Wright, “Expert Opinions in Forecasting.”

73.

Rowe and Wright, “The Delphi Technique as a Forecasting Tool: Issues and Analysis.”

BIBLIOGRAPHY

Acharya, Bikram, Jongsu Lee, and HyungBin Moon.
“Preference Heterogeneity of Local Government for Implementing ICT Infrastructure and Services Through Public-Private Partnership Mechanism.”
Socio-Economic Planning Sciences
(
2021
). Accessed
June
24
,
2021
. doi:.
Adler, Patricia A., and Peter Adler. “How Many Qualitative Interviews is Enough? ” In
How Many Qualitative Interviews is Enough? Expert Voices and Early Career Reflections on Sampling and Cases in Qualitative Research
, edited by S. E. Baker and R. Edwards,
8
11
.
Southampton
:
ESRC National Centre for Research Methods, University of Southampton
,
2012
.
Anderson, Jonathan.
“LIX and RIX: Variations on a Little-Known Readability Index.”
Journal of Reading
26
, no.
6
(
1983
):
490
96
.
Batini, Carlo, and Monica Scannapieco.
Data and Information Quality: Dimensions, Principles and Techniques, Data-Centric Systems and Applications
,
Switzerland
:
Springer International Publishing
,
2016
.
Bovee, Matthew.
“Empirical Validation of the Structure of an Information Quality Model.”
Proceedings of the Ninth International Conference on Information Quality, November 5–7
,
2004,
358
72
. Accessed
June
19
,
2020
. https://www.researchgate.net/publication/220918849_Empirical_Validation_of_the_Structure_of_an_Information_Quality_Model, Accessed
June
19
,
2020
.
Britchenko, Gennady.
Improving Information Flows in Enterprise Management
,
Kiev-Donetsk, Ukraine
:
Higher School Publishing Association, Head Publishing House
,
1978
.
Broydo, Vladimir L, and Olga P. Ilyinax.
Computer Architecture and Systems
, 2nd ed.
Petersburg, RF
:
Peter, St
,
2009
.
Brown, Thomas.
“An Experiment in Probabilistic Forecasting.”
R-944-ARPA
,
1972
.
Budiu, Raluca, and Jakob Nielsen.
User Experience for Mobile Applications and Websites: Design Guidelines for Improving the Usability of Mobile Sites and Apps
, 4th ed.
Nielsen Norman Group
,
2015
.
Corritore, Cynthia L., Beverly Kracher, and Susan Wiedenbeck.
“On-Line Trust: Concepts, Evolving Themes, A Mode
. ”
International Journal of Human-Computer Studies
58
, no.
6
(
2003
):
737
58
.
Dai, Wei, Isaac Wardlaw, Yu Cui, and Kashif Mehdi.
“Data Profiling Technology of Data Governance Regarding Big Data: Review and Rethinking
. ”
Information Technology, New Generations. Advances in Intelligent Systems and Computing
48
(
2016
):
439
50
. Accessed
June
19
,
2020
https://www.researchgate.net/publication/301632599Data_Profiling_Technology_of_Data_Governance_Regarding_Big_Data_Review_and_Rethinking Accessed
June
19
,
2020
9
,
2020
.
Decree of the Cabinet of Ministers of Ukraine
.
“On Approval of the Concept for the Development of Electronic Democracy in Ukraine and the Action Plan for its Implementation
. ”
No 797-r of November 8, 2017
. Accessed
June
23
,
2020
. https://zakon.rada.gov.ua/laws/show/797-2017-%D1%80#Text.
Definition of System
.
“Merriam-Webster
. ”
Springfield, MA
. https://www.merriam-webster.com/dictionary/system. , Accessed
January
16
,
2019
.
Demoulin, Nathalie T. M, and Kristof Coussement.
“Acceptance of Text-Mining Systems: The Signaling Role of Information Quality.”
Information & Management
57
, no.
1
(
2020
). Accessed
December
29
,
2021
. https://www.sciencedirect.com/science/article/abs/pii/S0378720617308765?via%3Dihub.
Ehling, Manfred, and Körner Thomas.
“Eurostat Handbook on Data Quality Assessment Methods and Tools
. ”
Wiesbaden
:
Eurostat
,
2007
.
El-Haddadeh, Ramzi, Vishanth Weerakkody, and Shafi Al-Shafi.
“The Complexities of Electronic Services Implementation and Institutionalisation in the Public Sector
. ”
Information & Management
50
, no.
4
(
2013
):
135
43
. Accessed
June
19
,
2020
. https://www.sciencedirect.com/science/article/abs/pii/S0378720613000153.
Eppler, Martin J.
Managing Information Quality: Increasing the Value of Information in Knowledge-Intensive Products and Processes
,
Berlin-Heidelberg
:
Springer Science & Business Media
,
2003
.
The First Political
.
“E-Country: When the Ukrainian State will be in the Smartphone
. ”
2019
. Accessed
June
19
,
2020
. http://1political.com.ua/news/e-kraina-koli-ukrainska-derzhava-opinitsya-v-smartfoni.
Flesch, Rudolph.
“A New Readability Yardstick.”
Journal of Applied Psychology
32
, no.
3
(
1948
):
221
33
.
Floh, Arne, and Horst Treiblmaier.
“What Keeps the E-Banking Customer Loyal? A Multigroup Analysis of the Moderating Role of Consumer Characteristics on E-Loyalty in the Financial Service Industry
. ”
Journal of Electronic Commerce Research
7
, no.
2
(
2006
):
97
110
.
Francisco, Maritza M. C, Solange N. A. Souza, Edit G. L. Campos, and Luiz S. De Souza. “Total Data Quality Management and Total Information Quality Management Applied to Customer Relationship Management.” ICIME, Barcelona, Spain,
2017
,
9
11
. Accessed
June
19
,
2020
. doi:.
Ge, Mouzhi, and Markus Helfert.
“A Review of Information Quality Research–Develop a Research Agenda.”
Proceedings of the 12th International Conference on Information Quality, MIT, Cambridge, MA
,
November
9–11
,
2007
,
76
91
, Accessed
June
19
,
2020
. https://www.researchgate.net/publication/220918901_A_Review_of_Information_Quality_Research_-_Develop_a_Research_Agenda.
Gharib, Mohamad, and Paolo Giorgini.
“Information Quality Requirements Engineering with STS-IQ.”
Information and Software Technology
107
(
2019
):
83
100
. Accessed
June
19
,
2020
. https://www.sciencedirect.com/science/article/abs/pii/S0950584918302313.
Ghose, Anindya, and Panagiotis G. Ipeirotis.
“Estimating the Helpfulness and Economic Impact of Product Reviews: Mining Text and Reviewer Characteristics
. ”
IEEE Transactions on Knowledge and Data Engineering, I
.
23
,no.
10
(
2011
):
1498
512
.
Giovanni, Enrico.
Understanding Economic Statistics: An OECD Perspective
,
Geneva
:
Organisation for Economic Co-operation and Development Publishing
,
2008
.
Grossman, Guy, Melina R. Platas, and Jonathan Rodden.
“Crowdsourcing Accountability: ICT for Service Delivery.”
World Development
112
(
2018
):
74
87
.
Hao, Qian, Xiangyan Shi, and Danlu Bu.
“The Chinese Stimulus Program from 2008 to 2010 and Accounting Information Quality.”
Journal of International Accounting, Auditing and Taxation
33
(
2018
):
64
78
. Accessed
June
19
,
2020
. https://www.sciencedirect.com/science/article/abs/pii/S1061951818302519.
Hargis, Gretchen.
“Readability and Computer Documentation.”
ACM Journal of Computer Documentation (JCD)
24
, no.
3
(
2000
):
122
31
.
Kang, Jee-Won, and Young Namkung.
“The Information Quality and Source Credibility Matter in Customers’ Evaluation Toward Food o2o Commerce.”
International Journal of Hospitality Management
78
(
2019
):
189
98
. Accessed
June
19
,
2020
. https://www.sciencedirect.com/science/article/abs/pii/S0278431918301300.
Kenett, Ron S, and Galit Shmueli.
“On Information Quality (with discussion)
. ”
Journal of the Royal Statistical Society, Series A
177
, no.
1
(
2014
):
3
38
. Accessed
June
19
,
2020
. https://www.galitshmueli.com/system/files/Kenett%20Shumeli%20JRSSA%202014.pdf.
Kincaid, J. Peter, Robert P. Fishburne Jr, Richard L. Rogers, and Brad S. Chissom.
“Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel
. ”
Naval Technical Training Command Millington TN Research Branch no. RBR
. (
1975
):
8
75
.
Klassen, V, Alexander Borek, Ajith Kumar Parlikad, and Robert Kern.
“Quantifying the Business Impact of Information Quality-A Riskbased Approach
. ”
Proceedings of the European Conference on Information Systems
,
2012
. Paper 239. Accessed
June
19
,
2020
. https://www.researchgate.net/publication/288807410_Quantifiying_the_business_impact_of_information_quality_-_A_risk-based_approach.
Lee, Jungwon, Jinwoo Kim, and Jae Yun Moon. “What Makes Internet Users Visit Cyber Stores Again? Key Design Factors for Customer Loyalty. ” In
Proceedings of the Chi 2000 Conference on Human Factors in Computing Systems (2000)
.
Lee, Yang W, Diane M. Strong, Beverly K. Kahn, and Richard Y. Wang.
“AIMQ: A Methodology for Information Quality Assessment
. ”
Information & Management
40
, no.
2
(
2002
):
133
46
. doi:. Accessed
June
19
,
2020
. https://scinapse.io/papers/2138752966.
Liu, Qi, Gengzhong Feng, Xi Zhao, and Wenlong Wang.
“Minimizing the Data Quality Problem of Information Systems: A Process-Based Method
. ”
Decision Support Systems
.
137
:
113381
. (
2020
).
Lopes, Nuno Vasco, and Delfina Sá Soares. “Morten Meyerhoff Nielsen, and António Tavares “Research Gaps on Public Service Delivery.”
Proceedings of the 10th International Conference on Theory and Practice of Electronic Governance
,
March
2017
,
465
74
.
Madnick, Stuart E, Richard W. Wang, Yang W. Lee, and Hongwei Zhu.
“Overview and Framework for Data and Information Quality Research.”
ACM Journal of Information and Data Quality
1
(
2009
):
1
22
. Accessed
June
19
,
2020
. https://dl.acm.org/doi/pdf/10.1145/1515693.1516680?download=true, .
McDowall, R. D.
Data Integrity and Data Governance
.
Croudon
:
CPI Group Ltd
,
2019
.
Mertz, Kevin, Matthew B. Burn, Sara L. Eppler, and Robin N. Kamal.
“The Reading Level of Surgical Consent Forms in Hand Surgery
. ”
Journal of Hand Surgery Global Online
1
, no.
3
(
2019
):
149
53
. doi:https://doi.org/10.1016/j.jhsg.2019.04.003.
“Ministry and Committee for Digital Transformation of Ukraine
,
2019
. Accessed
June
19
,
2020
. https://thedigital.gov.ua/news/na-testuvannya-mobilnogo-zastosunku-diya-zareestruvalosya-bilshe-58-tis-ukraintsiv.
Mirzoyan, Nataliya V.
“Analysis of the Application of the Comparative Approach in Accordance with the Federal Standard of Assessment “Real Estate Valuation (FSO No. 7
. ”
Property Relations in the RF
6
(
2015
):
68
79
. Accessed
June
19
,
2020
. https://cyberleninka.ru/article/n/analiz-primeneniya-sravnitelnogo-podhoda-v-sootvetstvii-s-federalnym-standartom-otsenki-otsenka-nedvizhimosti-fso-7/viewer.
Neely, M. Pamela, and Jack Cook.
“A Framework for Classification of the Data and Information Quality Literature and Preliminary Results (1996–2007)
. ”
Proceedings of the Americas Conference on Information Systems (AMCIS)
,
Toronto, CA
,
2008
. Accessed
June
19
,
2020
. https://pdfs.semanticscholar.org/c0eb/73e7626e209bf83066c7a703846881acfeab.pdf.
Nielsen, Jakob.
Usability Inspection Methods
. 1st ed.
New Jersey
:
Wiley
,
1994
.
Nielsen, Morten Meyerhoff. “Tutorial: How to Create an Ecosystem and Framework for Digital Public Sector Service Delivery.”
Sixth International Conference on eDemocracy & eGovernment (ICEDEG)
,
2019
,
6
9
.
Nielsen, Morten Meyerhoff, Nuno Ramos Carvalho, Linda Gonçalves Veiga, and Luís Soares Barbosa. “Administrative Burden Reduction Over Time: Literature Review, Trends and Gap Analysis. ”
Proceedings of the 10th International Conference on Theory and Practice of Electronic Governance
,
March
2017
,
140
48
. doi:.
Nielsen, Morten Meyerhoff, and Robert Krimmer.
“Reuse of Data for Personal and Proactive Service: An Opportunity Not Yet Utilized”
,
CeDEM15
:
Conference for E-Democracy and Open Government, MV-Verlag
,
2015
.
Order of the Cabinet of Ministers of Ukraine. “ On Approval of the Concept of e-government development in Ukraine.” No.649-r of September 20, 2017
. Accessed
June
23
,
2020
. https://zakon.rada.gov.ua/laws/show/649-2017-%D1%80#Text.
Pipino, Leo L, Yang W. Lee, and Richard Y. Wang.
“Data Quality Assessment.”
Communications of the ACM
45
, no.
4
(
2002
):
211
18
. Accessed
June
19
,
2020
. https://dl.acm.org/doi/10.1145/505248.506010.
Pontones-Rosa, Carolina, Pérez-Morote Rosario, and Santos-Peñalver Jesús F.
“ICT-Based Public Policies and Depopulation in Hollowed-Out Spain: A Survey Analysis on the Digital Divide and Citizen Satisfaction
. ”
Technological Forecasting and Social Change
169
(
2021
):
120811
.
Rafique, Irfan, Philip Lew, Maissom QanberAbbasi, and Zhang Li.
“Information Quality Evaluation Framework: Extending ISO 25012 Data Quality Model
. ”
World Academy of Science, Engineering and Technology International Journal of Computer, Electrical, Automation, Control and Information Engineering
6
, no.
5
(
2012
):
568
73
. Accessed
June
19
,
2020
. https://publications.waset.org/9538/information-quality-evaluation-framework-extending-iso-25012-data-quality-model.
Ramos-Lima, Luis Francisco, GastaudMaçada Antonio Carlos, and Lilia Maria Vargas. “Research into Information Quality: A Study of the State-of-the-Art in IQ and its Consolidation.”
Proceedings of the 11th International Conference on Information Quality, MIT, Cambridge, MA
,
November
10–12
,
2006
.
146
58
. Accessed
June
19
,
2020
. https://www.researchgate.net/publication/220918661_Research_into_Information_Quality_A_Study_of_the_State_of_the_Art_in_IQ_and_Its_Consolidation.
Redish, Janice C.
“Readability Formulas Have Even More Limitations than Klare Discusses.”
ACM Journal of Computer Documentation (JCD)
24
, no.
3
(
2000
):
132
37
.
Redish, Janice C, and John L. Selzer.
“The Place of Readability Formulas in Technical Communication.”
Technical Communication
32
, no.
4
(
1985
):
46
52
.
Reverso Dictionary
.
“Definition of synergistic approach
. ”https://dictionary.reverso.net/english-definition/synergistic+approach.
Rowe, Gene, and George Wright.
“The Delphi Technique as a Forecasting Tool: Issues and Analysis.”
International Journal of Forecasting
15
, no.
4
(
1999
).
Rowe, Gene, and George Wright. “Expert Opinions in Forecasting. Role of the Delphi Technique.”
Principles of Forecasting: A Handbook of Researchers and Practitioners
, edited by
Armstrong
.
Boston
:
Kluwer Academic Publishers
,
2001
.
Sackman, Harold.
“Delphi Assessment: Expert Opinion, Forecasting and Group Process.”
R-1283-PR
,
April
1974
.
Shankaranarayanan, Ganesan, and Roger Blake.
“Data and Information Quality: Research Themes and Evolving Trends.”
Proceedings of the Twenty-First Americas Conference on Information Systems, Fajardo, Puerto Rico
,
August
13–15
,
2015
. Accessed
June
19
,
2020
. https://pdfs.semanticscholar.org/19a6/33e02dad56b3702017d4cae0a789ae8391d8.pdf?_ga=2.169905827.24098602.1592767778-234589458.1592767778.
Shneiderman, Ben.
“Design Lessons from AI’s Two Grand Goals: Human Emulation and Useful Applications.”
IEEE Transactions on Technology and Society
1
, no.
2
(
2020
):
73
82
.
Shneiderman, Ben.
“Human-Centered Artificial Intelligence: Three Fresh Ideas.”
AIS Transactions on Human-Computer Interaction
12
, no.
3
(
2020
):
109
24
.
Sim, Julius, Benjamin Saunders, Jackie Waterfield, and Tom Kingstone.
“Can Sample Size in Qualitative Research be Determined a Priori?
International Journal of Social Research Methodology
21
, no.
5
(
2018
):
619
34
.
Uyar, Ali, Khalil Nimer, Cemil Kuzey Muhammad Shahbaz, and Friedrich Schneider.
“Can E-Government Initiatives Alleviate Tax Evasion? The Moderation Effect of ICT
. ”
Technological Forecasting and Social Change
.
166: 120597
.(
2021
).
Veiga, Linda, Tomasz Janowski, and Luis Soares Barbosa. “Digital Government and Administrative Burden Reduction. ”
Proceedings of the 9th International Conference on Theory and Practice of Electronic Governance
,
March
2016
,
323
26
. doi:.
Wang, Michael Chih-Hung, Edward Shih Tse Wang, Julian Ming-Sung Cheng, and Alex Fei Long Chen.
“Information Quality, Online Community and Trust: A Study of Antecedents to Shoppers’ Website Loyalty
. ”
International Journal of Electronic Marketing and Retailing
.
2
,no.
3
(
2009
):
203
209
.
Wang, Richard Y., and Diane M. Strong.
“Beyond Accuracy: What Data Quality Means to Data Consumers.”
Journal of Management Information Systems
12
, no.
4
(
1996
):
5
33
. Accessed
June
19
,
2020
. http://mitiq.mit.edu/Documents/Publications/TDQMpub/14_Beyond_Accuracy.pdf.
Wang, Richard Y., Veda C. Storey, and Christopher P. Firth.
“A Framework for Analysis of Data Quality Research.”
IEEE Transactions on Knowledge and Data Engineering
7
(
1995
):
623
40
. Accessed
June
19
,
2020
. https://www.academia.edu/28121620/A_framework_for_analysis_of_data_quality_research.
Wixom, Barbara H., and Peter A. Todd.
“A Theoretical Integration of User Satisfaction and Technology Acceptance.”
Information Systems Research
16
, no.
1
(
2005
):
85
102
. doi:.
Xiaofeng, Meng, and Ci Xiang.
“Big Data Management: Concepts, Techniques and Challenges.”
Journal of Computer Research and Development
50
,no.
1
(
2013
):
146
69
. Accessed
June
19
,
2020
. http://crad.ict.ac.cn/EN/abstract/abstract715.shtml.
Yagci, Ismail Art, and Sanchoy Das.
“Measuring Design-Level Information Quality in Online Reviews.”
Electronic Commerce Research and Applications
30
(
2018
):
102
10
. doi:.
Zhao, Yi, Jiayu Gong, Yun Hu, Zhenyu Liu, and Lizhi Cai.
“Analysis of Quality Evaluation Based on ISO/IEC SQuaRE Series Standards and Its Considerations.”
Proceedings of the IEEE/ACIS 16th International Conference on Computer and Information Science (ICIS)
,
Wuhan, China
,
May
24–26
,
2017
. doi:.
Zhu, Hongwei, Stuart Madnick, Yang Lee, and Richard Wang.
Data and Information Quality Research: Its Evolution and Future. Computing Handbook
, 3rd ed.
Boca Raton, FL
:
Taylor & Francis Group, LLC
,
2014
.

Author notes

Editors’ Note: This article by Igor Oliychenko and Maryna Ditkovska on Ukraine’s new “State in a Smartphone” service was accepted and reviewed prior to the incursion of Russian troops into Ukraine and the conflict and civic disruption that followed.

This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.