Alexopoulos Charalampos et al.
• Technical complexity: Despite the technical heterogeneity of the various public sector data sources, ease of data discovery and retrieval by the user is a fundamental requirement.
• Organizational barriers: Many EU member state governments and wider public sector organisations still view publishing PSI as an additional burden to their daily working routine, not recognizing the benefits it provides.
These considerable issues are confirmed by literature( Janssen 2012; Zhang 2005; Zuiderwijk 2012). Taking into account the big investments of governments for developing and operating PSI e‐infrastructures, and at the same time the above‐mentioned inherent difficulties of their exploitation, it becomes absolutely necessary to conduct a systematic evaluation of such PSI e‐Infrastructures aiming at a better understanding and assessment of value they generate( Alexopoulos 2012). Furthermore, the gradual emergence of a second generation of more advanced PSI e‐infrastructures, which are influenced by the principles of the web 2.0 paradigm( with users having a stronger role, which is not limited to consuming passively content, but also increasingly includes commenting, rating and improving or adapt it to specialized needs, and also creating their own content – see section 3), and offer new types of value, makes the systematic evaluation of them even more necessary.
However, a structured and comprehensive evaluation framework for PSI initiatives and projects is missing. This paper contributes to filling this gap, by making the following contributions.
• It develops an integrated evaluation framework scoping the overall impact of an open public data e‐ infrastructure,
• which includes an evaluation model with measurable evaluation dimensions and criteria, taking into account findings from previous public projects’, information systems and e‐services evaluation research, research on technology acceptance and information systems( IS) success models, as well as the objectives and capabilities of the traditional PSI e‐infrastructures following the web 1.0 paradigm and the emerging advanced ones influenced by the web 2.0 paradigm.
• and also a comprehensive evaluation procedure for using the evaluation model, which combines the detailed evaluation it provides with a higher level evaluation( scoping to measure general impact), based on both qualitative and quantitative methods in order to get deeper meaningful insights,
• and finally uses the proposed evaluation framework( both the evaluation model and procedure) for the evaluation of the PSI e‐Infrastructure developed in the European project ENGAGE and presents some first results.
The current paper is structured as follows. In the following section the background of the proposed evaluation framework is presented. Subsequently, in section 3 the features of an advanced PSI e‐Infrastructure are presented. Section 4 introduces the evaluation framework and describes the formation of the corresponding evaluation model, as well as, of the evaluation procedure for using it is defined. Thereafter, section 5 presents the results from their use for the pilot evaluation of the first version of ENGAGE( 1.0) PSI e‐infrastructure and finally, in section 6, the conclusions and further research directions are proposed.
2. Background
In this section, is presented the background on which our evaluation framework has been based, which includes previous research on the evaluation of public projects, and on the evaluation of information systems( basic concepts and methods, technology acceptance and IS success models and e‐services evaluation).
The Organisation for Economic Co‐operation and Development( OECD 1998, p. 3) defines public projects’ evaluations as“ analytical assessments addressing results of public policies, organisations or programmes, that emphasise reliability and usefulness of findings. Their role is to improve information and reduce uncertainty; however, even evaluations based on rigorous methods rely significantly on judgement. A distinction can be made between ex‐ante evaluations( or policy reviews) and ex‐post evaluations”. Since the beginning in the 1970s, literature classifies types of evaluation by their specific aim and therefore distinguishes between formative evaluation( conducted during a project in order to identify strengths and weaknesses and inform / guide following stages) and summative evaluation( guided after the end of the project in order to assess its degree of success)( Neumann 2002). Many different methods are used in public projects’ evaluation for information gathering and analysis( Bortz 2002), such as: Quantitative surveys, Web analytics, Usability testing, Focused( semi structured) interviews, In‐depth interviews, Collective semi‐structured interviews. The examination of the existing literature regarding public projects’ evaluation enabled us to design the proposed
103