Evidence-based Software Engineering
Why evidence is important in software engineering
Initially it is worth while considering why evidence would be beneficial to software developers, users and other stakeholders e.g. public purchasing bodies, certification bodies and the general public. EBSE is potentially important because of the central place software intensive systems are starting to take in everyday life. For example, current plans for advanced life-critical system such as drive-by-wire applications for cars and we are able medical devices have the potential for immense economic and social benefit but can also pose a major threat to industry, to society and to individuals. If systems are reliable, usable and useful, the quality of life of individual citizens will be enhanced. However, there are far too many examples of systems that have not only wasted large amounts of public money but have also caused harm to individual citizens (e.g. the automated command and control system for the London Ambulance Service). Individual citizens have a right to expect their governments to properly administer tax revenues used to commission new software systems and put in place controls to minimise the risk of such systems causing harm. There are many strategies to improve the dependability of software involving the adoption of 'better' software development procedures and practices. At a high level, the Capability Maturity Model and SPICE suggest procedures for improving the software production process. In addition, the professional bodies are establishing procedures for certification of individual software engineers. However, the high level process and the individual engineers are constrained by the specific technologies (methods, tools and procedures) they use. In most cases software is built with technologies for which we have insufficient evidence to confirm their suitability, limits, qualities, costs and inherent risks. Thus, it is difficult to be sure that changing software practices will necessarily be a change for the better. It is possible that EBSE can provide the mechanisms needed to assist practitioners to adopt appropriate technologies and to avoid in appropriate technologies.
The Goal Of EBSE
The goal of evidence-based medicine (EBM) is 'the integration of best research evidence with clinical expertise and patient values'. By analogy, we suggest that the goal of evidence-based software engineering (EBSE) should be:to provide the means by which current best evidence from research can be integrated with practical experience and human values in the decision making process regarding the development and maintenance of software. Thus EBSE would provide:
Steps that are needed to practice evidence-based medicine. Although there are some detailed medical references, it is easy to reformulate the steps to address evidence-based software engineering. In fact, we would hazard a guess that at least part of the attraction that evidence-based medicine has for other disciplines is the ease with which the basic steps can be adapted to other fields. However, it is important to remember that even if high-level process steps for evidence-based practice appear to be similar for medicine and software engineering, this does not guarantee that the underlying scientific, technological and organizational mechanisms that support evidence-based medicine apply to evidence-based software engineering.For this reason we consider each step in more detail below. The first point to note is that Sackettet al. consider EBM from the viewpoint of an individual medical practitioner who needs to decide how to treat a particular patient exhibiting a particular set of symptoms. For EBSE our viewpoint is likely to be somewhat different. In software engineering organizations, individual developers seldom have the option to pick and choose the technologies they are going to use. Technology adoption is often decided either by project managers on a project by project basis, or by senior managers on a departmental or organizational basis. Furthermore, in software engineering, our concern is not usually the specific task to which the technology is applied but the outcome of the project of which the task is a part Evidence-Based Software Engineering for Practitioners.
The Aim And Methodology Of EBSE
EBSE aims to improve decision making related to software development and maintenance by integrating current best evidence from research with practical experience and human values. This means we don't expect a technology to be universally good or universally bad, only more appropriate in some circumstances and for some organizations. Furthermore, practitioners will need to accumulate empirical research about a technology of interest and evaluate the research from the viewpoint of their specific circumstances. This aim is decidedly ambitious, particularly because the gap between research and practice can be wide. EBSE seeks to close this gap by encouraging a stronger emphasis on methodological rigor while focusing on relevance for practice. This is important because rigor is necessary in any research that purports to be relevant. Moreover, because most SE research hasn't influenced industrial practice,5 there's also a pressing need to prevent SE research from remaining an ivory tower activity that emphasizes academic rigor over relevance to practice. So, although rigor is a necessary condition for relevant SE research, it isn't sufficient. Medical evidence is based on rigorous studies of therapies given to real patients requiring medical treatment; laboratory experiments are not considered to provide compelling evidence. This implies that SE should n't rely solely on laboratory experiments and should attempt to gather evidence from industrial projects, using observation studies, case studies, surveys and field experiments. These empirical techniques don't have the scientific rigor of formal randomized experiments, but they do avoid the limited relevance of small scale, artificial SE experiments. Furthermore, there are substantial problems with accumulating evidence systematically and not only because accumulating evidence from different types of studies is difficult. A specific challenge in practicing EBSE is that different empirical studies of the same phenomenon often report different and sometimes contradictory results. Unless we can understand these differences, integrating individual pieces of evidence is difficult. This point to the importance of reporting contextual information in empirical studies to help explain conflicting research results. EBSE involves five steps:
However, EBSE isn't a standalone activity. Much of what’s needed to practice EBSE already exists in the concept of technology transfer and software process improvement. SPI involves several steps (each researcher and consultant has his or her own view of how many)—for example
Thus, EBSE provides mechanisms to support various parts of SPI. In particular, EBSE focuses on finding and appraising an appropriate technology for its suitability in a particular situation. This is an area where SPI is usually Supplementary Guidelines, Assessment Scheme and evidence-based evaluations of the use of Evidence. Based Software Engineering Overview to Evidence Based Software Engineering
Evidence Based Software Engineering (EBSE) has recently been proposed as a methodology to help practitioners improve their technology adoption decisions given their particular circumstances. In simple terms, EBSE first recommends the conduct of a Systematic Literature Review (SLR) to identify and appraise evidence relevant to the problem or technology under consideration. EBSE then recommends that the evaluators integrate the results of the SLR with (their) practical experience, circumstances and (professional) values.
The Five Steps Of EBSE
During the first step of EBSE, the evaluator constructs an EBSE question comprising five components: a statement of the intervention (i.e. the instrument of change, such as the introduction of a new tool) of interest, a statement of a baseline against which the performance of the intervention is compared, a statement of the users of the intervention (e.g. the users' experience and skills), a statement of the situation within which the users apply the intervention (e.g. the type of project, type of application) and a statement of the particular outcome that intervention is expected to change and ideally improve. Relating this back to the Borland advert, a simple example of an EBSE question is:
Does the Borland CaliberRM Requirements Management Tool [intervention] more effectively trace requirements [outcome], when used by experienced requirements analysts [users] in the development of a large software system [situation], when compared to the use of no formal requirements management tool [baseline]? A formal EBSE question is typically constrained to compare one intervention against one baseline e.g. comparing two RMTs, or comparing one RMT against not using any RMT.
In step 4, the evaluator attempts to make a recommendation on whether or not to adopt the intervention and this recommendation is made on the basis of the evaluator's conclusions from steps 3 and 4. It may be that the evidence is contradictory or inconclusive and a recommendation cannot be made.
The EBSE methodology was initially developed with reference to the widely-adopted Evidence Based Medicine (EBM) methodology, but more recently EBSE has drawn on other evidence based disciplines. Kitchenham et al. in particular recognise the problems with ‘transplanting’ an EBM-like methodology into software engineering.
Dybå et al. and Jørgensen et al. report anecdotal evidence on the success of teaching EBSE to undergraduate students at a Norwegian university and we have reported on two studies of the use of EBSE by students at a UK university. Janzen and Ryoo propose the development and population of the SEEDS1 repository (http://www.evidencebasedse.com), a web database of EBSE studies critically analyzed by students. Such a database would complement, for example, the NSF Center for Empirically- Based Software Engineering database (www.cebase.org), the EBSE database maintained at Durham University (http://www.dur.ac.uk/ebse/) and the US Department of Defence’sBest Practices ClearingHouse. EBSE has been the subject of two workshops, Realizing Evidence Based Software Engineering (REBSE), both co-located with the International Conference on Software Engineering (ICSE).
The indications are that EBSE has been used almost exclusively by researchers, with some application of EBSE by undergraduate students. We are not aware of any published, empirically-based evidence of professional software practitioners directly using EBSE, or SLRs, to make technology adoption decisions in their projects or organisations.
What constitutes 'best evidence'?
A preliminary empirical investigation of the use of evidence based software engineering by under-graduate students
A BRIEF OVERVIEW OF EVIDENCE-BASED SOFTWARE ENGINEERING
Since the emergence of software engineering in the late 1960's, researchers and practitioners have recognized that most of software engineering research has little practical justification. This is manifested by the fact that very few of the results from software engineering research lead to an uptake of tools or change of practice in industry. There has been a recurring debate over how to make software engineering research more accessible and, indeed, practical to software practitioners. Much of this debate has advocated empirical work. EBSE 'aims to improve decision making related to software development and maintenance by integrating current best evidence from research with practical experience and human values.' As a methodology, it is intended to direct research to the needs of industry, help practitioners to make more rational decisions about technology adoption, enable better choice of development technologies and increase the acceptability of software intensive systems that interface with users. The methodology involves five steps:
Make the point that EBSE complements software process improvement (SPI) and that EBSE is particularly beneficial in an area where SPI is traditionally weak i.e. finding and appraising an appropriate technology, prior to inclusion of that technology in an SPI programmed. A related distinction between EBSE and SPI is that EBSE concentrates on improving practitioners' use of existing findings in order to make decisions. In contrast, SPI concentrates on the introduction, assessment and management of a new intervention. It follows that EBSE is not intended as an approach for generating new evidence. Rather, the focus of EBSE is on gathering and appraising existing evidence. As suggested above, EBSE places a heavy emphasis on the use of systematic reviews, which are a means to provide a fair evaluation of a phenomenon of interest by using a trustworthy, rigorous and auditable methodology. EBSE can be located within a broader context of software engineering research that has:
In this tutorial, we’ll start to review the EBSE methodology. We will use the primary article on EBSE that has been published for practitioners. I plan to consider stages 1 and 2 of EBSE today and, if time permits, to also consider stage 3. Next week we’ll look at stages 3, 4 and 5 of EBSE. I need to emphasize that EBSE is not an ‘easy’ methodology to use. (There are a number of places in the article where the authors state that the stage or procedure isn’t easy.) The methodology tries to encourage both rigors of analysis and also relevance of analysis to a particular situation.
A guide to the tutorial
Teaching Evidence-Based Software Engineering to University Students
Graphically displays the elements of Toulmin's model. The primary elements of an argument, according to Toulmin's mode, are in bold letters and the secondary elements in italic Toulmin's model of argumentation can be viewed as a layout of argument. This layout of argument, we believe, is useful for the students as a mental model when evaluating and/or constructing arguments. In particular, we find the model useful during the exercises on the evaluation of arguments. The checklist in Appendix is based on Toulmin's model of argumentation. We recommend that the student start with the identification of the claims or conclusions made by the authors. These are normally found in the conclusion section of the papers or in the abstract, but may be found other places as well. Poor papers may, in fact, have no explicit claims at all. The students are requested to evaluate the claim, e.g., whether the claim is circular or vague. We also ask the students to identify the qualifiers, i.e., statements about the strength of the claim and the reservations, i.e., statements about the limitations of the claim. These are important when later evaluating the relevance of the evidence and the connection between evidence and claim. For example, a claim that is qualified with 'this weakly indicates a cause-effect relationship' should be evaluated differently from the claim 'there is a cause effect relationship.' Then, we ask the student to look for the data, i.e. the evidence supporting the claim. In particular, we ask them to evaluate the relevance of the evidence. We frequently find that the students are surprised by how little relevant evidence a lengthy software engineering paper contains. Finally, we ask the students to look for the warrant, i.e., the supporting connection between the data and the claim. This is frequently the most difficult part of the evaluation of the argumentation, where the critical appraisal ability and analytical skill of the students is most important. The students are requested to evaluate the degree to which the relevant data supports the claim. To support this evaluation step, the students are taught guidelines for how empirical studies should be conducted  and general guidelines on ethical issues pertaining to such studies when conducted on human subjects. The warrants may have a backing, i.e., an argument that supports a connection of confirmation or deduction between the data and the claim. When it is not obvious that the connection between data and claim is valid (or invalid), we ask the students to search for elements that the authors use to support it (the backing). This may, for example, consist of analytical argumentation or evidence supporting the specific interpretation of data conducted by the authors. We have experienced that familiarity with the model just outlined does more than change how the students evaluate argumentations. It also seems to lead to a more efficient and critical reading of software engineering texts that may be useful for the student in other contexts. For example, we have observed that most of them replace the mechanical reading of papers from the first to the last page, to a more information seeking, flexible reading strategy. If the information about potential vested interests and the claims are found in at the last page of a paper, it is natural to start the reading there. In addition, more knowledge about the elements of argumentation may have enabled the students to improve the quality of their own arguments.
Fundamentals Of Internet | Data Processing Concept | Anatomy Of A Digital Computer | Computer Graphics | Assignment Algorithm | Assignment Homework | Computer Assignment Answers | Computer Assignment Questions | Computer Assignment Solution | Computer Engineering Assignment | Computer Science Algorithm | Online Tutoring