It is often tempting to launch straight into an evaluation without taking the time to plan first. Planning is a crucial part of the process. The APCRC Checklist has been designed to help with the process of carefully planning your evaluation. It will enable you to become clear about what you intend to do, what data you need to collect and how you will undertake the analysis. It will enable you to think about creating a steering or advisory group of stakeholders. As well as seek advice from the right people at the outset of the process, agreeing roles and responsibilities. The key questions to ask are:
- what are we trying to achieve?
- is the role of the group to advise or steer the project?
- who will draft the plan (or protocol)?
- who and how will the data be collected?
- who and how will the data be analysed?
- who will write up the final report?
- what is the time frame?
Once these questions have been answered then a protocol can be drafted using the APCRC Template for an Evaluation Protocol as a guide. The protocol is an important document and will guide you through the whole project, and can be referred to for clarification as the project progresses.
Evaluation design and methods
Designing the methods that you will use for your evaluation can seem quite daunting, but it need not be complicated or challenging. What is important is that the methods and tools you use are appropriate for your evaluation questions. The approach you take will, to a large extent, be determined by the aims and objectives of your evaluation.
Quantitative and qualitative methods represent different ways data can be collected and used to inform your evaluation.
- Quantitative approaches give numerical results. For example, the percentage of participants still exercising six months after completing a cardiac rehabilitation programme. Quantitative methods are most often used to assess a project’s outcome.
- Qualitative approaches use narrative or descriptive data rather than numbers. For example, a description of the views and attitudes of those completing a cardiac rehabilitation programme, and their thoughts on how it could be improved. Qualitative methods are most often used in a formative evaluation to aid a project’s planning stage and when assessing participants’ needs.
Both qualitative and quantitative methods can be appropriately used alone or in combination in evaluation. For example if you wish to evaluate diverse or complex aspects of the service it will probably be best to consider using multiple methods within your evaluation. Remember, what is crucial is that the method chosen does actually measure what it is intended to and answers the question you are asking.
It is also important to make a distinction between evaluation and monitoring. Monitoring is different and is essentially a check of the extent to which a project or service is proceeding according to plan. For example, looking every week to see if there is a drop-off in attendance at a particular clinic or session. It is really a subset of evaluation and should not be used as a substitute for a full evaluation. Please refer to our tool to help you determine whether your project is research, evaluation or clinical audit.
The scale of an evaluation can vary greatly from small scale in-house evaluations to large scale complex or risky evaluations commissioned externally. For guidance on the level of evaluation recommended for projects of differing complexity please see the APCRC scoring matrix. You can use this grid to help you decide what level of evaluation would be best suited to meet the needs of your project. For example your evaluation may require a small in house evaluation, a medium scale evaluation with external support, or if a major piece of work it may need a large scale, externally commissioned and funded evaluation.
The following PDF documents have been produced to help you plan your evaluation, decide which methods are best suited to your question and ensure that the findings make a difference to service delivery and patient care.