How will I know if my BI project is successful?
Metrics for Success
A truly successful BI project engages people outside your field to understand the relevance of your research. BI activities are an excellent mechanism to not only engage the public, but also to demonstrate accountability with public funds.
How do you document and tell the story of your success?
Just as you collect data in your scientific research, you need to collect data on the efficacy of your BI project which means you need to plan for a project evaluation. Depending on your needs and budget, an evaluation can be simple or sophisticated and you can do it on your own, or work with evaluation consultants within your institution (if your institution has in-house expertise), or hire a professional evaluator to help you. For your proposal, you will not need in depth detail about your evaluation plans but you should demonstrate that you have considered how you plan to evaluate the BI component of your project. The evaluation results will help you share the success and/or offer guidance for future iteration.
How much is enough evaluation for my project?
The available budget is often the driver. If you have access to "in-house expertise", that can be useful for internal dot connecting between existing efforts and partners, but also to save on overhead and/or subcontract fees. It can be a helpful exercise to conduct a survey of which metrics your institution or your partners already gather to determine if there are data already collected or adjacent to metrics you wanted to measure which can save time and money. The project evaluator can help guide this process. If you are not planning to hire an external project evaluator, perhaps a partner organization can co-develop the evaluation plan utilizing parameters of value to their organization.
How much will the evaluation cost?
See above. The cost of a project evaluation depends on which parameters you intend to measure. For more information, check out our budgeting tips.
How do I find an evaluator (internal vs. external)?
Check with your home institution for in-house expertise or suggestions for evaluators. If there are none, considered working with your partner's evaluators. If they do not have one, consider looking at the resources listed through NSF and NSF programs like ATE, CAISE, or ITEST.
In your proposal:
A project evaluation plan outlines how you will systematically collect data from your audience to address questions such as:
- Was this BI project successful? To what extent did the project meet stated goals and objectives?
- Was this BI project valuable to your audience? Were your BI activities appropriate for them?
- Did your audience change as a result of this BI project, and in what ways (i.e. knowledge, skills, or behavior)?
For background on the different types of evaluation, you can check out the video COSEE NOW Intro to Program Evaluation
Reviewers will be asking:
- Does the plan incorporate a mechanism to evaluate success?
- Is there a plan in place to measure the project outcomes?
Example 1: Our evaluator Dr. Jane Doe, will conduct a formative and summative evaluation of this project. Many of the students selected for this program are potential first-generation college graduates (Kids Count Data Center, 2017; National Center for Children in Poverty. 2017). In addition, the education and experiences offered by rural school districts are often limited, resulting in inequities in opportunities for students and teachers (Palardy, 2008).
Our overarching objective is to enhance youths' positive identity as a scientist and to offer authentic insight into how scientists work.
Participating youth will:
- Engage in an authentic hands-on experiences in nanotechnology
- Interact with scientists to contribute to building a science identity
- Demonstrate understanding of 3-5 careers in nanotechnology (varying levels technician to PhD)
Instructors involved will:
- Gain access to authentic research and lesson materials
- Develop skills in how to use real world science in their instruction
- Engage in scientist-instructor partnerships
Survey data collected will help inform decisions regarding improving the project for youth and the final survey will determine how youth have changed over time. At the end of the project the summative evaluation will focus on changes to instructors' attitudes and the impact of the project on their teaching.