You will need to measure the impact your project has, or has not made, against your original intent. Reflect on the social values and ethical principles you have or have not promoted. Finally you will need to think about how to follow up, so your sensing project can continue to create value for your city.
In the end, sensing projects should be about co-creating social impact with the public. You need to show quantitative and qualitative results in the right ways to keep momentum, your stakeholders involved, and the public engaged.
Now is the time to reflect on your sensing project. Among the many questions and considerations that will help you at this stage here are three central themes.
What was the impact on public space and peoples lives?
Did I conduct my project in responsible and ethical ways?
How can I improve upon my experience and continue or conclude?
Whether your project comes to an end or continues, this is the time to think critically about the impact you have made with sensing project in your city. Success or failure can be framed in many ways.
Reflecting Social Impact
Evaluating Ethical Performance
Visit the video library for expert interviews and advice about Impact and Analysis. In this section we’ll be speaking with experts:
Ido Nap – Program Manager Sensing Dutch Police
Paul Manwaring – Founder IoT Living Lab and Co-founder CITIXL
Making people aware and getting them involved to the point where they want to participate is difficult so you have to show quantitative and qualitative results in the right way to keep the public engaged. At this point there are also operational, tactical and strategic considerations that relate to accountability issues and follow-up. For example re-establishing ownership, governance, escalation paths, roles, and responsibilities among other considerations such as:
- Can you continue by designing interventions and involving the public?
- Are you measuring changes, results, and social impact?
- Are you being transparent and publishing the data and results with visualisations and engaging stories?
- Have you been speaking with the public to generate a “data plus dialogue”?
- Are you communicating your project, creating awareness and accountability?
- How can you reflect on the impact you have made in the context of your project goals as well as in an ethical context taking into account social values?
- Is it possible to keep momentum and enable a continuous discussion?
- Who will now own this project and how will they be accountable?
- Have you established a follow-up plan and strategy?
6.2 Project Retrospective
Now is a good time to reflect on your experience, look at the original intent, review your goals and revisit the Decision Canvas you worked on in the beginning of your journey to responsible and ethical sensing. Project post-mortems and assessing lessons learned is important but think about the unique aspects of sensing and your project in a broader sense. It’s not just about measuring performance against KPIs, it’s also about how you did or did not achieve your goals in responsible and ethical ways. Here are some suggestions to help you assess your project’s impact:
- Revisit your original Decision Canvas with project stakeholders.
- Think about Lessons Learned. Document and share your experience.
- Measure the changes you have made with an impact analysis report to quantity the before and after of your project and any interventions you conducted.
- Publish the results and tell your story. It is important to be transparent and share your results.
No doubt your organisation will have its own way of reflecting on a project. We hope to add some value at this stage by emphasising the ethical aspects specifically related to sensing projects. Now would be a good time to (re) visit the Responsible Sensing Lab and read about the projects: Human Scan Car, Transparent Charging Station, and Shuttercam. Three well documented projects you can refer to.
6.2.1 Reflecting: Social and Ethical Impact
It is difficult to measure success in a sensing project you may or may not have achieved the goals you established. You may have had some serious problems or surprising results but what is most important is understanding how your project impacted the public. Did you change perceptions or behaviour and what does this mean? After evaluating the quantitative results of your project you should consider engaging the public again to get some feedback and gather the qualitative data about social impact.
6.2.2 Evaluating Ethical Performance
Responsible and ethical sensing as a practice is just now beginning to take shape. If we design and implement our projects keeping social values and ethical principles in mind we are pioneering important work. The impact we create in our cities is much more than solving problems that have to do with how people move in public spaces but how we approach solving these problems with privacy and dignity at the same time.
Now is a good time to hold an Ethical Performance Workshop with your project stakeholders. You can revisit the Applying an Ethical Framework and Measuring Ethical Principles in Step 3 or visit some of the data ethics links below to help you think about how to evaluate your ethical performance:
- Privacy by Design
- Tada! Data Disclosed
- Responsible Sensing Lab
- Cities for Digital Rights
- Value sensitive design
- Utrecht Data School
Recreational Crowd Study
For a specific example of a Tada Data Ethics Workshop to evaluate ethical performance, download this report from the Recreational Crowd Study conducted by CITIXL in collaboration with Marineterrein Amsterdam, City of Amsterdam CTO office and Tada.
Project stakeholders should be evaluating the ethical performance for each of their experiments. Teams are free to define their own values and principles, but most importantly see how they can continually improve on responsible sensing during the project life cycle. In the illustration below, you see a comparison of how teams ranked themselves on a hypothetical research project (based on the Tada ethical framework).
Cafe owners using Computer Vision to measure social distancing on their terrace.
Business district impact analysis of 3D Sensors in a street during lockdown.
Museum managers using WiFi Sniffers for smart ticketing to improve waiting times.
Health department using MM Waves to detect after hour activities of playgrounds.
Campus security using Sound Sensors to classify sounds like guns or breaking glass.
Facility managers using Motion Sensors to determine safe capacity of student assembly.
At this point there are also operational, tactical and strategic considerations that relate to accountability issues and follow-up. For example re-establishing ownership, governance, escalation paths, roles, and responsibilities among other considerations.
In agile projects there are various approaches to continuous improvement. Specific to responsible and ethical sensing it is useful to map out your journey, consider what impact you have had, and determine a path forward or a productive conclusion. Here are some considerations:
- Can you amplify the positive impact you have had with your project?
- Is it possible to scale, duplicate, and/or distribute your project?
- If you continue, how can you improve your project responsibly and ethically?
- If you conclude your project, how can you and the public benefit from lessons learned?
When we talk about agile development, design thinking, or experimentation in Living Labs it’s important to remember that these are all non-linear approaches to creative collaboration and conclusions are less important than applying lessons learned to improving your project or designing a better solution to test again. Here are some links to help you think about follow up so you can build on your experience and continue to create impact with your project.
- Agile software development
- 5 Stages in the Design Thinking Process
- Embedding experimentation in government
- Urban Living Labs: A Living Lab Way of Working
6.3 Video Library
At the end of this process, we suggest you revisit your original goals, evaluate your progress, and reflect on your experience. You can do this by applying the Ethical Framework example provided in Step 3. But for now, please take a few minutes to learn from personal experience and insights that will help you think about measuring impact for your sensing project.
All expert interviews ask these three questions:
What are the biggest challenges for monitoring in public spaces?
What are some approaches to address these challenges?
What advice can you give innovators thinking about deploying sensor technology?
Program Manager Sensing Dutch Police
Ido believes the challenge is in the way we interpret the results and how we follow up (with the public). Only with continuous measurement and feedback loops can we understand the consequences of our designs, interventions, and actions and how we can inform the public when we assess the impact of our project.
Cofounder CITIXL & Founder IoT Living Lab
It is important to look at the big picture and by using data for the common good we will create positive social impact. Paul reminds us that we need to make sure that our digital rights are protected in public spaces; that measuring impact is not just about getting “results”, but also about working together to reclaim our public spaces and protect the digital commons.
You have reached the very last step of our Toolkit. If you want to put these steps into action, visit the Let's get started section to learn more about our workshops.
Responsible Sensing Toolkit Workshop Trial
As you can see this whole process is quite complex. Even as a municipal innovator with experience in sensing projects it can be difficult to implement these tools on your own.
If you need help, sign up for our Responsible Sensing Toolkit Workshop Trial. This workshop (Workshop 1) will help you and your team to identify your dilemmas with the Decision Canvas as a guide. The Workshop Trial takes only one hour and helps you to set up a clear roadmap to a responsible and ethical sensing project. In particular cases this workshop is free of charge.
Please check out our workshops or contact Sam Smits.