Six years ago, the Transparency for Development (T4D) project sought to uncover new and unique evidence on the role of community-led transparency and accountability in improving health outcomes – and how and why we see mixed evidence when these interventions are implemented in the real world. Based on large-scale mixed methods research utilizing quantitative and qualitative approaches in two countries (Indonesia and Tanzania) and small innovative pilots in three additional countries (Ghana, Malawi, and Sierra Leone), the project now has a lot more information about how these popular approaches to citizen engagement and participation work – and ways in which they can fail to reach their goals. Join us to hear the T4D team share results – including the good, the bad, and the unclear – from this one-of-a-kind research project and discuss what is needed next in the field.
- Dan Levy, Senior Lecturer in Public Policy, HKS; Principal Investigator, T4D
- Stephen Kosack, Visiting Associate Professor of Public Policy, HKS; Principal Investigator, Transparency, T4D
- Jessica Creighton, Assistant Director and Principal Investigator, T4D
Jane Mansbridge, Charles F. Adams Professor of Political Leadership and Democratic Values, HKS
Refreshments will be served.
This event is co-sponsored by the Center for International Development.
Presenter: You are listening to AshCast, the podcast of the Ash Center for Democratic Governance and Innovation at Harvard Kennedy School.
Dan Levy: The outcome that we're really wanting is people to feel more empowered and to feel that they can participate and have a voice in their communities. And whether that leads to improvement in health, or in car crashes, or in whatever outcome might not be a realistic way of affecting them. It's really hard to change this outcome.
Presenter: Six years ago, the transparency for development project resorted to uncover new and unique evidence in the role of community led transparency and accountability in improving health outcomes, and how, and why this mixed evidence when these interventions are implemented in the real world based on large scale mixed methods research utilizing quantitative and qualitative approaches in two countries, Indonesia and Tanzania, and small innovative pilots and debris additional countries, Ghana, Malawi, and Sierra Leone.
The project now has a lot more information about how these popular approaches to citizen engagement and participation work and ways in which they can fail to reach their goals. On Wednesday, September 25th, the transparency for development teams shared results including the good, the bad, and the unclear from this one of a kind research project, and discussed what is next in the field. The conversation included principal investigators from the transparency for development project. Dan Levy, Senior lecturer in Public Policy at Harvard Kennedy school, Stephen Kozack, visiting associate professor of Public Policy at Harvard Kennedy school, and just good great assistant director, Jane Mansbridge, Charles F. Adams, professor of Political Leadership and Democratic Values at Harvard Kennedy school.
Archon Fung: My name is Archon Fung and I'm a faculty member here and work on democracy programs at the Ash Center. I am very delighted to welcome our illustrious panel, which is chaired and moderated by my good friend, Jane Mansbridge, who's the Charles F. Adams professor of Political Leadership and Democratic Values. She has studied participatory democracy all over the United States and in many parts of the world. As a matter of fact, she's just coming back from North Texas, [inaudible 00:02:12] fascinating project in looking at citizen deliberation. And this is the rest of the team, which I think Jenny will introduce. And I just want to welcome everyone to talk about, discuss, reflect upon this project that these people and myself have spent the last five or six years invested deeply in. Let me hand it over to you, Jenny.
Jane Mansbridge: I'm glad you're all here because it's ... I consider this a very exciting project. I'm going to just introduce our three folks. Jessica Creighton is based at the Ash Center, and she's Assistant Director of the Transparency for Development project, T4D. Isn't that a cool logo? She's managed the project in 2013. In addition to project management, she contributes intervention design, research design and writing. And before joining the T4D team, she provided field management and research assistance to the reconciliation conflict and development project, which is a randomized control trial also with innovations for poverty action in Sierra Leone. And she holds an EDM and International Education Policy from the Harvard graduate school of education.
Stephen Kosack is associate professor of public policy and governance at the university of Washington, Seattle, senior research fellow at the Harvard Kennedy school of government, and one of the principal investigators of this project. Steve is a political scientist who focuses on how governments become more responsive and effective for citizens without youthful wealth status or connections. He received his PhD in political science in 2008 from Yale, and previously, he was advisor to the late Senator Ted Kennedy, and a research fellow at the Brookings institution in Washington, and [inaudible 00:04:05] Brown, the London School of Economics, and here at the Kennedy school. And Levy is a senior lecturer at the Kennedy school and he's an economist by training, and he's been involved in evaluations of social programs for the past 20 years in a variety of contexts, including Jamaica, Mexico, Burkina Faso, and Nigeria. We've got quite a team here and they've got some very interesting stuff to tell you.
Stephen Kosack: All right, well, thank you so much everybody for coming to this exciting combination of, I'd say seven, more than seven years of researching and a lot of efforts besides the talk today is based on contributions from many, many different folks. Couple of pictures of some of the core team members here, also highlighting that we're a very diverse group of researchers and practitioners, as well as folks with many different methodological backgrounds and dispositions. Some of us have some quite a bit of research experience with RCTs, and I among them have a lot of experience in and disposition toward qualitative methods. It's a huge undertaking we're going to be talking about today. And I want to just highlight a couple of the other entities and folks involved. We've done our research in five countries and our partner organizations, potato in Indonesia, giant Tanzania, the Center for Democratic Development in Ghana, The Malawi Economic Justice Network in Malawi, and Washnet in Sierra Leone have been hugely instrumental to all the work I'm going to discuss. A lot of our data was collected by a number of farmers that were equally dedicated; survey meter in Indonesia, ideas and action in Tanzania, and we had independent researchers in Ghana, and Malawi, and Sierra Leone, and an exceptional steering committee and advisory committee who have been with us from the beginning.
And last but not least, are incredibly generous and supportive funders, the Gates foundation, the Hewlett foundation. and the UK Department for International Development. There's our website, there's a lot more than we can possibly discuss today on that website. I want to start us off with a very inspiring story that has been a big part of a lot of the work in this field and including our own, from Uganda in 2004.
In that year, there was an amazing governance intervention implemented in a number of communities, that basically provided information and facilitated discussions to community members about the quality of their maternal and newborn health care services. And this intervention, this program, was evaluated in a famous randomized controlled trial that year, and found that it, led to pretty astounding, improvements in care. A 33% reduction in infant mortality rates just a year later, big increases in utilization, all kinds of other very positive things. And just from two rounds of village meetings that encouraged communities to be more involved with the state of their health services and strengthen their capacities to hold those health providers accountable. Typically, we think of this kind of participation as coming from inside a community originating organically. But with the global push for more participation, more participatory development, there've been a lot of moves as well to try to encourage more of this kind of participation in more effective participation.
Often, this comes with or as encouraged by additional resources or authority. Some of you will be familiar with community driven development programs, which try to give foreign aid directly to communities to spend it rather than on programs designed from outside. Others of you will be familiar with participatory budgeting initiatives, which give money to, or at least the discussion of money, how to spend money to people in communities rather than having bureaucrats or politicians figure out how to spend it. [Acon 00:08:42] labeled this style of participation, empowered participation. But there are others, anyone who's familiar with this world knows some real questions about this kind of effort, especially around how outsiders and those developing and implementing development programs can sustainably empower communities without ... with these kinds of resources or authority without creating conditions of dependency and other kinds of incentives that drive them away from what they ultimately care about.
The kind of approach that I just outlined that happened in Uganda is a kind of interesting middle ground here. It does not originate from inside a community, so it's externally encouraged, and yet it doesn't offer any additional resources or authority to communities, and so tries to rely on their innate willingness and capacities to improve their public services. And that's the kind of approach that we're going to be focused on today. For the last seven years, my colleagues and I have been engaged in a effort to try to understand this sort of approach. This sort of approach has gotten a lot of interest. It's been tried around the world by dozens, if not hundreds of organizations. And there've been, other than the Uganda studied a number of other investigations into its effectiveness and they have been decidedly more mixed. Into our study, which is designed to offer evidence of whether these kinds of approach 's transparency and accountability programs can improve outcomes as well as how, why, and through what mechanisms, and in what contexts. It's a pretty unique mixed method, multi-country study. Our first phase, which we're going to be mostly focused on today is two very large randomized controlled trials in Tanzania and Indonesia, each involving a hundred communities as well as a hundred control. And I know we might on the Q$A for your time, briefly touch on a second phase. It's much smaller studies using an adaptive design in Ghana, Malawi, and Sierra Leon. Mentioned I'll be focused mostly on ... we'll be focused mostly on Tanzania and Indonesia today. As part of this, we have tested or we are testing a new sort of approach that's based on the community scorecard design that was shown to be so effective in Uganda in 2004.
But which we spent a lot of time with local partners in Indonesia and Tanzania redesigning, co-designing and iterating to make sure that it met basic assumptions about it's ... about the approach, but was also widely relevant to many different contexts and scalable so that it's the sort of thing that could be implemented in a hundred randomly selected places. Along I thought we had a couple of goals. One was that it be scalable and relevant to lots of different problems on the logic that every community's problems were likely to be different. Second, we designed it to be non-prescriptive and focused solely on encouraging participants in the program to develop and then work on the approaches that they thought would work in their communities on the logic that what's likely to work, the kind of participation that's likely to work to improve public services at one place may not work in another, and started free of outside resources or additional authority, so that it's encouraged participation, not empowered, or organic, or community driven development.
The end result was after a lot of other different design decisions and testing. Piloting is a series or was a series of six meetings in which a facilitator invited interested community members to come discuss information that the facilitator had gathered in the previous couple of weeks about the quality of maternal and newborn health care services in their community, access to it, a subjective and objective information about that as quality. And they took participants through a kind of process of developing a plan of activities that they could undertake to improve any problems that they perceived or that the information revealed. And what we're going to be discussing today is basically an attempt to see that this sort of approach is effective enough that you can essentially randomly pick communities to offer it, and it will on average be effective in improving their maternal and newborn care.
To do that, for the health side of this, we're going to be relying on a basic logic model, in which the sort of approach I just outlined, which we can think of as the inputs in this logic model, we hope lead to some communities, at least undertaking activities that they planned and flap would help to improve their maternal and newborn health care. Those activities were done results in one of three kinds of improvements, either increasing the demand for health services, improving the patient experience, or improving the health facility. And those in terms on lead us to a good thing. Service outcomes such as increased utilization or improved contents of care, and ultimately, to improve maternal and newborn healthcare outcomes, like hopefully, reductions in infant mortality and other good things that great healthcare can deliver. With that, I'm going to turn it over to Jessica to start us off on the results.
Jessica Creighton: Thanks, Steve. Hi everyone. Okay, so you just heard Steve explain the premise in the T4D project and describe the intervention program.Before I get, I have a question for all of you, which is, do you think that the T4D program successfully encouraged participants to design and implement social actions? This is basically that first link in the logic model that Steve just showed you.
Jane Mansbridge: [inaudible 00:15:06] my guess, right?
Jessica Creighton: Yeah, your guess. Yup. You can vote using this link here on your phone.Stephen Kosack: I should have prepared you for this. Sorry.
Jessica Creighton: Well, sorry you can't see it, but it looks like 85% of you said yes, and 15% of you said no. All right, so let's find out. The answer is yes.
Jane Mansbridge: As you guess.
Jessica Creighton: Before I get into the actions themselves, let me just first talk about meeting participation because it was at program meetings that community participants were encouraged to design and revise these social actions that they undertook. Using a few different metrics, we see that participation in meetings was relatively high and sustained throughout the T4D program. For one meeting, attendance was high throughout, though it did taper somewhat. The average number of participants per village in the first meeting was 14 in Indonesia, and 15 in Tanzania, and 11 ,and 10 respectively at the final meeting. And just so the intention was for about 15 community members to participate in this, in each village. We had observers sit in on some of these meetings to count how many times people spoke distinctly.
We see that participation levels as measured by percentage of people who spoke distinctly, at least once per meeting started high in both countries, though participation declined in Tanzania still by the final meeting, a solid two thirds of attendees were participating. And then depth of meaning participation, as measured by the average times participants spoke per meeting, was steady in Tanzania and even grew over time in Indonesia. All T4D communities planned social actions, the minimum number was two and most planned several, and in Indonesia, one community even planned 17. That's quite a few. Most also at least attempted these actions. In fact, participants in all but 11 communities across the two countries reported completing at least one action.
And this was certainly not guaranteed considering the T4D intervention program was voluntary and participants were not paid to undertake the action. Now, let me describe some examples of these actions. Photo one here is of a complaint box that a community in Tanzania installed at one of the health facilities. Photo two is the foundation for a new facility that a community in Tanzania started to build. Number three is in Indonesia, and this is community members coming together to plant a medicinal herb garden at one of the health facilities.
And number four is a poster from an informational campaign, and one of the Indonesian communities, and on it is posted the duty schedule and contact information for the midwife, the contact information and names of all of the community representatives, or the people who participated in this program, and also some information from the scorecard about the uptake of health services in these communities. Not only did communities design and attempt to social actions as I just illustrated, these actions were diverse in nature. One question we had when designing the T4D intervention was whether communities would each design actions unique to their circumstances, or whether they would all converge around a small number of action types. What we saw was a wide range of actions with 43 distinct goals, which we were able to classify into the 12 pathways on this slide. You'll see that despite the wide range and actions, there was one striking similarity across nearly all T4D communities. 93.5% designed at least one action aimed at increasing awareness, knowledge, or community attitudes. More specifically, these tended to be education or socialization campaigns. The next most common pathway was improved facility access, which included attempts to build or request a new health facility, advocating for ambulance services, fixing roads, organizing or advocating for outreach services and arranging community organized transportation. Our analysis uncovered a few additional trends. Firstly, we classified actions as either collaborative or confrontational, and found that the actions were overwhelmingly collaborative in nature. This was not driven by the volume of education actions and suggests that when a transparency and accountability program does not prescribe a particular strategy, communities will choose to be collaborative.
Secondly, the majority of the actions were short route, meaning they targeted the health facility or provider directly rather than the government officials higher up the accountability chain. This was especially true in the case of government actors above the village level. Our assessment is citizens may have been uncomfortable approaching higher level of government officials, or may have been unaware of, or unable to navigate the formal chains of accountability above their village government or frontline service providers. And this is something we attempted to explore further in phase two of our work. Thirdly, when classified by accountability type, we found the majority of actions took a self help approach, with only about a quarter pursuing solutions through true social accountability channels. Also of note is that these trends and the actions themselves were remarkably similar across the two very different country contexts. And now I will turn it over to Dan to discuss impact.
Dan Levy: Great. Thank you Jessica. We saw that the participants design and implement the actions, and now the big question is, did those actions translate into improvements in the wellbeing of the communities that they were being ... that they were targeting? I want to tell you a little bit about what were the main questions on impact that we examined, and I'm going to tell you about what were the outcomes that we measured, and then we'll see what happened. The question is, what is the impact of the T4D program on the key outcomes of interest? The method that we use is a randomized controlled trials. I see the J-pop people here verifying that this was well done. I hope they will conclude that it was.
200 communities were randomly assigned into treatment and control groups in each country. The data sources were mainly household survey that we conducted at baseline and at end line, and facility survey. The household survey, the total sample size is about 12,000 households. If you've ever collected data in a developing country, you know how challenging this was. Most of the results that I'm going to present are going to be on based on the household survey. An important aspect of this is that we ... given the nature of the intervention, we collected data on households who had a woman who had given birth in the last year. The data that we'll show you is not going to be representative of these villages, is going to be representative of households in these villages that had a woman who had given birth recently. And then in terms of integrity, we did a pre-analysis plan in which we said, "We're going to measure the outcomes, we're going to measure the impacts on this outcomes."
We tied our hands so that we couldn't come here today to tell you about the three significant findings that we found in some appendix of our report. What you're going to see here today is what the ... we're reporting on the outcomes that we tied our hands to. Okay. I'm going to talk a little bit about the key outcomes of interest so that you have a sense, but they are derived from the logic model that Steve presented. There's one set of outcomes related to utilization, so that women give birth at a facility, that women give birth with a skilled provider. And in the case of Tanzania, did they do prenatal care? Then in terms of content of healthcare service, we had content for prenatal care and delivery, and we also had content for postnatal care. Then up one more level in the logic model, we also had child health outcomes. A proportion of children who were born, who were stunted or underweight. And then finally, we have a measure of citizens perception of empowerment and efficacy, again, based on the sample of women that we interviewed.
All right. The answer for our outcomes for Indonesia. And what you have here is I'm going to present all the outcomes to you in a single graph for Indonesia. And the line in the middle is zero effect. And what I'm going to show you is 95% confidence intervals for each of these outcomes. They're all standardized, so they're measured in effect size. If the confidence interval overlaps with the line, it means the effect was not statistically significant. If you are not too clear on that, I see several of my students in the audience, they can clarify this for you.
It's only in the semester, but we've done this. All right, so here's birth with a skilled provider. As you can see, that confidence interval overlaps with zero, which mean we cannot statistically distinguish the effect of the program on this outcome from zero, so it's not a statistically significant effect. And then I want to show you the rest of the outcome. And this is it. I want to say one more thing about this. For those of you who are statistically inclined, you might say, "Well, maybe there was an effect and the program ... and the study wasn't able to detect it." As you can see from this true confidence intervals, they're pretty tight, which means that if the program had a policy relevant effect, we would have detected that with our study. All right. Tanzania, same story. The big puzzle that we have now is there are all these actions that were conducted, but on average, communities saw no impact on the key outcomes that we saw to measure. Steve is gonna explain why. Good luck Steve.
Stephen Kosack: I see it as a highway act. Before we get started, let's do one more elicitation of priors. Given what you just saw, the next question is what is participation for? For the people who came to the meetings, engaged with them, planned activities, tried those activities, saw a foundation of a new facility, a complaint box, a new herb garden. But two years later ... And I think we didn't quite mention that this end line survey in which we are judging the effects of this program was conducted two years after the program. Two years later, or during the program itself, do you think that this process was empowering for those who did it? I'd say the split here, which is a bit less than our priors on whether communities and participants in communities designed and implemented actions is closer act to the truth as well.
In order to understand this, we're going to dig a little more into the other methods that we used as part of the study. In addition to the surveys and observations, the facilities that were the basis of the results that Dan gave you, in a smaller number of communities, we also asked some researchers to sit in on the meetings and see what the discussion was like, whether the facilitator was dominating it, whether the participants were choosing approaches themselves. We also interviewed quite a few people who went to the meetings as well as others in their community about the activities. And still in a smaller number, we had a second observer watch the individual participation, and that's the source of what Jessica presented earlier on who spoke and often, as well as we conducted some interviews with participants before the first meeting and after the last meeting about how personally able they felt to improve their community.
And then in a still smaller number we asked ethnographic scholars to live in the communities before, during, and after the program to get a sense of how they understood and responded to it. That's going to be the set of information we're going to use to understand how they perceive the program and particularly whether they perceived it as ... their participation in it as empowering. First, during the course of the program itself, the experiences were mixed. Most started pretty optimistic that they were going to be able to improve care in their community. 85% in Indonesia, and 88% in Tanzania looked to the researcher who was sitting in on the meeting as though participants were optimistic. By the last meetings, the meeting six, about three months later, most in Indonesia were still optimistic, and some who were sort of skeptical found that they had been able to achieve more change than they had expected.
But in Tanzania it was quite back. About half and half of those who started optimistic ended up optimistic ... ended up skeptical. And in 17 of these 18 where they ended up skeptical, the observer thought that not a single participant in the meeting was optimistic that they were going to be able to sustain improvements in their care. Points to the complexity of this sort of experience. It's just six meetings, but it's quite a complicated experience for most of the people who did it. These are echoed in these individual interviews that we did before and after. Interestingly, we asked the same question, how able are you to make improvements to your community? In Indinesia, participants were quite mixed in how, on a one to four scale, how able they felt. In Tanzania, they were much more optimistic at the start. And then at the end, many in Indonesia reported a higher level of self efficacy, very few decline, while in Tanzania, it was again, far more mixed, and most stayed the same.
That was doing the program. Now, we interviewed them again two years later to see what participating in the program was like. In the interim, you could imagine all kinds of things happened after the facilitator left for the final time. Interestingly, by the end, reflections on participation in the program had changed from this mixed story to being almost universally positive. In almost all communities, most of the people who ended up attending these meetings had quite a detailed recollection of what they had done and how it had worked or not worked. They tended to recall at least one activity that they thought had been successful, but many also thought at least one thing they had tried had not been successful.
And then reflecting on what ... the results of the activities, 83% of communities in Indonesia, and 95% of communities in Tanzania, participants generally thought that their activities over ... had improved health care in their community. How do we understand this disconnect? I'm going to briefly give you five reasons that the perceptions of participants differed from the perceptions that we ... the objective information that we gathered for the now impact that Dan described earlier. Five reasons.First, the participants approaches that Jessica described earlier were not unusual. In fact, for the most part, similar things were also happening in the control communities. The program was designed to encourage participants to design the approaches that they thought would work best in their community, and they tended to use or rely on what they knew, which was, all the kinds of activities that Jessica outlined earlier.
Only three of the dozen or so categories of approaches that Jessica outlined, were different ... were more common statistically in the treatment group of communities, and some of these were only barely significant. In Tanzania, there were more of these kinds of activities overall, but in Indonesia, they weren't even that. There are basically similar number of these kinds of activities in the treatment and the control group. Second, when we asked two years later why participants thought that they had improved healthcare in their community, the results were often a bit vague. In many they were clouds tangible improvements that they could see from their activities and that they remembered two years later, but that was less than half. Just to unpack this a little bit, the numbers I showed you earlier most recalled attempting at least one activity in both Indonesia and Tanzania.
Most thought they had at least one activity that was successful, and most thought that they had improved care in their community. When we tried to unpack this a bit and see if we could think that the activity was successful in the sense that it achieved its goal, we had a much smaller number, so a much more stringent standard here, but we had a much smaller number, a little under 50% in both. And some of these activities were ... they wouldn't results in some kind of a tangible improvement, like just a one off cleaning campaign of the facility. When we looked again to see whether they were called achieving something tangible, like a new ambulance, more staff, complaint box, or improved access to this ... like say a new road to the facility, but the numbers were even smaller.
41% in ... of communities in Indonesia, and 30% in Tanzania were ... had something tangible that they had achieved as a result of their activities, so less than half. Less than half is not nothing, right? I think that's important to know here. This wasn't enough on average to move the ball on the outcomes, but it also wasn't nothing. Similarly, much of the activity seems to have occurred for most communities during the course of the program. Only about a quarter of communities, participants kept meeting long after the program ended, after the facilitator left for the last time. Actually in Indonesia, I think it was around 14%, and Tanzania about a quarter had met sometime within the last six months. 8% in Indonesia, and 40% in Tanzania had met within the last two months.
Again, not nothing, and remember that this was just a series of six meetings, participants were not paid, they were not given any resources. It's interesting that that number were still meeting and engaged in activities later, but it's still far from a majority. It helps again to explain some of this disconnect. The fourth thing I want to point to is, we often as researchers like to think that when we do an intervention, it is the first time that any sort of thing like this has ever happened in these places. That's almost never true. And there's very good evidence from our surveys, especially from our ethnographic work that the perception of this program, the experiences of this program was very influenced by the memories of past programs that these people had experienced. Especially that development programs typically pay, and they typically offer resources. A program like this, which offers no pay, and no resources was against early expectations, and it took some time for those to be worked out, and lots of some very severe disappointment in several where they ... when they learned that there would be no resources.
And in at least one of the eight communities where we asked the ethnographic scholars to live, this was ... this disappointment was basically dispositive. There was no more activity after they realized that there would be no resources. And most of that wasn't the case. Eventually, a group came to understand the goals of the program is being different than what they had expected. [inaudible 00:37:02] that, and found the experience to be empowering. Seven of the eight in the ethnographic ... of the ethnographic communities where we had ethnographic scars that eventually happened. But it wasn't for everybody, and it took a little while in some cases.The final was that many of the benefits of participating in the kind of a more empowering side of it were personal and not related to having improved care. We asked participants whether they were glad that they participated in this program, and it was almost universally the case that they said that they were. We asked them if they had experienced personal benefits as well as whether they had experienced personal costs. There were some who had said they experienced personal costs and it was less than a third in both places at where anyone said that, and there were things like, "Well, this was a waste of time. We didn't get anywhere." Or "We were treated suspiciously by others in our community who either thought we were trying to improve candidate they didn't really care about, or that we were being paid and not doing this as a voluntary thing as we ... they ended up being." Those were the costs and they were pretty ... there were some who experienced that, but almost also that they'd experienced personal benefits as well.
And some of these were pride in the things that they had achieved. New ambulance, new facility, new staff, seeing men accompany their wives to the facility, and being treated better when they got there. Many expressed pride in achieving those things, but some, an equal number actually, also expressed some sorts of more personal benefit, like learning new information, or being able to speak publicly about community issues, which were certainly empowering to them, but had little to do with the effect of their activities on care in their community.
Just to end here, we have three types of experiences. Break it down very abstractly. A sizeable number, but not certainly enough to move the ball where they ... community members who participated in this program both perceived that they had had some role in improving health care in their community, and remembered some sort of a tangible improvement that they had achieved.
And in that, as you can say that participating was empowering as well as helpful maybe for improving their care, although definitely usually not transformative. Not enough to show up in our measures two years later. And equally maybe or even larger number of communities there was a perception of efficacy, but no memory of actually achieving anything tangible. And in those we could say that there was either a kind of placebo where, so health care generally was improving, outcomes were improving in these communities, and so they could have thought that was a result of my activities, even though there was very little connection between those activities and the improvement. It could have been partly a performance related to expectations of future resources that they were doing for the facilitator. It could have been something more personal, and it could've been something else. And this is something we're exploring a lot more right now in our other work.
And then in a small number of communities, there was not a perception of efficacy, no any memory of any sort of tangible improvement and then those, we can say that participating in the program was disappointing. With that, I want to ask the final question. Given all of this, do you think we should use transparency and accountability to improve health? Yes. 63%, yes, 30%, no. Interesting. But 3% not true. Only 7% no. Okay. Well that sounds like a good entree to a discussion.
Jane Mansbridge: Thank you very much. Before we start, I just want to make my own small comment. Is it you're commendably cautious about your results, and I am completely convinced that you didn't affect birth weight. But I'm thinking that an ambulance, a new road, these things aren't nothing. And if in your own assessment, not just their optimistic assessment, but if you're in your own assessment, 30% to 41% did something, that seems pretty good to me. Yes, it didn't affect birth weight within your ... and it might not ever affect birth weight, but it might affect other things. It doesn't seem ... I read the paper as well, and in the papers, it's sort of ... effect is low. Wait a minute guys, you actually did something.Stephen Kosack: Can I just add a caveat on ... The perceptions I gave you on tangible improvements are their perceptions. They're their memories.
Jane Mansbridge: Oh, I get you.
Stephen Kosack: When we then go to verify with ... We can only do this in a smaller number of communities where we had key informant interviews. That's 60 of the 200. In those where we try to verify with other data sources that there's a memory in the community these improvements, we have some evidence in the surveys that we did. The number is going to be less than 20%. I think it's good, but it's not enough to move the ball on average.
Jane Mansbridge: 30 to 40 ... 41% thinking we got something done, even if you couldn't actually verify what they got done, I think is good. But questions and comments, and so ... you want to take the questions? Okay.
Monica: My name is Monica. I'm from the school of public health. I'm also from Indonesia. Thank you very much for the very interesting presentation. I have two questions. First, I'm still trying to understand why community participation is a proxy for transparency and accountability. And then a second, who are the participants of the meeting? You said the average of the participants are about 15 people in a meeting, and I think a village in Indonesia has 2000 to 3000 people. How could 15 people represent the total population? Thank you.
Aki: Thank you so much for the presentation. My name is Aki, I'm also from Indonesia, from the Kennedy school. I have two questions as well. The first one, do you mind talking or explaining more about the gender dimension of the intervention? I know my understanding is this program was designed not as a gender transformative program, but when I went through these communities, I saw a lot of the community leaders who were involved, who participated in the program were women. And I guess because of the focus on maternal health, I think it's just very intuitive for this program to have some kind of gender dementia related to it. And my second question, my understanding is that there was a conference in Indonesia, where the team presented this findings to stakeholders back in Indonesia. Considering that the key outcomes ... there's no end back on key outcomes, how the government of Indonesia specially knew. I know the ministry of village was involved. How did they respond to this ... the results, and what are some of the key takeaways that the government can learn from this study? Thank you.
Jane Mansbridge: I think that's [inaudible 00:44:45].Dan Levy: Memory would taxed very heavily if we take more questions without answering them for us. So thank you.
Jane Mansbridge: [inaudible 00:44:52] answer.
Dan Levy: I think Steve and Jessica got a better position here.
Stephen Kosack: Sure. And I may actually bring in some of our colleagues as well if they are willing to contribute to this discussion. Let me answer the question first about community engagement, or participation, or being a proxy for transparency and accountability. I think one of the ideas of transparency and accounting programs is that they encourage or engage citizens in participating in improving public services. In fact, some of the field is called transparency, and accountability, and participation, TAP.
And the idea of this sort of program is that by providing information in the form of a scorecard about the quality of care, and then encouraging community members to take action to try to improve that care, that that may lead to some sort of accountability of the healthcare to their needs, so an increasing kind of responsiveness. And that is the causal dynamic that a lot of researchers are interested in exploring because it's not clear. It's a very long causal chain to get there. And so, that's the kind of question that we have been asking, and many others in the field as well. It's not a proxy for ... it's a sort of a mechanism through which you might get transparency leading to accountability and more responsive public services.
Jane Mansbridge: The transparency is the scorecard.
Stephen Kosack: The transparency is the scorecard.
Jane Mansbridge: That word accountability would be the upper level officials, which you say have a very small percentage, less than 10%. Actually, what they did was much more participatory things.
Stephen Kosack: Right. That's right.
Jane Mansbridge: They called it accountability. That's part of your point, right?
Stephen Kosack: Yes.
Jane Mansbridge: Transparency is there ...
Stephen Kosack: Mm-hmm (affirmative).
Jane Mansbridge: ... but the accountability, at least in our formal and our normal sensitive work was only a small part of what he's got as outcome.
Stephen Kosack: Jessica mentioned that about a quarter was social accountability. And that's because a lot more than just 8%, 10%, did try to engage with their frontline providers and that was another form of accountability. But it's true to say that when you think of accountability as accountability of top level governance to citizen's needs, then it was a very small proportion. And that's again, something we're exploring in that in the second phase. The second question you asked, I think I'm going to ask Jessica to comment on the question.
Jessica Creighton: Sure. As far as who the participants were, we actually did multiple rounds of piloting to figure out who should come to these meetings, and whether they should just be an open invitation, or if people should be purposefully recruited and how many and all of that. And where we ended up was ... that 15 was a good number as far as being able to work with and manage the amount of people who were there for the facilitator to make sure that everyone in the rooms voice's were heard, and especially because they were putting together these action plans, could really be part of this action planning process.
But as far as who these people were, the facilitators really made an effort to get a cross section of people, and with an emphasis on trying to work with people who weren't necessarily leaders in the past or that type of thing. Trying to get some new faces in the room to really understand the experiences of people with their their health system. But that being said, there also were some people with leadership skills, and they tried to get a gender balance, that type of thing. But to at least point especially in Indonesia, I think we saw the balance was definitely skewed towards women. Yeah.
Stephen Kosack: Interesting. Yeah. As far as-
Jane Mansbridge: You had a [inaudible 00:49:00]
Stephen Kosack: The conference.
Jane Mansbridge: You wanted to ask about.Stephen Kosack: Yeah. I don't know. Jessica ... Courtney made me want to ... We were there, the three of us together, last week. Yes. And I'm still paying for it [inaudible 00:49:19]. Actually, it was a great conference. I will say just one word about it that I think that the results were to some degree disappointing because there was some optimism there as well on the potential of citizen engagement to improve a lot of governance issues. And I think sort of interesting to see the deep commitment of the government to citizen engagement, and the interest in that. As long as I think among the organizations who were there, not a huge amount of surprise that the final level outcomes didn't move. Just exactly as professor Mansbridge said that this is not like the first time that this sort of result has happened.
And so I think you recognize that it's pretty hard to do it in a way that's truly empowering, and gets you all the way to final level outcomes, because you could provide some other support or resources to do that, but then that might erode the initial empowerment. You can provide a lot of extra support to participants as they do this, so a much longer program. As Jessica mentioned, we tried that and found that it ended up creating a bit more reliance on the facilitator, and there was ... So there's just ... there's a fine balance here to strike. And I think given the way that it seemed like civic participation was most effectively encouraged, it wasn't hugely surprising that it didn't lead to final level outcomes. Yet, I think there was some interest in maybe even excitement around the generally positive experience that participants seem to have. And just as professor Mansbridge was saying earlier, some excitement in that ... at least a minority of communities, they had some association of their activities with tangible improvements. I don't know if that's a decent summary of ...
Jane Mansbridge: Lots of hands. Let's take another couple, one, two, and then we'll get through that.
Jane: Hi, my name is Jane, also from Indonesia, from [inaudible 00:51:28]. I was just wondering, you started with an inspiring story about how it worked in Ghana, and how-
Dan Levy: Uganda.
Stephen Kosack: Uganda.
Jane: Oh, yeah, Uganda, and how it decreased maternal mortality rates by 33%. And why did it work there and not in Tanzania and Indonesia?
Shabbir Cheema: My name is Shabbir Cheema. I'm a senior fellow here at the center. My question concerns your impact, and the determinants of that impact that you tried to explain. But my feeling is that maybe ... And I didn't read the paper, I apologize for that. Just listening to you. My feeling is that participation is a means to an end, and in itself as well. And we cannot generalize more participation before doing some kind of multi level institutional analysis. At least I didn't find that in your presentation. If the participation is low, maybe the reason is not the community. Maybe the reason are the environmental factors. And I would have thought that you were ... gave an explanation about this, the low impact. Thank you.
Stephen Kosack: Sure. I guess. This is something ... the question about ... I think they're related questions, actually, that they're ... why in some contexts you see huge improvement and in others you don't. One hypothesis that we're exploring is that the context matters in both institutional ... political-institutional context as well as the healthcare context. The healthcare system in Uganda in 2004 was just worlds away from the healthcare system in Indonesia today.Jane Mansbridge: It's much worse.
Stephen Kosack: It's much worse. Yeah. Infant mortality rates in Uganda at the time were well over a hundred, in Indonesia today, or at least at the time of the study is 28. This is just a much more capable system. We had a panel at the American political science association a couple of weeks ago where we brought together some of other scholars who have done work. And there are actually quite a number of our CTS now that have been done on this work.
And it does seem like there's at least among two of the studies out there, some consistency that in systems where there's lower rates of infant mortality, there's just more potential for community engagement of this kind to do some work on its own. And that as a system gets more mature, and more of the solutions requires some sort of structural response, which means that there needs to be more government engagement. And as you'll recall from what Jessica presented, very little of the activities involved higher levels of government. This is something that we are exploring as ... in our second phase, which maybe it gives me a chance to talk quickly about that. In that phase, in the small studies in Uganda ... Sorry, in Ghana, Sierra Leone, and in Malawi, we added a group of officials who had expressed some willingness and interest in engaging with communities to try to help them with what they were doing in order to improve the quality of their care.
And this does seem to lead to two things that you might expect even when I just said. One is that more of the activities did try to engage with officials. Quite a bit more, it was 60%, and both Ghana in Sierra Leon, and we saw a higher rate of tangible kinds of improvements. That would suggest some support for that. However, it wasn't an all three. In Malawi, we saw a nighter. And I think there are a couple of potential reasons for that, but I think it's not like this is a universally effective thing, even if you do add that additional component. That kind of gets as well to the institutional context.
In designing the second phase, we had a theory of going into this that it's easier to do this kind of work where providers are willing to be responsive to citizens, and where officials are willing to be responsive to communities and citizens when they try to improve services. In trying to engage with what we called in the second phase of government champions, this might just be easier in places that are more institutionally open, so more democratic places. And so we picked deliberately three additional countries where there are different levels of institutional capacity, but they're all quite open, and so democratic. We hesitated to come out with a claim about the institutional environment was just two countries, and we had two regions in each country. That gives us four basic institutional settings, but that's still a small number. With the addition of the second phase, we have five now, five countries, and that allows us to say ... will allow us to say something much more about the institutional environment.
Hannah Leslie: Right. Thanks. I'm Hannah Leslie from the School of Public Health. And I noticed you said a few times you felt like the intervention wasn't enough to move the ball forward on health outcomes, but I'm curious if you reflected back on your logic model. There's seems to be a very strong assumption that the type of improvements the communities can do themselves or advocate for will result in improved competence of care of the providers and more of that care. And that will be the right care to prevent the mortality that is accounting for the deaths that are causing ... that are happening now. I'm curious if with the facility survey that you talked about or in the additional three countries, you have any more insight on whether the types of change the communities can produce. If the comment box is on the outside of the facility, what happens to the comments in it? If the facility foundation is built, who comes in staff there? How good are they at preventing mortality? If you have any further reflections on that part of the logic model for your ongoing or future work.
Betsy Osborne: My name is Betsy Osborne. I'm also at the School of Public Health. And many years ago I was Asia ... I was Indonesia Program Director here when you're just getting started. It's very exciting to see you five years old, and some results, and some novel results. My comment is actually echoing with the what two other people have just said. As you go forward, it would be great to see the logic unpacked a little bit more about how TAP initiatives. What do you think about how that can affect both the demand side and the supply side of, in this case, maternal and child health care? Because it seems as though there'll be slightly different mechanisms. And one of the biggest challenges we know in differentiating a high functioning health system in a lower functioning health system is the basic showing up for work type performance of community health workers and low level health workers. As you have an opportunity to look at this work going forwards, it'd be great to see that logic unpacked a little bit on the supply side and demand side of that.
Jessica Creighton: Sure. I'll start, but you guys should weigh in as well.
Stephen Kosack: Totally for Dan.
Jessica Creighton: [inaudible 00:59:12]. I was going to say there's certainly a long causal chain here. And we did make an effort to measure along the chain. We didn't focus much on intermediate outcomes today. But we did measure a whole host of intermediate outcomes. Things like, yeah, was there a comment box on the facility? That type of thing that weren't the ultimate outcomes that we looked at, and we found very few statistically significant in all of those, just slightly higher than what we would that then what would be suggested by random chance. But I'll let Dan maybe describe the actions themselves, and some of our theories around that.
Dan Levy: I think the logic model question is a very good question. You think it does help to put the field in perspective? It's not like we invented this intervention out of nowhere. The whole field is counting on, this intervention is improving development outcomes, not just in health and education. If there's anything that I've learned in 20 years doing this kind of evaluation work is that it is extremely hard to have an impact on outcomes in communities being an external actor and just trying to do it. Both of your work at the School of Public Health, and I'm sure you can think of many interventions that seemed like, "Oh, of course this is going to cause an impact on a health outcome." And they don't. Starting with every time we go to a doctor, they tell us to do something, we might do it, might not do it and that might not result in our health improving. That is a very, very direct intervention.
Here, the logic was that the communities would be able to come up with actions that they themselves thought would help improve the system. I think this is the underlying logic of a lot of the field. I, myself, personally feel like, "Well, they might know what's best for the community, but they might not know what's best for improving the health outcomes in the community." At the other extreme, would have been a super heavy top intervention where, right? To improve healthcare, these are the five things you need to do. And that I think has also been shown not to work.And so the question of what's the right balance? I don't know. It seems like if you give food choice to the communities, they select some actions that could affect health, but they didn't. But many actions frankly that don't, a lot of actions that were tried were things like an education campaign. Well, lots of education campaigns that are much more resource than the ones they probably tried don't change health outcomes. I think my answer to both of your questions is, I guess, heavy respect and skepticism for the idea that we ... that an intervention can move an outcome, and much less so if the intervention is designed ... is co-designed with local organizations, but it is very hard to move these outcomes.
Stephen Kosack: What do you gotten a question? Unpacking the mechanisms. I think we were definitely thinking that utilization would help move some of these needles-
Jane Mansbridge: Inside the utilization.
Stephen Kosack: Utilization ... giving birth at a facility, giving birth with a skilled provider. And so this is a debate within the team. One thing you should know is that in both countries over the study period, there were pretty dramatic changes countrywide on those statistics. In Indonesia, between 2012, countrywide birth with a skilled provider increased from 79% to 86%. A birth at a facility increased 55% to 74%, about 12% and 11% changes in Tanzania as well. And so the baseline rate, this is what makes it very, very different from Uganda in 2005, but baseline rate is like way, way higher for utilization. And the trend is the baseline pre-post is upward for both ... for at least for the controlled countrywide.
And so in order to have a measurable fact on the RCT, you have to beat the high baseline and the strong upward trend. I'm not so sure it was about some fundamental mistake in the logic model other than we should have [inaudible 01:04:17], which you might well say. And I probably would say in retrospect, we should have targeted things that were at a lower baseline where the secular trend wasn't upward.
Jessica Creighton: I have one quick anecdote to add to the point of just looking a little bit more deeply at it, and unpacking what was going on. We did have these key informant interviews that Steve mentioned, and your comment about the comment box made me think of this. In one of those communities, they did successfully put a comment box on the health facility, and they decided that they were going to open it publicly, I guess 30 days later. And so they went to open the comment box, and they were surprised that there were no comments in the comment box.
Dan Levy: A failure to the logic.
Christian.: I'm Christian from the school of public health center for health decision science. I'd love to get your perspective on these answers that people are giving. Obviously the audience here thinks that transparency and accountability still have some impact to health. The fact that your results shows now when you're reflecting back and you can do it differently, would you change the intervention? Would you change the time frame? Would you change the outcome that you're looking at? Maybe maternal mortality ... mortality and child health is not the right one. Maybe car crashes would be something that's shorter period that ... as previously mentioned, that could be more easily targeted. Maybe scorecard and community meeting is not the way, doesn't mean that transparency and accountability doesn't work. Just that one specific aspect that you are trying on this trial is not working at this time. I was wondering, reflecting on these answers and if you were able to do it differently, what would ... what is it that would you change?
Jane Mansbridge: We are actually five minutes over.
Katie: Hi, I'm Katie [inaudible 01:06:19], alumni and worked with you all a long time ago at the very start of this project. I have a question about unpacking the logic model a bit more and thinking about sub-analyses in areas, like, if you think of the logic model as a decision tree and when different pieces of it panned out, where the linkage is held. And as you look at the linkages, do you see, actually ... I don't know if you have the statistical power to do so. But if you've looked at ... in areas where the linkages panned out, do you actually see these outcomes or even when say the transparency initiatives led to the activities, led to the comment box being used, for example, do you see in those cases that there was an impact and you just don't see it on average and the averages that you showed us?
Just unpacking some of the maybe sub-analyses and impact on the margins for the different ... when different aspects of the logic model held. And then just unpacking findings a bit more when you ... I know you presented what you put in your pre analysis plan, but at a practical level, I'm curious about whether you found any positive impacts that you maybe didn't anticipate or you didn't hypothesize in your pre-analysis plan and what the implications of those might be.
Jane Mansbridge: Two questions, we'll just add those and then we'll wrap up.
Dan Levy: On the first question, we're not claiming that this study should be the definitive answer on whether we should do transparency interventions. Their message here is not that it shouldn't be done, but I think they should serve as a ... I would say if there's one thing that characterizes the field is not a lot of pessimism about the interventions, it's more optimism. And I think they should be ... contribute to a dose of realism about, what is it that we can expect this interventions to be able to do? Because I think, as we said before, it could be that the outcome that we're really wanting is people to feel more empowered, and to feel that they can participate and have a voice their communities. And whether that leads to improvement in health, or in car crashes, or in whatever outcome might not be a realistic way of affecting them. It's really hard to change this outcome.
I wouldn't leave this room thinking, "Oh, we're just saying we shouldn't do this intervention." And say, "But I do hope that, yeah, there's a lot of interest in making this the solution to many of society's seals." My sense is that they should provide at least some realistic assessment of, "Well, maybe that's not so easy to do through this interventions." I think with respect to Katie's question, what find ... what I found sobering of this intervention, or of this study is that is not that ... it's not just that we didn't find the impact zone on this outcomes that we declare in our pre-analysis plan, is that when we tried to see the intermediate outcomes, by and large, we didn't see much. [inaudible 01:09:58] Steve said that these things were already happening. Maybe controlled communities we're also putting in common boxes. I'm sure, Katie, if you look at a hundred outcomes, some of them are going to be significant. And if we were frankly less honest researchers we will show you the three that we found and say, "Oh, this is ... the intervention is a marvelous." I guess that's not who we are.
Jane Mansbridge: And on that note ...
Dan Levy: Thank you.
Presenter: You've been listening to AshCast, the Ash Center for Democratic Governance and Innovations Podcast. If you'd like to learn more, please visit ash.harvard.edu or follow the Ash center on social media at Harvard. Ash.