By Jim Masters
Center for Community Futures
TABLE OF CONTENTS
A. Key concepts in ROMA
B. Where did ROMA come
from and why should we do it? 10
DOL, USDA, HUD, HAS
D. Where do I get help locally? Performance and outcome measurement in city and county government and at the United Way ...............................15
E. Can I get help from my local college? Outcome and performance measurement and the social sciences..............................16
F. How ROMA is the same or different from previous program structures and reporting systems used by CAAs?
G. How ‘wide’ should ROMA be? Across which programs in a CAA? H. Who should lead the effort in our CAA? 30
I. PROCESS approaches to developing outcome measures
Case Study 1 - Pennsylvania.................................33
Case Study 2 - New York ....................................34
Case Study 3 - Washington ..................................35
Case Study 4 - Missouri ....................................36
Case Study 5 - California ..................................38
Case Study 6 - Rensselearville..............................39
J. Bring your funders along… 42
K. Possible roles for state CAA associations 43
L. Approaches to developing of the CONTENT of outcome measures 44
M If scales and ladders are as good as this author says they are, why are they controversial in some CAAs?
N. Peeling the onion: updating the assumptions on which your current operation is based 1
O. Peel another layer: review your purpose, vision & values
P. More content traps to avoid in system design
Q. Transition from the old to the new
R. Reporting system development
S. Linking your new
outcomes measures to your financial system
T. Software and ROMA
U. Future development
R.O.I., . Community Services Network, Too much generic activity, Persist!
Appendix A. The big picture
Appendix B. Where to put strategies under goals Forward
Appendix C. Toward A Certified Statement of Outcomes, by Frank Monti, C.P.A., Innovating Magazine
22 Steps to ROMA Implementation: Peeling the Onion
This workbook helps CAAs, State CAA Associations and State CSBG offices implement the Results Oriented Management and Accountability (ROMA) framework and principles as adopted by the CSBG Monitoring and Assessment Task Force (MATF). The assumption is that the reader is familiar with ROMA basics and has already read the ROMA Guide and the other materials published about ROMA, all of which can be downloaded from the ROMA web site at This workbook includes only brief references to ROMA as a context for the suggestions herein.
This workbook is primarily for people who have read the ROMA Implementation Guide and are still having some problems implementing ROMA, or who started to implement it and have stalled. The workbook provides:
* a decision making framework, describing some of the major choices you can make about implementation,
* the context for ROMA -- other approaches to outcome measurement in Federal, state, local governments and in other networks (“we are not alone”),
* previous management and reporting systems used by CAAs and implications for ROMA implementation (“front-line staff must love their reporting system in order for it to work”),
* how the six goals evolved and why their use is crucial to continuing the concept of the CAA as an umbrella agency,
* process options to use inside your CAA, at the state level and at the national level,
* criteria to use to develop the content of results measures (and traps to avoid),
* why some standardized results measures (and some optional) are needed at all three levels,
* why the controversy some people have with scales and ladders are really about the family development strategy,
* ROMA and your financial system,
* why creation of ROMA results measures inevitably leads you upstream into a review of your goals, objectives, strategies and programs activities, and
* on each topic, the author’s conclusions and recommendations.
Each topic has three sections. Background information is provided; the author tries to show a range of existing practice so you can “find yourself.” If you are in the right place for you and your agency then you stay there. If you should move up or down on the spectrum then do it. Next, the author states his conclusions (underlined.) Then, thirdly, the author gives his recommendations (bold face) for the steps you should take on that topic.
How to use this book.
You can skip topics and steps, or change the sequence. All materials cited are also listed in the Bibliography.
There are seven points about ROMA to be made up front.
1. This shift is real. Whether or not you think it is a good idea, the new emphasis on describing what you do in terms of results and outcomes is here to stay. It is driven by changes in social values, by the general public’s heightened expectations for effectiveness in government and by far-reaching Federal laws that passed with huge majorities in Congress. It is taking place in every agency and program of the Federal government, and it is taking place at the national, state and local levels in all publicly funded programs. This is not going to go away. This is something every CAA should be doing – and if you are not implementing it already, then get moving!
2. This is big. The shift involves more than just installing a new reporting system. It requires re-thinking what you are trying to accomplish with the families and communities with whom you work, which requires a re-examination the causes of their current situation, which then carries you into rethinking the types of activity in which you engage. So outcomes thinking will eventually take you into an A-to-Z review you your existing agency operations.
3. You can do this. This is the sixth major shift in management frameworks for community action agencies since the Economic Opportunity Act was passed in 1964. This workbook includes a brief overview of each of the previous systems (program structures and reporting and how ROMA is similar or different). You did the first five. You can do six.
4. This takes time. Two to three years is a typical time frame for an agency or a group of agencies to do the kind of collective learning process that takes them beyond the existing way of thinking. It is going to take five years to flesh this out at the local, state and federal levels. This five-year time frame is about the same as the last three shifts in management systems in the community services network.
5. There is good news. The Federal government and most state governments are not telling you exactly how to do this. The approach that OCS, NASCSP and NACAA have taken in leading this effort is to give YOU a lot of latitude in designing your approach. The “bad news” is that since nobody is telling you exactly how to implement ROMA, you have to make up a lot of this on your own.
6. The flexibility in implementing ROMA is good, but it creates a serious risk of diffusion if everybody invents their own measures and definitions. We must create a common language that we can use to communicate with each other and to retain the statewide and national character of our anti-poverty work together. Some approaches that address this issue are included in this workbook.
7. A key challenge is whether you use this new approach and the implementation effort to change your strategies, or just pour old wine into new bottles. In creating the existing reporting system in 1981 -- 1985, the intention then was to attach as much CAA activity as possible to the new CSBG. Then, we were trying to put old wine into new bottles. Today, in implementing ROMA, that would be the wrong choice.
Some people say that ROMA should be used only to tell the existing story better. I think there are some problems with the existing story. We have too many strategies that are based on obsolete notions of the economy or in muddled theories of human development. Peter Drucker advises us that about 1/3 of all goods and services become obsolete each year, either because of advances in knowledge, technology or competition. ROMA offers an opportunity to assess what is working and to identify what is not working too well. Alvin Toeffler, author of Future Shock, says that for the next millennium we must all be able to “...learn, unlearn and relearn.” Let’s use ROMA to unlearn and a few things and to invent some new strategies.
This publication came out of a conversation about “Why are some CAAs having so much trouble with implementation of ROMA?” This workbook attempts to reduce misunderstanding, to get you past a few of the confusing spots and to facilitate implementation of ROMA. This is a temporary document whose usefulness will be eclipsed as CAAs develop capacity and experience and move into ROMA. On the other hand, during the transition, it may save you weeks of wheel-spinning and pointless agony. So – read on!
This publication was assembled by Jim Masters. For the past three years, he has been a consultant to the H.H.S./O.C.S. Monitoring and Assessment Task Force; and on the Survey Subcommittee and the Training and Technical Assistance Subcommittee. However it can not be stated too strongly that the contents of this workbook do not represent the official position of H.H.S. – or anybody else -- other than the author. The author takes full responsibility for any errors – and for the personal opinions included herein.
He used the ideas, input and assistance from the MATF members, State CSBG Agencies, NACAA staff, NASCSP staff and other consultants -- and incorporated their ideas in this publication. Credit is given -- where the author can remember where he got the idea.
Thanks to the following individuals for their feedback on earlier drafts:
Charles McCann, Missouri State CSBG Administrator
Julie Jakopic, NASCSP
Colleen Wagner, NACAA
Send Jim YOUR ideas, or contact him and ask him questions about the ideas in this pamphlet, phone (510) 459-7570.
A. The concepts in Results Oriented Management and Accountability (ROMA)
The first words in ROMA are -- results oriented -- but, if you look at the flow of a program, results come last. They are what happen as a consequence of the operation of the strategy or program.
Inputs Operations Outputs Results/Outcomes
People Program Participants X of this Benefits to person
Money Activity by staff Y of that Benefits to community
So ROMA is an effort to shift our attention away from the inputs and operations of daily activity to the benefits that are produced for the participants and the community as a whole.
Staff are accountable not just for doing their j-o-b, but also for creating the results their job is supposed to produce. Each staff person becomes accountable for producing the results.
Conclusion. These are pretty simple concepts, but are challenging to implement.
Step 1. Start stretching your brain! Expand your thinking past the edges of the old boxes in which we have been working successfully for the past three decades.
The best single one page summary of ROMA that I have seen was written by Scott Anglemyer, from the Kansas CSBG Office. He calls it the "Cliff's Notes" version of ROMA.
1. ROMA is not a set of rules, procedures, and forms. It is a way of thinking about what the community action network does.
2. That way of thinking is essentially that the management of community action agencies should be driven by outcomes.
3. Managing for outcomes involves planning, carrying out those plans, and measuring the results, and using those measurements in the next cycle of planning.
4. The "Results" in ROMA are often discussed as outcomes. When we talk about outcomes, we mean the changes that occur in the lives of families, in communities, or in our agencies as a result of our agencies' activities.
5. Because it is a management system, ROMA is more than just outcomes. But it is outcome driven, and emphasis in workshops and on the ROMA web site tends to be placed on measuring outcomes because that is the part of ROMA that most agencies have the least experience with and knowledge of.
6. States are required to implement ROMA and to report to the H.H.S/O.C.S. on outcomes. Those requirements do not specify what measurements must be used.
7. Most states passed on similar requirements to their grantees ‑‑ they require agencies to measure outcomes, and may establish a general framework, but they usually do not require that specific measures or techniques be used. Therefore, in most states each CAA is free to measure outcomes in the way that makes the most sense for your agency.
8. The things you see on the ROMA web site are examples of measures and measurement techniques that you can use. They are not the only things you can do. OCS has deliberately kept ROMA open‑ended to encourage diversity and innovation across the network.
B. Where did ROMA come from and why should we do it?
Social values and economic interests translate into public policy anywhere from five to ten years after they have “happened in the streets.” ROMA comes from a huge shift in social values. In the 1980s, public confidence in government was dropping. President Reagan and other political figures began mirroring and adding fuel to the public impression that government was inefficient and ineffective. The business sector was praised as being better at measuring their results -- at describing the bottom line of their activity.
In the late 1980's, the Government Accounting Standards Board promulgated standards on measuring results and costs of government programs. Congress picked these ideas up and incorporated them into the Chief Financial Officers Act of 1990 which required federal agency financial managers to focus more on the results produced by expenditures.
At the same time, board members and managers of many publicly funded programs were also asking how to better measure the results of their efforts, as were state and local officials.
There are many good reasons for doing a better job of describing our results that are given in the ROMA Guide published in April, 1999 by H.H.S./O.C.S. and the Monitoring and Assessment Task Force. Still more reasons are found in the workshop handouts published by The Rensselearville Institute:
* Results matter -- to program participants, staff and funders.
* Partnerships are needed to produce most results. It takes a joint commitment to success.
* New concepts and new words are needed to lead to new thinking.
* People are the single most important piece of the whole equation. People, not plans or money, get things done.
* Planning and doing are inseparable.
* Action, change and learning are keys to success.
* Success means being result-focused.
The social values shifts prompted changes among financial managers and eventually led to new statutory requirements. In 1993, Congress passed the Government Performance and Results Act, (G.P.R.A.) Gyp-ra, as it is called, required creation of strategic plans and reporting on results by all Federal agencies by 1997. And it required that budget requests be submitted using the categories in those plans by the year 2000.
Conclusion. The Government Performance and Results Act of 1993 is one of the key elements prompting the shift toward results and outcomes at the Federal level.
Step 2. Review G.P.R.A. It is a key driver. Use it as a touchstone to assess your efforts.
C. How is G.P.R.A. being implemented outside the CSBG? How are Other Federal Agencies approaching the G.P.R.A.?
Let’s look at how it is being implemented in DOL, HUD, USDA, and H.H.S.
1. Department of Labor. DOL has a centralization/ decentralization cycle that recurs about every ten years.
The Manpower Development and Training Act of 1962 initially granted substantial authority to states. Over the remainder of that decade, control over programs was slowly centralized back at the Federal level.
The Comprehensive Employment and Training Act of 1973 (CETA) was one of President Nixon’s Special Revenue Sharing Block Grants (along with SSBG and CDBG). Under CETA, local Prime Sponsors (most of which were CAAs) were granted substantial authority over program planning and management. During the 1970's DOL pushed most of the CAAs out of the prime sponsorship role and replaced them with city or county governments. DOL also wrote so many Federal regulations that by the late 1970's the program was not only a bureaucratic nightmare, but in the process of creating it DOL had once again moved into the driver’s seat.
JTPA replaced CETA in 1981, again devolving substantial authority to states and the local PICs. Then, DOL began imposing rigid performance contracts and cost-reimbursement standards in which the SDA could make progress payments for number of people assessed, trained, placed, and still on job after 30, 60 or 90 days, etc. (As an aside, I predict this performance-contracting model will come into human development and public charity services, too.)
In the fourth decentralization cycle in as many decades, Congress has once again delegated substantial authority to the state and local levels through the Workforce Investment Act (WIA). In WIA, responsibility is devolved to states, previous program categories are merged, “one-stop” shopping is encouraged, and the local Workforce Investment Board has broader the representation and a larger role. DOL has developed twelve G.P.R.A. related performance indicators for the WIA, and will eventually negotiate targets on each with the states. The states will then presumably negotiate with each local WIB about what they will accomplish on the 12 performance indicators. Thirteen states are in “fast track” implementation of the WIA and supposedly “started” their new system on July 1, 1999. The other 37 come on board by July 1, 2000.
As this is written, DOL is in the throes of another reorganization. Many top-level career staff are leaving, reducing the capacity of DOL to implement WIA in a timely manner. The upshot of this is that now is the time to work with your State agency and local WIB on what they are going to do and with whom they are going to do it. As one long-time employment and training expert it, “the deck is being shuffled -- don’t wait until all the cards are dealt.”
2. HUD has three new block grants. The new consolidated planning process (ConPlan) developed in the mid 1990's provided a vehicle for HUD to introduce goals and performance measures. HUD is lucky enough to have enough concrete stuff (no pun intended) to count, but as we have seen in the recent Congress, effective program operations, a reinvented and refurbished Federal agency, and substantive results measures are not enough to insulate an agency from the vagaries of the budget process.
3. USDA developed 13 different strategic plans – one for each mission area. The plan for rural development (housing, economic development, facilities development, rural utilities, co-ops) had dozens of NEW results and outcome measures in it. Staff task forces worked over a period of three years in an iterative process to develop and refine their measures.
The 1996 Farm bill required development of strategic plans in each state, and more measures were developed during the planning process in each state. Money continues to flow from both state and Headquarters offices depending on the program. You may want to get involved either in development of the state plan and the performance measures in it and/or in the special task forces attached to the national programs like co-ops and rural utilities.
A. The overall H.H.S plan has five goals. Most of A.C.F. fits under its Goal 2, “Improve economic and social well being of individual, families and communities.”
B. Agency for Children and Families. The A.C.F. has consolidated several plans from the operating agencies and programs including, Head Start, TANF, CSBG, and others. They have created a framework of 4 goals, with 8 program objectives and 2 administrative objectives. They are given next.
Strategic Goal 1: Increase economic independence and productivity for families
1. Increase employment
2. Increase independent living
3. Increase parental responsibility
4. Increase affordable child care
Strategic Goal 2: Improve healthy development, safety and well‑being of children and youth
5. Increase the quality of child care to promote childhood development
6. Improve the health status of children
7. Increase safety, permanency, and well‑being of children and youth
Strategic Goal 3: Increase the health and prosperity of communities and Tribes
8. Build healthy, safe and supportive communities and Tribes
Strategic Goal 4: Build a results‑oriented organization
9. Streamline ACT organizational layers
10. Improve automated data and management systems
Most of the departments in A.C.F. are taking G.P.R.A. very seriously, because the budget submission is now built using the same categories as the plan; i.e. the budget process uses the plan formats to request funds. And, this is exactly what the framers of G.P.R.A. intended.
The H.H.S. web page has the entire A.C.F. plan for the year 2000. It is about 90 pages long. The plan for each coming year is released after the President’s State of the Union Message, so the
2001 plan will be released in February of 2000. It is in draft form and is about 115 pages long, but it is “embargoed” until after the White House has their way with it.
You can download your very own copy from:
On the left of this home page there is a column that includes “performance planning in A.C.F.” Click on it. Then at the bottom of the next page click on “annual performance plans.” And at the bottom of the next page click “here” for the most recent edition, and download it.
Note: The 6 goals for CSBG and the programs managed by OCS fit into the H.H.S./A.C.F. framework. Most CSBG program activity fits under A.C.F. Goal 3, “to increase the health and prosperity of communities.”
The new IDA program fits under A.C.F. Goal 1, “To increase economic independence.”
ROMA as a process relates to Administrative Goal 4, “to build a results oriented organization.”
C. Office of Community Services. The O.C.S. was one of the first Federal agencies “out of the gate” to implement the G.P.R.A. We come back to the approach used by the O.C.S. to develop the 6 goals and measures and ROM, starting on page 18.
D. The Head Start program has developed 25 performance measures of the effects of the program on knowledge and skills acquired by children and parents, and other typical measures. The extent to which Head Start program nationally is meeting most of the standards relating to outcomes for children and parents will be answered through the FACES study. In this national research project, 42 local programs have been selected in a stratified random sample of all 1,700 programs. Interviews and testing of 3,400 families who are participants in those 42 programs will provide the data. In other words, Head Start is answering most questions about their results and outcomes in changes in the children and parents through a research project that is taking place in only 42 of their 1,700 local sponsors. This whole approach is described in their report, Head Start Program Performance Measures, Second Program Report, which is also IM 98-19, issued November 24, 1998.
Information about other aspects of program operations will be answered though the HSFIS system and the PIR’s, which collect information from all programs, but the crucial questions are being done through a research project and a stratified random sample.
E. LIHEAP. They have developed two performance standards so far, and are asking for feedback from the field for additional measures.
F. TANF. I should say something relevant about TANF here. So I will say that I agree with the article that former H.H.S. Counsel Peter Edelman wrote -- after he resigned from H.H.S. in protest along with Mary Jo Bane and David Ellwood -- in Atlantic Magazine titled “The Worst Thing That Bill Clinton Ever Did.” This is an anti-welfare strategy, not an anti- poverty strategy. Yes, the rolls are declining. In 1966 there were 6 million women on public assistance. And between 1966 and 1970 CAAs help another 3 million more women go onto AFDC. This was one of the best things we ever did. So now we are pushing women back into bad relationships, back to their parents, into makeshift living arrangements, and pushing them out of parenthood.
G. Community Economic Development. Most of the Community Development Corporations, including those funded from the O.C.S. discretionary funds, are involved in the Success Measures Project operated by the Development Leadership Network, Baltimore, MD. The created working groups in ten locations around the country, assisted with only $2,000 in funding per site, to try to create outcome measures for the complex community development and community building projects. They struggled with all the same issues address at USDA, HUD and H.H.S. efforts, but did not use much of the experience from the other efforts. It was more of a “whole cloth” re-invent the wheel type of effort, which of course can sometimes produce fresh new tools. The topics areas in which they worked are: Housing, Business and Job Development, community participation, levels of hardship, and the relationship between funders and implementors. This is still very much a work-in-progress and only working drafts are in print. Contact Susan Naimark, PH 617.971.9443. FAX 617.524.2250 or
In the meantime, a very useful tool is already available from the Virginia CAAs. They developed and piloted this in five agencies. It is the “Community Matrix Focusing on Neighborhood Organization and Civic Capital.” It is a type of scale-and-ladder that shows how you can help a community move from dis-organization to higher levels of functioning. This is VERY useful – it is a major contribution to our field. Thanks to the Virginia CAAs for developing this, and thanks to Ted Edlich and others at TAP for making it available to others.
The North Central Regional Central for Rural Development, Iowa State University, also has a useful publication Working Toward Community Goals: Helping Communities Succeed that combines both community development strategy and reporting of results. It came out of a collaboration between the Aspen Institute, Ford Foundation and USDA. Their web page is:
Also, check out the material produced by the Northern California Council for the Community. Their publications are Goals and Success Indicators of Healthy and Self-Sufficient Communities, and Bay Area Partnership: Building Health and Self-Sufficient Community for Economic Prosperity, Status Report. This is based on use of social indicators and is similar to Oregon Benchmarks, but brought down to a community level.
A VERY useful publication to use to look at all the Federal block programs in terms of their purpose, structure, relationship to states is the GAO publication, Grant Programs: Design Features, Shape, Flexibility, Accountability and Performance Information.
Conclusions. Our whole universe is in motion! All Federal agencies are changing their program and reporting structures. You are being asked to “coordinate” and “plan with partners” but Federally funded program structures are changing rapidly.
Step 3. Get used to an increase in ambiguity and anxiety. It is going to be with us for several years.
D. Where can I go for help locally? What is happening in performance and outcome measurement in city and county government, at the United Way?
The United Way of America has an excellent workbook (five bucks!) on implementing performance measurement systems. It includes an 8-step process with detailed actions to take at each level. It is Measuring Program Outcomes: A Practical Approach. They also publish excellent materials on strategic planning.
The International City Management Association (ICMA) has several publications on performance measurement. One, co-produced with the Urban Institute, is a standard reference work for cities seeking help in measuring the effectiveness of police, fire, solid waste, parks, libraries, transportation, and other municipal services. It is How Effective Are Your Community Services? Procedures for Measuring Their Quality. Another workbook from the ICMA gives a historical perspective, commenting on:
...the resilience of performance measurement as management strategy, noting that a common strand of management thought links the planning-programming-budgeting-system (PPBS) of the 1960's, zero-based budgeting (ZBB) of the 1970's, and management by objectives (MBO) of the 1980's to the current enthusiasm.... As noted by Ehrenhalt, “It is a good idea. It was a good idea in 1943. But it is basically the same idea it was in 1943. It just keeps getting renamed, but...this time, momentum seems to be building....” (Accountability for Performance, p.2.)
Another useful series of publications are from The Evaluation Forum. They have a Field Guide to Outcome-Based Program Evaluation that has a complete overview of designing and conducting an outcome evaluation, and a publication on Managing the Transition to Outcome Based Planning Evaluation that goes into detail on managing the organization dynamics (fear, anxiety, resistance, ambivalence, etc.) involved in the transition.
The Alliance for Redesigning Government at the National Academy of Public Administration is another good source. Try them at:
And if you want to check out the Oregon Benchmarks, go to the Oregon Progress Board at: . And, the National Performance Review, now called the National Partnership for Reinventing Government is at: .
Conclusions: The emphasis on results is taking place at all levels and in all sectors.
Step 4. Find local people to talk to about what they are doing and how they are doing it. There is comfort in numbers -- and maybe some help.
E. Can I go to a local college for help? What about help on outcome and performance measurement from the social sciences? Will my old social science textbooks help?
Social workers, psychologists and educators have thousands of tests and instruments for measuring individual performance and family functioning. Many of them are “normed” based on giving the same test to hundreds of thousands or even millions of people. A qualified person can administer one of these tests to a person or use it to assess a family and tell you exactly where they stand in relationship to the rest of the population that has been tested on the subjects covered by the instrument.
You can definitely get help from a social scientist on:
* how to make your measures valid and reliable,
* how to construct surveys and do sampling, and
* facilitating your process.
However, if you ask for help on content, you are going to get that person’s subject matter imported into your program. It’s the content they have handy, and it is content in which they have faith. It is the place they will go to when they can’t figure out what else to do. If you hire a social worker to help develop the content of your outcome measures, they will tend to develop measures like those used in social work. If you hire a public health expert to help develop your measures, they will tend to use material from public health. Yet, some people hire a consultant from another discipline on the premise that there is some “pure” scientific methodology they can translated into your field and that they do not bring their content with them. I don’t believe it. I think you will wind up adopting their content.
In the early 1990's, a substantial amount of effort went into trying to identify “risk factors” among youth and to associate these with likely use of drugs. Programs then intervened to reduce these risk factors in the hopes that this would reduce drug use. This is classic example of weak theory that reveals only part of the picture -- followed up with weak (low salience) interventions. One gets tantalizing glimpses of bits and pieces of the reality of youth and drug use, but the intervention programs can not muster the amount of re-parenting and social control needed to change how most youth behave or to persuade them to change how they behave.
If you want to see the incredibly complex discussions in the arenas of mental health and public health, look at Assessment of Performance Measures for Public Health, Substance Abuse and Mental Health, by the National Research Council. This is THE book on performance measures for these subject areas.
Micro-economics, political economy, and rural sociology provide some tools to help agencies and programs measure their results with families and communities. Some very interesting new work is being done by rural sociologists that show the relationship between the social system and economic activity in rural areas. Professors Jan and Cornelia Flora at the North Central Regional Center for Rural Development at Iowa State are leaders in this research. Check out their publications.
Other academic fields, including sociology, anthropology and political science, are very difficult to apply to development of outcome measures. Most of the theory in anthropology, sociology, economics, social psychology, and political science is too general to provide tools for immediate use, and they appear to be years away from developing the practical tools you can use to measure change at the local level.
In the physical sciences they distinguish between “big science” and “small science.” The big science is the grand theories, the small science is the use of those theories to accomplish certain tasks. In the social sciences, we have a lot of big science, but not much small science.
Conclusions. Unfortunately, most of the social sciences provide little assistance in developing the content of outcome measures at the community, family or individual level. If it was easy, you could pick up a book and plug in the measures.
Step 5. Avoid getting lost in the gap between big science and small science. Don’t try to pretend that the small science is there when it really isn’t.
Some other ideas to avoid this problem are given in Section L, page 40, Approaches to Developing the Content of Outcome Measures.
F. How is ROMA the same as or different from previous program structures and reporting systems used by CAAs?
There have been six eras of program structures and reporting systems that were permitted or required by the Federal Government. The elements of those systems relevant to implementing ROMA are described briefly below.
1. 1964-1967. Everybody Does Their Own Thing. The Community Action Program was created with the passage of the Economic Opportunity Act in August, 1964. The largest portion of funding was called “local initiative,” where funds were made available to locally designed programs and projects “that were developed with the maximum feasible participation of the poor.” Initially, local groups sent a letter, a brief narrative and a few budget forms to OEO describing their effort – and they were funded based on that local project description. The reporting of progress and results was done against the material that had been submitted in the application. Reporting was done quarterly, and was largely anecdotal. There were no mandated forms or procedures. The local program structure was whatever the CAA said it was. There were frequent and frantic calls from HQ and the Regional Office to CAAs to determine the number of people being served in various types of programs. Every Congressional inquiry generated a unique data collection process. After only three years of operating like this, it was generally recognized that it was difficult:
* to describe the results of what any one agency did in language that was easily understood by others,
* to summarize the results across agencies,
* to measure quality or progress because there were no generally recognized standards.
2. 1967-1974. Grant Application Process. In 1967, OEO instituted the Grant Application Process, or “GAP.” It was developed by OEO staff, CAA staff, and McKinsey and Company. GAP asked a set of standardized questions about the activity and budget of any project to be funded, although the selection of the purposes and activities of each project were left up to the local agency.
A set of about 150 “program accounts” were developed to create standardized descriptions of what was happening at the local level. For example, Program Account 29 read something like “Adult Basic Education, to assist adults in acquiring reading and writing skills up to the 8th grade level.” Program account 30 read something like “G.E.D. Prep, to help adults get up to the 12th grade level in reading, writing and 1rithmetic – so they could take the GED exam.” In the spirit of local initiative, local programs could still vary their operations for the pre-printed definitions, and at the end of every major category (e.g. education) there was usually a blank called “other” so if you could not shoehorn your description in to one of the existing program accounts, you could make up your own. However, the program account structure made it possible to add up the results of different agencies that were operating “similar” programs. Also, the program accounts often grew into separately funded programs of their own, e.g. foster grandparents, family planning, senior meals, senior employment, etc.
Two new forms, the CAP 84 and CAP 85, asked for detailed information about program activities and program participants. OEO staff were besieged with inquiries from the press and from Congress asking “How many left handed people under five feet tall got services from ABC CAA south of main street in town X last month/quarter/year?” This example exaggerates the type of inquiry a bit, but not much. OEO’s thinking went something like this: “If we require that the CAA report the basic demographic characteristics of every person every CAA comes into contact with, we will be able to answer these questions.” Unfortunately, this was before the age of computers, and boxes and boxes of these forms literally piled up in the halls of the Federal offices, with no way to deal with them. Also, there was little use for the data at the local level. OEO came to it senses and realized that (a) the time it took to collect and compile this was not worth it, and (b) the answers to these types of questions were not found to be useful to CAA staff, and (c) if it was really needed it could be quickly determined from a SAMPLE of the records in an local agency, so OEO stopped requiring the demographic data be collected but continued using the program accounts.
3. 1975-1979. Standards of Effectiveness. Late in 1974, with the Assistance of OEO staff, Congress created 5 Standards of Effectiveness” against which program activity and results were to be reported. The Community Services Administration (C.S.A.) the successor to OEO, consolidated the program accounts into two accounts, which were “01, Administration,” and “05, Programs.” Congress seemed concerned with the administrative cost ratio, and this system was set up to report that. The Community Services Administration asked for reports on how CAAs were doing in terms of the Standards and what they were doing in the two program accounts.
A partially‑developed management‑by‑objectives system tried to fill the gap (pun intended) between the so-called Standards and the new "non" program accounts, but failed. So there was no traceable pathway between local activity and the results that were supposedly produced. The infrastructure to make this work just was not there.
There were many complaints from Congress and from the "management types" in the Federal government about the poor quality of management in CSA and CAAs. Some of this was baloney and some of it was real. In 1976, President Carter commissioned Charles Aragon to head a task force to review management capabilities and systems in CSA and the CAAs. It was the age of zero-based budgeting (ZBB). The "Aragon Report" presented President Carter with three options: (1) terminate the whole apparatus completely, (2) merge it with the Social Services Block Grant, and (3) re-tool it. President Carter -- the engineer -- checked box number 3, re-tool. (An interesting report, worth reading in its own right.)
The “re-tooling” of the anti-poverty program took place from top to bottom, and was to require that all CAAs and the Federal agency (CSA) use an integrated planning, management and reporting system. In 1977 and 1978, CSA invested heavily in staff time and consultants to develop the Grantee Program Management System (G.P.M.S.). Ten CAAs (one in each region) pilot tested the new system and it was improved based on that effort.
4. 1979-1981 and beyond. The Grantee Program Management System (G.P.M.S.) Implementation began in 1979. Each CAA created a multi-year strategic plan, in which they described the causes and conditions of poverty, and -- continuing the tradition of local initiative -- described their locally selected goals and objectives for addressing those causes. It was generally perceived that any change at the community, state or national level took several years worth of effort, and only changes affecting a few individuals might be accomplished in any one year. C.S.A. would make a commitment to the multi-year plan, which amounted to a commitment for multi-year funding. Then, the CAAs would report on the result and outcomes of their efforts. The C.S.A. staff were going to synthesize the results of local operations as reported by all CAAs into state and national reports. And, an impact evaluation (wow!) would be done by each CAA on at least one program every three years.
In designing GPMS, we had learned from the past. There was an explicit recognition that the front‑line managers had to own the information system and the information. They had to benefit directly from it. The GAP client‑characteristics requirements and the post‑GAP Standards of Effectiveness had not been seen by CAAs as being relevant to or needed for day‑to‑day operations, so the people at the CAA level did not produce much useable information.
The GPMS was a system that people at the CAA could use to do their jobs better. The GPMS was largely implemented in Regions 1, 4, 6 and 7, and partially implemented in Regions 2 and 8. Regional 3, 9 and 10 were still at the starting gate on GPMS when the plug was pulled by the passage of the block grant in July of 1981.
5. 1981-1991. The States Begin Administering the CSBG. In July of 1981, President Reagan was successful in persuading Congress to repeal the E.O.A. On September 30th, 1981, C.S.A. was closed and all 1,000 employees were laid off. On October 1, 1981, the Community Services Block Grant was born. Brilliant lobbying by David Bradley, Charles Braitwait, Bob Coard and others -- and some serious horse-trading -- resulted in 90% of the CSBG funds being passed through to continue funding to the existing designated entities (including all CAAs), and the tri-partite board structure was maintained. The Office of Community Services was created to administer the new CSBG at the Federal level, and to pass the funds through to states. It hired about 35 people, including some who had been laid off with the closing of C.S.A.
The G.P.M.S. was the “system in use” in Regions 1, 4, 6 and 7. The GPMS or portions of it were picked up in a few other states, but the staffing needed to make it work as a comprehensive system was beyond the capacity or interest of most states.
Funds that had been placed at the Institute for Local Self Government (ILSG) in Berkeley, CA for the implementation of G.P.M.S. were reprogrammed in the closing days of C.S.A. to assist the states in making the transition to their increased role in administering the block grant. A “road show” was sponsored by the National Governor’s Association to help states develop their approaches, and the CSBG presentations were paid from these carry-over funds. This author was hired to do the road show (and as the project director for the subsequent survey, and as Project Director for development of the N.V.R.S.). The “new” OCS staff, including the Reagan appointees, agreed that knowing what the states were doing with the CSBG funds would be useful, and approved use of the carry-over funds stored at the ILSG to conduct a national survey of states, which they would complete voluntarily. The results of the first survey were found to be useful by the OCS and the states, so OCS re-funded the ILSG and its subsidiary, the Center for Local and Community Research (CLACR, Berkeley, CA) to work with the states on development of the voluntary reporting system. A working group of state directors and consultants was formed to guide development of the new system. H.H.S. career civil servants, NACAA and NCAF were consulted at each step.
In designing the National Voluntary Reporting System (now the CSBG IS), there were two basic options. One was to continue the reporting system designed in GPMS, to report inductively and to synthesize, at the state and national level, the results being produced in each CAA. CAAs would pick their own results measures. The second was to deduce new information categories from the CSBG. Although the first option was by far the strongest in terms of actually assisting in the work of CAAs and building off of the investment in GPMS, the working group concluded that the new reporting system had to show Congress what was happening in terms of "their" new legislation -- the CSBG -- and it had to do it quickly. It was decided that it was desirable to “find a home” for as much local activity as possible as fast as possible in the new CSBG, so a reporting framework was created drawn deductively from the CSBG language itself.
The working group reviewed the information systems from the Older American Act, the SSBG, Head Start and a couple of other social service programs and asked: "What kinds of information does Congress look for from social service programs." At the time, they were asking primarily about inputs, outputs and about client head counts. So that is what we gave them. We reviewed whether additional detailed client information was needed to improve services to people or because Congress really used in decision making and the unanimous answer from the working group was "no!"
A “taxonomy” or set of definitions was created under the leadership of the National Association of State OEO Directors, hence renamed the National Association of State Community Service Programs (NASCSP). The taxonomy was reminiscent of the old program accounts. It clustered different types of existing program activity under major headings, and provided for “other” to be reported by the CAAs in each major category. It was an intentional effort to put the old wine into new bottles. We were trying to legitimize the activity of CAAs under the new CSBG language.
This was a sensitive time. Some states backed President Reagan’s position that the entire CAA network should be abolished, and chose not to report on the activities and accomplishments of the C.S.B.G. To emphasize the purely voluntary nature of the reporting process, it was named the National Voluntary Reporting System.
After two years of reporting on activity at the State level, in 1984 it was agreed to add a set of standardized survey questions that states would ask CAAs to complete. At the request of those who were telling the CSBG story to funders, requests for anecdotes about individuals and families were added to the system. These provided a compelling personal dimension to the statistics. Over the next two years, the system continued to evolve and the number of states participating continued to increase. In 1987 the system was re-named the CSBG Information System, and the National Association of State Community Services Programs (NASCSP) took over management.
The major point of this explanation is that this developmental process took about 5 years to create the new program structure and the new reporting framework. The system was working to provide information about CAA operations in terms of CSBG categories, as evidenced by a letter from David Bradley to Jim Masters on June 30, 1987, which stated:
“Your objective data has been essential in preparing reports for Members of Congress and key Committees on the range and number of services provided to low-income families. Information that your organization has gathered and compiled has been used in the development of public policy and legislation as well as in program management.”
And, the CSBG reporting framework we had developed was starting to wear out. There was a low-key but across-the-board shift taking place in the late 1980's. Congress was asking fewer questions about inputs and client demographics and more questions about results in programs such as Head Start, senior programs, employment and training programs and drug programs.
Congressman George Miller’s subcommittee was also using evaluations of programs to generate increases in funding for WIC and Head Start. H.H.S./O.C.S. did an excellent job of responding to this type of interest by requiring evaluations of each of the projects in the Demonstration Partnership Program, but did not extend the evaluation concept into the state-level CSBG program through T&TA, through the assurances or other methods. This is still needed. It does not take very many evaluation reports to legitimize a program – my recollection is that WIC had only about 6 evaluation reports from local programs – but it takes more than zero.
It appeared that the existing reporting system had served its purpose of bridging CAA activity into the “new” CSBG, but that it did not have the capacity to answer the new questions about results that members were starting to ask. It needed to be updated to respond to the increasing interest in results. In the late 1980's, H.H.S. and NASCSP were faced with several pathways.
One approach was to request additional demographic information about individuals who were participating in programs. Another possible approach was to develop a new system to find out more about the results of our work with individuals. Another was to encourage or to generate more evaluations. Another approach could have been to more fully describe the strategies in CSBG other than services to individuals.
Charles McCann, the Missouri State CSBG Director, illustrates the differences between working with people and working with community by using a metaphor about the difference between the fish and the pond. “Working with individuals (goals 1 and 6) is like working with the fish. Working at the level of social or economic institutions and at the community level (goals 2 and 3) is like working to improve the pond. And, increasing our capacity to do both of these types of activities would take place under goals 4 and 5.”
Ideally, H.H.S. and NASCSP would have started work on all four of the above approaches. But, as a network, we just weren’t ready or able to do all that. We started work on # 1, to collect additional demographic information on individuals. (From a one-year perspective this looked like a big mistake. In a five-year framework it looks more like an opportunity that was not seized. In the thirty-five year framework it looks like a mild slowdown in our development.) The collection of demographics increased the local reporting requirements. It reinforced the dynamic in those states where the state agency wanted to see a specific recipient of a service attached to each CSBG dollar spent. It made CSBG look more like the other service delivery programs and the income maintenance programs, because attention was focused on individuals who received service, on “client characteristics.” Ironically, the new client data requirements are very similar to those (GAP) that were dropped by OEO in the 1969 because it was discovered the data did not help improve local program operations. (To the extent that client demographic data are needed for statewide or national totals, it can be obtained by sampling from CAAs and programs. If Gallup can do a profile of what Americans believe, accurate within plus or minus 3%, with 1,300 people it seems unnecessary to collect data on millions of program participants if the data has no local use in enhancing participant outcomes. Data collection for its own sake absorbs time and energy that should be used to do other things.)
The conceptual trap here is the idea that we have to describe the total number of people helped by the CSBG. “How many people did you help last year?” seems like a reasonable enough question. And, trying to answer it seems like a reasonable response. We are trapped in our own perception that we have to answer this question to describe or justify the CSBG, and that the answer is complete enough to stand on its own. The problem comes from the fact that while we can count the numbers who receive services or stuff, we have no comparable measures for counting the number of people who benefit from institutional change efforts. So we are in a situation where a part of what we do is presented as or perceived as the whole of what we do. The people who ask, “How many people were served by the CSBG?” are asking an incomplete question, and when we answer “Xteen million people were served by CSBG” we are giving an incomplete answer. We are creating a problem for ourselves. We need both to help them rephrase their question, and our answer needs to go beyond the number of individuals helped.
“Please ask me both ‘how many individuals and families did we help, AND how did we improved the community in other ways?’”
“And my answer will be: ‘We helped:
1,245 individuals in the following ways...,
346 families in the following ways...,
and we improved the community in the following ways...,
and we improved agency and system capacities in the following ways....
And all those are needed to fully explain what we accomplish with the CSBG.’”
Ironically, as we re-balance our activity to devote more resources to the other four goals, the total “numbers served” will go DOWN, because we are devoting more resources to efforts in which we do not yet have ways of representing numerically the people who benefit from the activity. But, I’m getting ahead of the story. Back to how ROMA evolved.
It is only with the arrival of G.P.R.A and ROMA that attention and ability of the network as a whole has expanded to include development of additional measures to describe the work that CAAs do on “the pond.” And, we still have much work to do to “get credit” for the important and good work we do there.
Conclusions from the use of the first five management systems. The collection and use of information about the type of local activity and who benefits is inherently a local managerial function. Every front‑line supervisor and program manager should have an array of tools for monitoring staff performance and measuring individual and program effectiveness and efficiency.
There is no way to measure the outcomes or impact of programs without the enthusiastic commitment of front‑line managers and workers to collect information on what happens to whom, and why. If the manager believes that the information collected is useful in terms of (a) providing quality services to participants, and (b) carrying out their job, and (c) pursuing the mission of their agency as they perceive it, then they will collect the information ‑‑ and use it. If they do not perceive the usefulness of the information, then they will tend to not collect it at all or will delay until the last minute and then do a cursory job of it.
Previous efforts at information collection within the CAA network that have proved this point.
Step 6. Start building a list of criteria you will use to create your new results measures and you may as well add your opinions about issues like these to your list.
Now that we have gleaned a few lessons from previous management systems, we continue to look at how ROMA was created.
6. 1993-2000. The Development of and Shift to the New Approach. In 1993, Congress passed the Government Performance and Results Act, requiring all Federal agencies to develop strategic plans by the year 1997, to report on the results and outcomes of each program, and to begin requesting funds from Congress using those strategic plans and outcomes by the year 2000. This represents a fundamental shift in how the Federal government administers programs. It was fairly clear how this was to work in programs actually operated by the Federal government, e.g. the Army Corps of Engineers, the U.S.D.A. rural development programs, and the Social Security Administration. It was much less clear how this was to work for block grants.
At this same time, several state governments and state CSBG offices were also grappling with the results and outcome issues, encouraged by political leaders or a citizenry who want them to “re-invent” government and make it more effective and responsible.
The H.H.S./OCS responded by creating the Monitoring and Assessment Task Force, and inviting it to develop goals and results measures for the CSBG. The membership was drawn from all the stakeholder groups in the CSBG network. This is a very important point -- the membership of the MATF was designed to represent all the institutional interests in the network -- the individual CAAs, NASCSP, NACAA, NCAF, and OCS. This is as good a partnership team between Federal, state and CAA as I have seen in my 33 years in community action.
The OCS, under the leadership of Don Sykes, Peg Washnitzer and Mae Brooks, deserves high praise for creating a highly participatory process and for taking the time needed to shape a consensus.
Through a sustained process, the MATF designed the six goals and identified possible measures. The MATF created the Results Oriented Management and Accountability principles and systems as a workable and best available approach to provide a foundation for CAAs to use in their anti-poverty work for the next decade.
As is the case in any gathering of community action folk, there were substantive and strong disagreements among the participants about the causes of poverty, about what strategies work to reduce poverty, and what CSBG should be used for, and about what CAAs should be doing. Most CAAs employ people or have board members who have very different belief systems. This has been true since the original legislation “pulled in” people who believed in very different theories, strategies and program approaches. This is an important point – that the Economic Opportunity Act of 1964 was a big tent that incorporated strategies that assumed that the problem and the “fix” for poverty rested with (a) the economy (wages and access to work), (b) social values (racism, legal injustice), (c) the adult individuals and families (family development and motivation), and (4) children and youth development. The fact that these strategies were based on different assumptions about how our society works and were logically inconsistent with each other did not trouble the authors of the legislation or the social movements that were part of OEO -- and these conflicts have persisted within public policy and within the CAA network to this day.
At the MATF meetings, one person would argue that all change begins at the individual level, and only work with individuals is valid. Another would argue that only change at the national level make a big impact, and our effort should be focused on public policy. Others argued that the new approaches in community building and community development were “where it’s at.” And others argued that our primary role was in providing for basic needs of food, shelter and clothing for those who do not have enough. There were also healthy disagreements about whether our work should be focused at the national, regional, state or local levels.
If you made a chart of all this -- on one axis there are the institutional stakeholders, and on the other axis there are people with these various theories, and on the third dimension the levels at which we should be working -- you get a real “Rubik’s cube.” And, there is no particular relationship about where a person is ‘centered’ on their institutional location and their theory about poverty. This made for a very exciting process!
The MATF works mostly by consensus. Why do I repeat that? Because the six goals and sample measures developed by the MATF represent a CONSENSUS among both the institutional interests and the individuals who have the different theories of poverty and anti-poverty work. (And, the measures represent a compromise. More on this later.) The six goals span and provide a place for the various theories of poverty that have existed since the inception of CAAs and that exist today.
Goal 1. Low-income people become more self sufficient. This is for the folks who think that solving poverty one person at a time is what we should be doing (self sufficiency for individuals and families). The focus here in on increasing household income.
Goal 2. The conditions under which low-income people live are improved. This covers changes in social or economic systems or at the community level, and includes strategies such as community building and community development.
Goal 3. Low-income people own a stake in their community. This covers empowerment and civic participation, and includes strategies such as Dialogue 2000.
Goal 4. Partnerships among supporters and providers of service to low-income people are achieved. This seeks to build the partnerships and network capacity to do what needs to be done.
Goal 5. Agencies increase their capacity to achieve results. This seeks to build our individual agencies.
Goal 6. Low-income people, especially vulnerable populations, achieve their potential by strengthening family and other supportive systems. This covers human development (e.g., Head Start). It also provides a place for those who want to focus on child and family development, working with senior citizens, or on basic needs by giving people stuff.
The implementation and use of the new goals began in 1998. The agreed upon approach was that states would begin collecting information from CAAs and reporting on at least one measure under each goal. This M.A.T.F. decided against mandating any measures at this point. This is similar to the approach used in the G.P.M.S. system in 1979 -- 1981, which allowed CAAs complete flexibility in selecting measures on the premise that their activity could be compiled into state and national reports. The compromise was that “every measure is as good as any other measure,” or “all measures are equal” in terms of acceptability in reporting on progress toward the goal. This compromise came in part because we were running out of time, and energy, and did not really know which should be mandated from the start. The idea that “any measure will do” seems like s short-term solution at best. This is obviously only a starting point, and as time passes additional measures should be agreed upon and adopted for use at the state and national levels.
Further movement toward outcome measurement. The 1998 Amendments to the CSBG require that ROMA or a comparable system of measuring results and outcomes be used, and that all States and CAAs develop such a system. The reality is that unless an agency, or group of agencies, or a state wants to invest a lot of time and effort to develop an alternative system, ROMA is – by default – going to be the system that gets used.
As long it is possible to develop an alternative if ROMA is no longer working for you, then use of ROMA probably is the best approach to start with. (All management systems have a limited life and rot away. Remember PPBS, ZBB and MBO?)
Conclusions. An enormous amount of work and compromise went into the creation of the six goals. They provide a “home” from the diverse and often conflicting theories about poverty that exist in the C.S.B.G. universe. The goals provide a home for all approaches to poverty. The goals themselves do not favor any one approach over any other. The six goals are the best framework for you to use to move forward in developing outcome measures. The selection or development of measures is in its infancy. Eventually six to ten standardized measures should be adopted for each goal at the state and national levels. There should always be the option of a CAA using additional , local measures.
Step 7. Use the six goals and the initial measures as the basic framework for the future.
At the local level, relate your existing activity to each goal. Suggestions on how to relate existing strategies to the six goals are found in Appendix B.
At the state level, the CAAs and the State Office should reach agreement on a limited number of measures under each goal and require all agencies to report on them
At the national level, the six goals are the public face and the official tally of CSBG activity. They should be used to position the CSBG in the web of the other human development, community development and public charity programs. Either the MATF or the CSBG IS Committee should select a limited number of measures under each goal and require all states and CAAs to report on them. The answer may be “zero” or “none” for your CAA, but there should be a set of standardized measures for all the C.S.B.G.
You must work both individually as an agency, and collectively as a network at the state and national levels, to design the new system. Form working groups at the local agency and at the state level to implement ROMA.
G. How “wide” should the local implementation effort be? Across which programs should ROMA be used in a CAA?
You have to figure out the scope of ROMA usage. Is ROMA just for the CSBG? Or, for ALL programs your CAA operates? Or for – some of them – but which ones? The answer to these questions comes in part from your purpose and mission. Are you a comprehensive local agency that seeks to weave everything together? Or, do you run programs on a stand-alone basis? You have several templates you could use for ROMA implementation.
A. Some states try to reinvent their whole system, public and private. The Oregon Benchmarks are an example of a state-wide state level effort.
In Oregon, the move toward an outcome orientation came from a number of sources. The state legislature adopted a set of 259 benchmarks to assess the progress of the state toward strategic goals. These “Oregon Benchmarks” became the basis for establishing budget priorities within the state government. This statewide focus led all organizations and programs receiving state funds to look at the results of their services, not just CSBG or CAAs. This environment helped move forward the process of adopting an outcome orientation. (NASCSP, page 14)
B. Or, perhaps a whole sector in a community decides to try to change its entire service infrastructure. Portland, OR, Sunnyvale, CA, Indianapolis, IN and Broward County, FL are examples of county-wide or city-wide system change efforts. These preceded ROMA and were driven by elected officials and/or the City Manger, but they show that change can take place on a system wide basis. Doing this requires major commitments of resources.
C. An agency-wide transformation. The approach here is to transform the entire agency, from its corporate culture, vision, mission, values, goals and certainly the relationships between its programs. Jack Burch, Executive Director of the CAC for Lexington-Fayette, Bourbon, Harrison Counties (Lexington, KY) and Eleanor Hunneman, former Executive Director of the Bucks County Opportunity Council (Doylestown, PA) have led efforts like this under ROMA. Jerralyn Ness, Executive Director of the Community Action Organization (Hillsboro, OR) led an effort like this before ROMA arrived.
D. An agency-wide but phased outcomes development process to upgrade reporting and planning systems. This would involve all programs, although it does not seek the level or degree of transformation as the preceding example. Most agencies that do this will do so in stages, with a several of the programs making the shift each year. I recommend that you use the six goals to provide the framework for the entire agency.
Ask your board and your funders, or propose to them that you can better show the results you are producing and the relationship between programs if you use ROMA across all your programs -- therefore you want to use ROMA across the agency.
If you have 107 programs, you can not hook all of them together under ROMA at once. Develop a phase-in schedule. Even if you are running “only” 10 or 20 programs, two or three years seems like a reasonable period of time to shift your programs into a ROMA framework.
E. A few programs are involved in developing outcomes.
F. A single funding stream, perhaps just the C.S.B.G., has outcomes developed for it.
G. One program or a part of one program is converted, for example, the family development program in C.S.B.G.
H. One staff group or team makes the shift.
I. One person makes the shift.
J. Nobody makes the shift. Numbers are invented to keep the state and peer agencies happy.
Conclusions. Where you start in the magnitude of your effort is a major decision. Start as far up daisy chain as you have horses to carry the load. Use the six C.S.B.G. goals as a framework for all your program activity. Just as H.H.S. has a plan that includes all of A.C.F., and A.C.F. has a plan that includes all of C.S.B.G., you can develop use the C.S.B.G. framework to encompass virtually all program activity. The six C.S.B.G. goals create the umbrella of the future for a CAA.
Step 8. If your agency wants to continue the tradition of the CAA as an umbrella agency, then use the 6 goals as the agency-wide umbrella framework for all your strategies and programs. Go for an agency-wide, phased implementation process.
H. Who should lead this effort?
This brings up the subject of leadership. Any change from the program level on down to the individual level (E through J, above) can be led by one or two staff people. Any change that is agency wide or community wide (From D to A, above) will require concerted leadership from the CAA Director and top managers. It takes champions for big changes to happen!
Conclusion. If you are thinking about agency wide activity, it takes champions for the effort at the highest levels.
Step 9. Check to see if you have the right scope of leadership for the level of change effort you are undertaking.
I. PROCESS approaches to development of outcome measures.
There is an argument to be made that the process you create for this change effort is much more important than the content you create, because the content is going to evolve over a three to five year period. Four examples illustrate some of the possibilities in developing outcome measures.
1. Every CAA Does Its Own Thing. The state takes a hand’s off approach – and lets each CAA develop whatever it wants.
Provides maximum local latitude.
Unable to relate each CAAs activity to other CAAs.
No systematic connection of CAA activity to programs in state government.
Slow – everybody has to reinvent their wheel.
Reporting on state level or national level activity is difficult.
Eventually you have to address the need for commonality, but by then everybody is defending their definitions.
2. The State Office Mandates the System.
Fastest way to implement.
The virtues of standardization -- to the extent they actually use it, every CAA does it the same.
May help connect the CAAs to other programs in state government
Categories may be arbitrary.
Does not reflect the real world as understood locally.
New definitions must be learned by everybody; training costs go up.
Resistance to “state bureaucracy.”
3. It is a group effort, but factions form among the CAAs, and one group is allowed to win.
One faction consists of the “lumpers” who want to combine a lot of types of activity into very general measures, the other consists of the “splitters” who want to develop detailed and very specific measures. (These types of factions also form among linguists, and among ethnologists who study race.) The two factions reach a stalemate. One faction outlasts the other and “wins” and imposes their definitions on all CAAs. The losers have a serious case of sour grapes and evade use of the new system as much as possible.
It feels like a victory.
Yet another schism.
The credibility of the system is continually under attack.
4. A Partnership Implementation Process takes place involving the State Office, the state CAA association and the CAAs. (A PIP!) After argument and controversy, a consensus is reached.
Input from all concerned.
Most people will be happy with the result
Reduces implementation costs because people understand it and support it.
Old issues will surface and must be addressed or set-aside for progress to occur.
Power struggles can develop.
Development time and costs may go up.
The question is, how can externally imposed requirements for information be implemented to produce real improvements in programs for low‑income people? The answer is that the CAAs themselves must embrace the requirements as new tools for (1) understanding the community as a whole and the individuals and families with whom it works, (2) improving their personal effectiveness in their assigned tasks, (3) enhancing their management capacity, (4) achieving their mission, and (5) making their case to the public and other key stakeholders. The CAAs have to feel like they own the reasons and mechanisms for producing the new information. This means a leadership role for any new ways of assessing effectiveness or reporting on results must come from within the CAA network itself. Fortunately, this is an approach that is permitted (but not required) by H.H.S/OCS, by NASCSP and NACAA in designing the new system.
Conclusions. Eventually you are going to have to some standardized definitions and measures that everybody agrees to use. Start now, otherwise you have to go back and do it again. Differences of opinion and conflict are a normal part of the design process. The goal is as much consensus as you can muster out of the entire system.
Step 10. Create and sustain a partnership implementation process that involves all CAAs. Join with the other CAAs in your state, and the state office, to develop common definitions and measures. Don’t avoid doing this because there is a likelihood of conflict.
CASE STUDIES. Now let’s look at some specific examples of approaches that were initiated in various states. These are from a very useful report prepared by Julie Jakopic at NASCSP in 1996 which showed several different initial approaches being used by states, CAA associations and CAAs. This report is on the NASCSP web page, These descriptions are included here to illustrate different types of initial approaches. They are not intended to be a current description of what is happening now. Contact each state for current information.
CASE STUDY # 1: PENNSYLVANIA
Pennsylvania started with an almost philosophical approach, helping people acquire “outcomes thinking” through training at their “outcomes university.” The approach started with an analysis of the underlying attitudes and philosophy of service delivery, and sought to create “outcomes thinking” in all aspects of agency operations. This more general approach worked well for them. Their products are up on the ROMA web page. The rest of this page is an excerpt from the 1996 NASCSP Report (which is on the ROMA web page an in the ROMA Guide.).
Pennsylvania approached the issue of results-oriented management and accountability from a unique direction, with the emphasis not on particular goals or measures but on making outcomes and “outcome thinking” part of how agencies conduct their business. Beginning with a two-day statewide conference held in November of 1994, the state embarked on a process of ensuring that agencies not only began to look at what happened as a result of services, but to integrate an outcome orientation into all facets of how the agency performed. The primary goal of the training was to help agencies move from a program or service-oriented management process to one that focused on outcomes, across programs and services.
Subsequently, the Pennsylvania state office, through a contract with the state CAA association assessed not only what training and technical assistance agencies might need to implement an outcome-based management system, but also what skills and resources they already had. The agency assessment looked at different dimensions of agency management:
1. Outcome thinking - the degree to which the agency has moved in shifting from a service oriented management process to an outcome-oriented management process.
2. Applied technology - the degree to which an agency uses outcome measurement tools.
3. Management implications - the extent to which management practices have been adapted to handle outcome information.
4. Board communication - the extent of the Board of Directors’ awareness of, support for and ability to communicate with the staff in outcome concepts and language.
5. Internal outcome application - the extent to which outcome concepts are incorporated into the day-to-day functioning of the agency.
6. External outcome application - the extent to which outcome language and concepts are used in the communications the agency has with its community partners.
7. Staff development - the extent of training provided to staff on the numerous areas of the transition to outcome-based management.
8. Applied communications - the consistency with which an agency uses outcome language and concepts.
(NASCSP, Page 16 -17, 1996)
Contacts: John Wilson, PA CAA Directors Association, 717. 233.1075
Dennis Darling or JamesEtta Reed, State CSBG, 717.787.1984
CASE STUDY # 2: NEW YORK.
New York. “One of the ways the CSBG office in New York sees its role is to make it easier for CAAs to assist low-income families and communities. The focus in New York was to create an environment in which the local agencies could build on what they were already doing, but begin to change the way they thought about their work by looking at the outcomes of their efforts. In many ways, New York used a conservative strategy to implement a results-oriented approach. As a starting point, for 1996, local agencies were asked to tie the narrative descriptions of success that they report, to one of the six goals developed by the MATF. While there was no other major change in 1996, agencies had the chance to work with the state office on what changes would be required in later years.
For 1997, agencies were asked to select measures for the MATF goals they would use to measure success, on an agency-wide basis. Reporting on agency level measures represented a major shift from looking at results program by program to looking at the sum total of what the agency as a whole accomplished. Each agency could select any measures they wanted, but were asked to try them for two years. This would make it possible for each agency to establish a baseline for their measures in year one and compare them to year two. If a measure proves to be unworkable after the second year, it can be changed.
Agencies were provided training on outcome measures and results-oriented management and accountability in preparation for selecting measures and preparing their community action plans. Also, for several years the state had been working with the CAAs in the area of family development, creating a Family Development Specialist Certification Program. This program placed an emphasis on what goals were achieved by the families assisted and led the way for agencies to begin to think about what their other programs achieved.
As in all states, there was still some initial resistance to implementing an outcome orientation. There seemed to be three primary concerns: tracking outcomes takes resources and training, difficulty dealing with reporting not just on CSBG funds, and being concerned that their level of effort (previously captured in units of service) would no longer matter. The state has worked closely with agencies around these concerns. Training has been provided. Also, by looking at units of service in conjunction with results, agencies have begun to explore describing their work in terms of return on investment; i.e., what was achieved for a particular level of investment.
The reciprocal sharing of data in New York has also helped in overcoming resistance. The agencies in New York provide fairly comprehensive reports to the state, however, they make these reports annually. From these annual reports, the state then can prepare its annual Management plan and respond to requests for data from other state agencies, state legislators, the National Community Action Foundation, and even the annual Community Services Block Grant Information System Survey conducted by the NASCSP. The state also compiles a report that is provided to agencies so they can see how they are faring, relative to their peers. This makes the data collection effort, while comprehensive, useful to the agencies and ultimately streamlines the time they spend reporting information.” (NASCSP, pp 14 –15)
Contacts: Evelyn Harris, State CSBG, 518.474.5741.
NY State CAA Assn, Bonita Hagan, 518.238.1955
CASE STUDY # 3: WASHINGTON STATE.
Washington State. “Convened a volunteer work group composed of one third of the state’s community action agencies, two state CSBG staff, an instructor in program assessment from Seattle University and the Executive Director of the state CAA Association. Meeting over a two year period, developed a workbook of CAA programs. Each CAA program description in the workbook includes a clearly defined goal and outcome with a list of acceptable methods for collecting outcome information. Methods include standards for how large or small a sample should be taken to be valid.
Using the workbook, eliminates additional expense to the CAA since it identifies outcomes from its existing data collection system. The new system also creates the basic infrastructure that CAAs can use to develop a program mission and clear, reasonable statements of long-term goals to which performance will be assessed.” (NASCSP page 23 - 24)
Bob Swanson, State CAA Assn, 206.527.7336
Denny Naughton or Ed Barton, State CSBG, 360.753.4979
I hear that they have done a lot more in Washington.
I also had a good description of what they did in Tennessee. I downloaded files from someplace –and have lost them.
CASE STUDY # 4: MISSOURI
Missouri concentrated on developing one particular type of outcomes – for their family development and case management program -- in an excellent partnership approach to outcome measure development. The Missouri Association of Community Action and all 19 of the CAAs in the state were involved in a process to develop their version of “scales and ladders” for Case Management and Family Development. The product can be viewed from the web page, Select the About MACA” box. At the bottom of the page select the “Publication” box. Then select “Fact Sheets” then “Outreach.”
Their product is interesting. They use 12 scales, each of which stands on its own. In other words, they do not try to “add up” the results of the different scales. Each scale is used by itself. This largely solves the problem of trying to load too much responsibility for producing self-sufficiency onto the family development approach.
About 90% of the CAA employees had never used a measuring tool before. They designed it so that it could be read and understood by a person with a GED, and so that a person could be trained to use it in one day.
In my opinion, their process was truly outstanding. At the start of the process, each of the 19 CAAs had their own version of case management and of scales and ladders. Only two or three of the CAAs had done testing of the validity and reliability of their scales. Because each system was unique, they did not understand each other’s definitions or scales. Their funding sources were questioning how the CAA could justify carrying a program participant over from year to year. The CAAs decided they needed to develop a statewide program that provided a rationale for continuing to work with a person on a multi-year basis.
The CAA outreach staff – the people who would actually use the measurement and reporting tools – led the organizing effort! They sought and obtained support for the project from State CAA Association Executive Directors meeting, the state CSBG director and the State CAA Association staff. The met several times, usually for two days, over the next 24 months. They had hoped to do it in 12 months, but it took 24. The Missouri Association for Community Action, MACA, played a key role in organizing and supporting this process of consensus building.
All 19 of the CAAs agreed to the project and process. Twelve of the nineteen CAAs attended most of the meetings. A total of 45 people were involved. They utilized staff from CAA Outreach and Case Management programs, Head Start, Supportive Housing, and the HUD funded Self Sufficiency programs that operate in public housing. The MACA got a small T&TA grant from OCS, and the State CSBG office also contracted with them for this work.
Staff turnover created a problem with continuity of the work. A neutral facilitator from the University of Missouri Extension Service helped them manage the meetings. About halfway through the process, they brought in a social science professor to help deal with the validity and reliability issues and to help resolve differences of opinion about how broad or narrow each measure should be. One group wanted “soft measures that could be used in a wide variety of circumstances. Another group wanted very specific measures that measured one thing at a time. As is often the case – after full and frank discussion -- they reached an acceptable compromise on each measure. There was disagreement over whether they should test the product, and about the degree to which each measure must be valid (it actually measures the thing you are trying to measure) and reliable (it works pretty much the same way over time and in different locations). All this had to be talked through and argued out.
Each draft went out to all 19 agencies, where it was reviewed not only by the CAA Director and outreach supervisors, but also by the workers who would be using it. They felt, correctly, that this high level or participation would generate a sense of ownership for the product –and produced the leaders in each CAA who would make it happen in that agency. All the CAAs understand the definitions they developed, and all CAAs are using them. Training of staff and supervisors took more time than they thought it would. They now have a cadre of trainers, one or more from several CAAs, who circuit ride around the state to do the one-day orientation training.
The many staff involved in this process were not only making up an outcome reporting system of course -- they were also making up new ways to think about their programs. This makes implementation much easier, because the front-line troops understand the new way of thinking about the programs and they perceive value in using the system.
Missouri CAA Assn, Elaine West, 573.634.2969
State CSBG, Charles McCann, 573.751.6789
JIM’S WISH LIST: I wish every CAA Association would do this!
CASE STUDY # 5: CALIFORNIA
The California state Office funded the Marin CAA to develop a “readiness” instrument to help CAAs figure out where they are in adopting ROMA and to help them develop a matrix for the goals. The NASCSP implementation study reports:
“Initially, agencies were given broad descriptions of what each level meant for each matrix and each agency defined each cell. During the pilot effort, it became clear that for purposes of usefulness and aggregability, more guidance and greater inter-agency consistency was necessary. As a result, narrative descriptions of each cell of each matrix are being prepared. Agencies will be given the opportunity and flexibility to further define the indicators especially relevant to local conditions that would be used to measure their progress toward these standardized outcomes.” (NASCSP)
The individual agency readiness assessment tool developed in California is on the ROMA web page as well. Unfortunately, California collapsed the six goals into three, thus eliminating important differences between them.
Community Action Marin, Bill Hamilton, 415.457.2522
State CSBG, Mike Micciche, 916.323.8694
JIM’S WISH LIST FOR CALIFORNIA: I wish you would go back and fight it out and develop standardized definitions for what goes inside the boxes on the matrix. And, I wish you would use all six goals!
CASE STUDY # 6: RENSSELEARVILLE INSTITUTE.
The Rensselaerville Institute Approach. They have a planning and outcomes reporting system, which they have used successfully with CAAs, Health Maintenance Organizations, education programs, and prevention and treatment programs. They are working with several state CSBG offices and state CAA Associations on implementation of ROMA. A quick summary of the TRI approach is given here. TRI proposes creation of performance targets, and measuring progress towards those targets by using milestones. It is more of a “home grown” way to do outcome measurement that does not involve scales and ladders. However, the differences between this approach to outcomes and scales and ladders is that they use different structures -- the content is not necessarily different. Many of the milestones developed in the TRI approach are outputs, not outcomes, and are therefore roughly equivalent in usage to the “interim outcomes” or “initial outcomes” as defined by some other people. Here are excerpts from the TRI materials.
Outcome Funding is a product of The Innovation Group of The Rensselaerville Institute. A key premise of The Group is that many organizations fail to recognize the distinction between having a bright idea and doing something with it. What is rare is the person who can take an idea -- regard less of the source -- and apply it beginning next Monday morning.
Prototyping is a way to move the idea to action. This consultancy helps investors and program implementors to understand and use tools designed to reward results rather than just effort. The Outcome Funding mind set is built on several principles:
1. Investors and implementors function most effectively as partners rather than adversaries;
2. Project success is strongly correlated with a focus on results rather than activities;
3. Efficiency is enhanced when a single set of tools is used for both securing funds and managing implementation;
4. The presence of the right people capable of carrying out a project is more important than a perfectly written plan.
The general sequence of steps carried out over a 4-7 month period.
1. Investor Briefing. This introductory session is designed to introduce managers and leaders of the investor organization to the principles and components of the Outcome Funding framework. Managers are asked to assess the approach against current issues and priorities. Those wishing to be considered for the prototype are encouraged to participate in Step 2.
2. Assessing Context/Designing the Prototype. A small design team reflecting central investor interests is established to assess the organizational context for a prototype project. A Brief Investor Outcome Statement is defined specifying the overall outcomes to be accomplished and any "givens" that must be taken into account.
3. Orienting the Investment Community. Direct implementors are offered an in-depth orientation on the concepts underlying Outcome Funding and are encouraged to consider becoming part of the prototype experience.
4. Identifying Prototypes. A small manageable set of prototype volunteers are recruited using criteria of readiness and capacity.
5. Preparing the Performance Targets/Specifying Milestones. A series of workshops and individualized learning sessions are conducted to help providers develop performance targets, products and milestones. Investors sign off on these targets and implementation begins to test the usefulness of these tools in guiding implementation.
6. The Prototype Period. A test period (most often 2 - 4 months) now occurs. Consultation is available throughout, along with a mid-course check-in to determine issues and progress.
7. Capturing Learnings and Assessing Results. Implementors and investors are brought together to determine the usefulness of the prototype experience and to make design recommendations for further development, testing, and implementation.
Here are the questions they answer during the 7-step process.
THE TARGET QUESTIONS:
1. Who are your CUSTOMERS and how many do you plan to serve?
2. What are your PERFORMANCE TARGETS for those customers and how will you know if you reach them?
3. What are the core features of your PRODUCT?
4. Who are the PEOPLE delivering the product?
5. How much will it COST to achieve the performance targets?
6. What is the logic of MILESTONES for customers on their way to the target?”
(From TRI training materials distributed in Detroit, 1999)
Contact: Elliott Pagliaccio 518.797.3783, or 518.377.1567.
Of particular interest to this writer:
they have excellent trainers,
funders are re-invented as investors,
participants are re-invented as customers,
their customer outcome funnel that shows progress from activity to activity,
annual report format in outcome terms,
they work on a performance contract – results guaranteed!
If you are looking for a systematic approach to developing measures within one program or within one “stovepipe,” TRI is one place to look.
JIM’S WISH LIST FOR TRI: Examples and exercises now focus only on goals 1 and 6 (need some for goals 2,3,4,5). This makes CSBG look too much like a service delivery system that does nothing else. Since OCS expects reports on all six goals, the other goals should also be addressed.
Step 11. Get the State Association and the State Office into a process. Work on one goal at a time, like Missouri is doing. Set up process ground rules like “You can’t drop out. Everybody must work until we get a consensus.”
J. Bring your funders along.
Like the local agencies, the state agencies and other funders are at very different places in regard to attitudes about, knowledge of, and use of outcome measurement. See if you can persuade them to work with each other on a design process, so much the better; e.g. the State offices in Ohio and New York that fund most CAA programs are working together on an approach to ROMA.
The Rensselaerville Institute advises that you:
* meet your funders (investors) halfway,
* help the investors figure out what their role is,
* reach agreements on customers, conditions and desired behaviors,
* agree on what the community would look like if there was no program (a baseline),
* send the investors drafts of everything you do,
* recognize that your initial measurement efforts are a learning process,
* report your results in way that helps the investors “tell the story” to other key players,
* recommend the investors drop some “old” type of reporting, or change the frequency,
* learn together, and
* expect it to take two or three years.
Conclusions. Funders are at vastly different places on desire and capability to develop outcome measures. Just like the rest of the world.
Step 12. YOU take the lead. Contact your funder(s) and bring them into your development process. Be a player, not a chip. Here are a few talking points for discussion with state agency funders.
A. Contract language and ROMA. What parts of ROMA are mandatory and what parts are optional? This should be negotiated.
B. Exactly what does each funder think they are “buying” from you? What results do they want to see happen because of your efforts?
C. ROMA is NOT JUST “units of service” like the SSBG. Some funders may be focused only on services to specific individuals, which would be under Goals 1 and/or 6. However, ROMA goes far beyond services to individual and includes results sought at the level of social system or communities, and changes in capacity of individual agencies and networks of agencies. Some education and negotiation may be needed.
D. The time needed for implementation is measured in years, not months.
E. Funders also have a role in helping to develop the new system, to test it and refine it, and to install it.
F. A separate, temporary reporting system may be needed to reporting progress in implementation. How often? How detailed?
G. What’s in it for them -- the state office or other funder. Find a role for everybody!
K. Possible roles for state CAA associations.
Over the past two years, I’ve become convinced that the State CAA association should be the leadership vehicle in managing the process of helping CAAs collectively develop the results measures that all CAAs will use. There are many possible roles for the State CAA Association. It should help initiate and manage the process of creating results measures and definitions on a statewide basis.
What do your member agencies need? Can you buy it wholesale for them? Continuous learning is a process, not an event. It will be three or five years before there is “certainty” about these new measures.
B. Develop and manage design teams.
The Missouri Association of Community Action Agencies helped to organize and manage the design process, provided the facilitation and research needed for ALL the CAAs to develop a set of scales that ALL local agencies were comfortable with. The state associations in California, Pennsylvania and New York were key vehicles and leaders as well. You can do this, too!
C. Liaison to state and Federal funding agencies.
You could be a conduit – finding out what ALL the relevant state agencies are doing in developing results and outcome measures, encouraging them to work together. Keep your members up to speed – and advocate for their interests.
D. Contract with state for implementation.
There is plenty of work to be done. Your association could be the contractor for some of it.
Conclusions. This must be done in partnership with the State CSBG office, and other partners if you can entice them in. Good examples of this kind of effort exist in Missouri, New York, and Virginia.
Step #13. Energize your association to provide leadership to organize the implementation effort.
L. Approaches to developing of the CONTENT of outcome measures.
The foundation of ROMA is the six goals and suggested outcome measures. Look at the management systems that exist in a CAA (from the NACAA Executive Directors Manual). They are:
Community participation organization and development
Program implementation, and reporting
Staff and board development, T&TA
Or, consider the 8 management systems the Head Start program. They are:
Record Keeping and Reporting
On going Monitoring
Because of the way ROMA is coming at you, the selection of specific measures to be used for reporting at the local level or within a state is often presented as the starting point in most CAAs for discussion. At first glance, that appears to be the challenge. This probably makes the implementation process more difficult, because the discussion assumes the rest of existing program operations are a given and that only one thing – reporting measures – must be changed. Wrong on both counts.
Whether you are looking at an agency or a program, changes in any one part of the system are going to precipitate changes in other parts of the system. In CSBG, if we change the goals and the measures, then everything in between is going to be affected as well. The outcome measures selected become both crucial descriptor of but also drivers of program strategy, because in order to produce the desired result you have to use a specific strategy that affects that measure. Put another way, Julie Jakopic from NASCSP says that “Because we end up doing what we measure, we need to choose carefully how we measure what we do.”
So we must consciously broaden the discussion about implementation of the new reporting system to also look at the conditions in the community, to include our theories about why the society works the way it works, and to include a review of the strategies we use to change it. I know that this expansion of awareness takes time and many people resist doing it, because helped I peel this onion over about a three-year period with USDA rural development programs. Any expansion of discussion to this broad range of topics immediately provokes resistance, especially from the defenders of the status quo. To them, an effort at large scale change implies that what we have been doing is somehow wrong, that we are guilty of not doing what we were supposed to be doing, and so on. One element of change management is to ELIMINATE RESIDUAL GUILT about changing something. This is done by the leaders who say things like:
“What we did was not wrong, it just was what we were doing. We did the best we knew how to do at the time, now we are looking for something else to try. Don’t feel bad about the past – feel good that we learned from it. Now, it is time to move on.”
There are several ways to go about this. You may use more than one, but it is useful to unravel them and look at the different assumptions on which they are based. And, no matter which way you start, you are probably going to wind up moving to a review of most of your systems and programs.
A. Inductive approach from existing program operations. Start with what you’ve got, and see how far you can take it.
In working with the U.S.D.A. Rural Development mission areas, John Johnston and I assisted several working groups (housing, business, utilities) in development of results measures. We found that we could construct usable outcome measures (1) by starting with the programs existing output measure(s), and (2) working outward very slowly and carefully toward the family and community, and (3) tracing every step, (4) to make sure there was a powerful link between every step. This was an inductive strategy -- to build what the evaluators call a logic model -- that starts with the program and moves outward by inches. And it inevitable moves back to question basic program strategy and activity.
B. Deductive approach from a plan.
This is the approach used by The Rensselearville Institute. Construct a plan, identify your goals, link your programs to the goals, and then figure out how to measure what you do by filling the gap or establishing the connections between the goal and the strategy.
C. Use a deductive approach that starts with social indicators and tries to bring it down to connect with a program.
In recent years, approaches such as the Oregon Compact have used social indicators (the Oregon Benchmarks) as a way to focus attention on a subject area. A large-scale citizen planning effort led to the selection of the 259 measures, which were then boiled down into a set of about 20 high-priority measures. They are used to focus attention on a topic, e.g. “Let’s all do what we can to reduce teen pregnancy.” But they are not used as a way of measuring performance of individual programs – they are used as signals that this is an important topic and people should be working on the issue. This also seems to be the approach being used by some of the Empowerment Zones and Enterprise Communities (EZ/EC initiative). EZ/EZ appears to have confounded both social indicators and benchmarks. And, it is hard to figure out how changes in the small populations covered by the zones are going to show up in social indicators.
Conclusion. My advice is DO NOT adopt social indicators (unemployment rate, crime rate) as outcome measures for your efforts. Remember, in the 1970's both the United Way and H.H.S. made a serious effort to use social indicators to measure progress in the social programs they funded – and they abandoned it. It did not work because most social indicators are the consequence of dozens or hundreds of factors that are working together to produce that result. The unemployment rate, for example, is impacted by interest rates in Thailand, high-school graduation rates in Japan, migration patterns in Mexico, technology patents in Germany, and on and on. The MATF is suggesting you use social indicators as descriptors of the CONTEXT of your CAA’s work, i.e., the unemployment rate in the community may influence your ability to place people in jobs, but you should not assume responsibility for tying to change the unemployment rate per-se.
D. Customer satisfaction measurement. You can make a pretty good argument our focus on program outputs is mis-focused because it gives too much credence to the past assumptions, and that in a service business the only thing that matters is what the customer thinks has happened.
Most service businesses – airlines, hotels, restaurants, banks, doctors, accountants – ask their customers what they think and give them ways to give them systematic feedback through mail-in or drop-in-the-box survey forms. Most Federal agencies now do the same thing, including the Social Security Administration and the DOL employment and training programs. Most cities and school districts also measure customer satisfaction.
Peter Drucker once said that the only purpose of a business was to find a customer and keep them. I guess in social service the goal would be to find a customer, then help them achieve their goal, and not keep them.
One way to tell if a CAA is approaching the measurement of results as a partnership between the agency and the program participants – and the “power balance” is equal – is if the CAA asks the program participants what they think in a systematic way. CAAs are behind in this area. Hopefully ROMA will prompt more CAAs to find out systematically through surveys what their participants think happened and how they perceive the benefits. (The author is highly opinionated on this subject and in the interest of full disclosure should report that he develops and conducts customer satisfaction surveys for Head Start programs.)
The ROMA Guide has information on how to construct and implement customer satisfaction surveys. I encourage you to use them!
E. Develop scales and ladders to connect what you do with the results produced. One of the new tools adapted for use in ROMA are scales and ladders. There are several examples on the ROMA web site. I would argue there was and is no problem with the concept of scales and ladders themselves. You can use the scale and ladder approach to measure just about anything. Put the worst case scenario at the bottom, the best case at the top. Create intermediate steps that show progress from the worst to the best. Eureka – a scale and ladder! I repeat, I think that the scales and ladders are good tools. The problems in using them come from the “loading” of their content. (More on this in the next section.)
F. Mix and Match. Most CAAs use a mixture of the approaches described above. It is important to be clear about which method you are using, otherwise you may find yourself where you think the discussion is about the content of a measure when it is actually about the logic system you are using to develop the measure.
M. If scales and ladders are a good way to measure change in an individual or a family, why are they controversial in some CAAs?
Although the differences of opinion appear to be about scales and ladders, the controversy is really about the family development strategy itself. There are three “issues” with the family development strategy and we should take them head on – and not confuse them with the methods used to measure what family development does, i.e. with scales and ladders. One issue is the desire some people have to limit the purpose of family development only to enhancement of self-sufficiency. A second issue is the amount of CAA activity or C.S.B.G. money that should be committed to the program strategy of family development (e.g. some or all). The third issue is how CAAs position their family development program in relationship to the many other agencies that use the same approach.
Before addressing these issues, let’s look at the evolution of scales and ladders as used by CAAs. Scales and ladders come from the convergence of several streams of activity.
One stream comes from social work methods, including case management, which have been in use since the 1930's.
Another came from new methods developed for use in child welfare. In the early-1980's, the H.H.S. Children’s Bureau child welfare program made grants to ten Universities (one in each region) to devise methods that would reduce the incidence of child abuse in families. The grant to the University of Iowa’s Department of Social Welfare focused on family assessment and family development. Rather than wait until the child was abused and send in a person with a M.S.W., they wanted to identify developing problems and intervene earlier, hopefully with people who had some training but did not require an M.S.W. This is a very legitimate approach. The Mid-Iowa Community Action Agency adapted these social work concepts to anti-poverty work, and developed the original version of scales and ladders to measure how a family was functioning and to measure family progress (the results of the CAAs work with those families) toward self-sufficiency.
In the early 1990's, the scales as developed by MICA were then refined and extended in use by the Bucks County CAA, when Eleanor Hunneman became CAA Director there. The timing on the work she was leading coincided with the increasing interest in measuring results, and the scales and ladders concept was brought to the M.A.T.F. by H.H.S/O.C.S., where it was seen as a valuable tool for use by CAAs. And now to the issues.
The first issue in using scales and ladders for the family development strategy in the C.S.B.G. universe was that some of the proponents of family development wanted to assume that the only purpose of family development was to create economic self-sufficiency. The big problem here is that unfortunately there is not a powerful connection between receipt of social services and household income level. The social work methods used in family development are not terribly powerful methods for helping people increase their income. If they were, then every program everywhere would be using them.
There are many reasons why a person’s income goes up or down. Worldwide, in all of the industrialized economies, capitalist (Hong Kong), communist (China), socialist (Finland, Sweden), and state-managed capitalism (Japan, Taiwan and the U.S.), about 25% of workers go up at least one quintile (20%) in income each year, and about15% of all workers go down at least one quintile in income each year. Economic mobility has mostly to do with changes in the structure of the economy, globalization, evolution of technology, and the effects of competition between businesses. The effects of these changes on an individual or family can only be marginally affected through case management methods. If, for example, you are a farmer, it may be distressing to learn that interest rates in Thailand and the amount of rainfall in Australia has more to do with your annual income that your hard work and experience, but that is where its at. (Pay attention to the World Trade Organization’s debates! I feel like the demonstrations in Seattle in 1999 are to the future of work what Washington, D.C. was in 1963 to the civil rights movement.)
Looking at the whole low-income population, about half of people who are poor today receive no publicly funded social services at all. And, about half of all people who are poor today will not be poor two years from now. There is of course some overlap between the two groups (services yes or no, income changes yes or no) but there is no proven correlation between receiving social services and increases in income. If there was, each of us would be able to cite the evaluations that proved the connection.
Many of us have an intuitive sense, or maybe it is just wishful thinking, that there is a connection, but the evaluation results do not make a compelling case for this. Of the 41 million people who have a diagnosable mental illness, of the 30 million alcoholics, and the 9 million substance abusers – of those who are working age -- about 85% of them get up and go to work every day. To argue that all these folks must address “their problem” first as a precondition to earning money is just incorrect. Our country has millions of drunks, drug addicts and really nasty people who earn quite a good living. If household income is the test, then they are self-sufficient. And to me, self sufficiency in an anti-poverty program is mostly about helping people earn money.
Sure, some of the families in a “self sufficiency” program get jobs and their incomes go up, but that is true of the families who are NOT in a program as well. We can not claim that our family development work either causes or “adds up” to an increase in household income. The program evaluations have compared people who are receiving these types of services and those who are not, and there is not much difference -- if any -- between the changes in income in the two groups. The only measurable changes show up after about five years, driven primarily by changes in educational attainment. At the point the woman gets an A.A. degree or a 4-year college degree, her income starts to go up.
There is an enormous amount of “churning” or cycling in and out of poverty, and we do not have very good methods for measuring this social mobility. To attribute the positive movement of a small number of people to a program approach (family development) that has marginal impact on changes in income is grasping at straws. We can certainly claim we are producing changes as measured on the scales, but we should not claim that these produce changes in household income. The research simple does not support that claim. To keep up with the litany of disappointing results on the relationship between services and changes in income as described in evaluations and other research reports, sign up for the free newsletter from Harvard University, The Evaluation Exchange: Emerging Strategies in Evaluating Child and Family Services.
There are many good reasons to do family development other than trying to make the family self-sufficient. All these relate to C.S.B.G. goal 6. These include: to reduce stress within the family, to enhance family functioning, to help an individual resolve personal problems, to improve child rearing practices, to improve the quality of life in a neighborhood, to get them into better housing, and so on. It is perfectly acceptable to use public funds to produce improvement in these areas. Family development makes a lot of sense in Head Start, in child welfare, in working with people who are homeless, and with new immigrants. This author organized training on case management for CAAs starting in 1990 when it became clear that social work methods were moving back into community action; NACAA cosponsored it. The Center for Community Futures sponsors three summer institutes (4 days each) on family development on the U.C. Berkeley campus using U.C. faculty and M.S.W. graduates. I like social work, family development and case management. But, to say the ONLY purpose of family development or even that the PRIMARY purpose is to produce self-sufficiency is too narrow a focus. It creates a huge gap, a discontinuity, between the rhetoric and the provable results, and in the long run this will result in you sawing off the limb you are sitting on.
In Missouri, they let each row or column stand on its own. This avoids the problem of exaggeration. For example, housing and nutrition can also be about health and safety as well as contributors to the ability to increase assets or to hold employment. So the problem is not in the use of the scale or ladder per se, it is in the assertion about what it MEANS because a person has moved up on a scale. To claim that moving from level two to level three on a nutrition scale causes a change in household income is just too long a leap for this author. That is ideology, not causality.
A second issue in some CAAs is that some of the more enthusiastic proponents of family development seemed to want to put all C.S.B.G. resources into that program strategy. In other words, they wanted the only strategy to be used by CAAs to be family development focused on self-sufficiency – as measured by scales and ladders. As the old paradigm continued to “hollow out” and lose power and credibility, scales and ladders appeared to some people to be not just the new kid on the block but the only kid on the block, so some people said “let’s just use it for everything.”
The first issue is a disagreement about how effective family development is as an anti-poverty strategy; i.e., how much of a change in household income could be loaded into the family development approach. The second is a disagreement about the amount of resources that should be committed to that strategy.
These disagreements were mingled into the discussion of use of scales and ladders, and there was a confounding between strategy (what we do to produce change) and measurement (how we describe the results of those changes). If the family development scale is primarily an observation tool for charting movement and its elements have little to do with the reasons for that movement, then we need to dig deeper and to find the causes of the movement. Put another way, we need to be sure the measures are valid -- that they actually measure what we say they do. And, taking it one step further, that those measures “add up” to what we say they add up to. And this is the problem of claiming that changes in the sub-scales add up to an increase in self-sufficiency. We slip off into a closed universe, into a tautology. (A tautology is a mental model that relies on its own internal logic for validation, but it has no external connection to reality.)
The third issue is that there are about twenty other service delivery networks that are using family development as a program strategy. Some of them are 100 years old, like the Settlement Houses. Some are 60 years old, like the Family Service Agencies. Some are brand new, like the Family Support Program and the Family Resource Centers. Some have tried it and failed, like AFDC from 1937 to 1967 -- remember that before 1967, all AFDC caseworkers had college degrees in social work! To position ourselves in their midst without having clear cut boundaries and without having a distinctive “brand name” will inevitably result in efforts to absorb us.
Conclusions. Using scales and ladders for measuring individual and family functioning in CAA family development programs is a good use of new tools. To assume the family development program strategy is a powerful way to change income and that all of CSBG should be devoted to it is a big mistake.
Step 14. Have all CAAs in your state work together to develop scales for use on a statewide basis. Put the program strategies that really do change income under Goal 1, to enhance self-sufficiency. Put the rest of family development that enhances family functioning under Goal 6. See Appendix B on page 62 for suggested links between strategies and goals.
Fortunately, as the community goals and agency goals are starting to be fleshed out, the CSBG network is re-balancing itself to put more emphasis on the other goals and strategies. And yes, scales and ladders can be used to measure agency capacity and community improvements.
In addition to the ROMA web site, another place to look at scales and ladders is at the California State University at Monterey Bay, the Institute for Community Collaborative Studies, at or, call 408.582.3624.
So if there really is some FUNDAMENTAL RETHINKING to be done in your agency, then what are some of the issues to be addressed in doing this? How do we sort out what we can really accomplish with our family development approach? Or any other approach? This usually requires updating the assumption about the economy. See next page.
N. Peeling the onion: Updating the assumptions on which your current programs are based.
Too many people are using invalid and antiquated assumptions about the social, economic and demographic characteristics of their region and community. New technology and the globalized economy have fundamentally changed the economic opportunity structure, so we need to change the methods we should use to help people get money. If you are still operating on the assumptions used even ten years ago, then ROMA will not be a very powerful tool because you are using it in an obsolete conceptual universe. Don’t use ROMA to improve the production of buggy whips or covered wagons. Use ROMA to re-position your agency and strategies to deal with the economy and social values as they exist today. Review your assumptions about the universe and make sure they are accurate.
It is during this phase that you hammer out your intentions and definitions of, for example, self-sufficiency. Most definitions of self-sufficiency are statements about the social values and behavior we want low-income people to adopt and are not based on current economic realities. The premise in too much of the discussion about “self sufficiency” is that “If they would just think like I do, it would lead to changes in their income.” This is an amazing assumption. It locates the nexus for action -- the arena of things to be changed -- between their thinking and my thinking, rather than between their capacities and what the marketplace will pay for. The huge bulk of changes in this arena will not produce changes in household income.
A person can be fully self-sufficient and be as poor as dirt. A person can be wealthy and not be self-sufficient. Most of us are dependent on our families, friends, community -- and tax breaks or government subsidies -- in one way or the other. What does YOUR agency mean by self-sufficiency? Does it mean a family should be receiving no welfare – but it is O.K. to lap up that mortgage-interest deduction? Or, they should not be receiving Food Stamps -- but it is O.K. to live on low-interest student loans? Medicare for Grandma is O.K. -- but Medicaid for her granddaughter is not good? You may have to unravel your definition into its several parts, and discuss each individual element of your definition separately.
The National Economic Development and Law Center uses the phrase “economic security” instead of self-sufficiency. I know what “economic security” looks like. I’ve been in community action for 33 years and I still can’t figure out what “self-sufficiency” looks like.
A good environmental scan should lead you to review and challenge your existing assumptions. The following chart shows why some of the old assumptions about how people get money are obsolete.
Why Some Historic Anti Poverty Strategies No Longer Work
One adult works
Enough to support a family.
Two adults must work
Have the D-a-d
Most dads were present
Too many dads missing.
Graduate High School or get a GED, and
The ticket to a non-poverty level job was a GED
Now a college degree is needed to assure a job above the poverty level
Get any kind of job
Low-skilled work often paid well. Only a handful of people who worked full time were poor
Minimum wage is now a poverty job. 11 million Americans work full time and are below poverty.
On the assembly line
Lots of low-skill jobs out there.
Anything that can be done by an unskilled person will be done by a robot or in a low-wage country
Break the race barrier through anti-discrimination, civil rights
If you could get into work, you got out of poverty
You can break the color barrier, and still not have a good job.
Social class was not important. Could still be lower class or working class
Could still escape poverty
Most non-poverty jobs now require middle-class communication and social skills and behavior.
Economic value of work was clear.
65% of people worked to produce cars, housing, clothes, food and other necessities, e.g. 65% of jobs produced tangible products, the value of which can be measured.
20% of jobs produce tangibles. The $ value of most jobs is not clear. We must also “pay” people for working thru the tax code.
Remove barriers to get out of poverty
Remove one or two barriers and they were out of poverty
Remove 10 barriers and either it still does not get them out of poverty, or there are still 20 more barriers. What happens to these folks?
What are the factors that determine how people get money now? Educational attainment explains about 1/3 of income differences. A high school diploma is required even for poverty-level employment -- and nowadays you’d better have a college degree to insure an increase in income over time. Race and gender explain another 1/3 of income differences – so anti-discrimination is still a powerful engine for increasing income. Another group can grind out a living through 60 -- 80 hours a week of self-employment activity because persistence in fact usually pays off.
The amount of money in publicly funded employment and training activity (JTPA and its successor) is only about 25% of what it was in the 1980's. The gist of it is that in this economy there is nothing that you can learn to do in 6 weeks or 12 weeks or 24 weeks that is going to guarantee you a job that is above the poverty level or a job that lasts more than a few months.
Hope may spring eternal, but we should stop deceiving ourselves and our customers about the opportunity structure that exists today. The American Dream is no longer working for the bottom group of wage earners or for people who do not have a college degree. We should start telling that story, and move to the changes in public policy and the tax code that can do something about the real problem.
We are in a different ball game. Globalization will continued to drive wages down and ship jobs offshore. Low-income people in America are the losers in this dynamic. Want more of this depressing news? Download “Jim’s Environmental Scan” for free from our web page, http://www.cencomfut.com.
Conclusion. The implementation of ROMA should precipitate an intense discussion among board, staff and others about (a) what is poverty today, (b) what are we trying to do about it, and (c) how can we actually change the opportunity structure in which we operate, and (d) how can we help people get more money. Good! It is time to “let go” of some of the historical approaches.
Step 15. Re-visit your assumptions about the economy and the rest of your environmental scan.
This seems like a good place to talk about letting go – about getting off of the dead horse of obsolete assumptions. I’ll just put in this wonderful piece that came to me through the Internet (thanks to whomever it was who sent it to me).
Ancient wisdom says that when you discover you are riding a dead horse, the best strategy is to dismount. However, in organizations we often try many other strategies, including the following:
1) Changing riders.
2) Buying a stronger whip.
3) Falling back on: "This is the way we have always ridden."
4) Appointing a committee to study the horse.
5) Arranging a visit to other sites to see how they ride dead horses.
6) Increasing the standards for riding dead horses.
7) Appointing a group to revive the dead horse.
8) Creating a training session to improve riding skills.
9) Compare the state of dead horses in today's environment.
10) Changing the requirements so the dead horse no longer meets the standard of death.
11) Hiring an external consultant to show how a dead horse can be ridden.
12) Harnessing several dead horses together to increase speed.
13) Increasing funding to improve the horse's performance.
14) Declaring that no horse is too dead to beat.
15) Doing a study to see if outsourcing will reduce the cost of riding a dead horse.
16) Buying a computer program to enhance dead horse performance.
17) Declaring a dead horse less costly than a live one.
18) Forming a workgroup to find uses for a dead horse.
19) Changing performance requirements for the horse.
20) Promoting the dead horse to a management position.
O. Peel another layer: review your purpose, vision, values and mission.
Review your agency’s purpose, vision, mission and values. Are they clear, or do they need sorting out? Confusion at this level will make ROMA vastly more difficult to implement.
Conclusion. You will think you are arguing about an outcome measure or a performance target when you are actually arguing about the purpose or vision of your agency. You may need to clear the decks for action.
Step 16. As part of your “plan for planning,” you should:
A. Clarify or redefine your agency relationship to society -- your agency purpose -- which describes the domain(s) in which you will operate; e.g. anti-poverty, human development, education, housing, or provider of basic (safety net) needs.
B. Review or refine the agreement between your Board members and partners and stakeholders about the agency vision -- about what BIG results it hopes to achieve?
C. Review or refine your mission -- the statement of how you will do what you want to do. The mission creates boundaries for selecting the types of strategies the organization will use. What strategies are you going to use or not use?
D. Sort out your values. Confusion between values and mission can mean an agency wants to be funded because of its commitments and good intentions, not because of the results it produces for participants or the community. Peter Drucker has an excellent discussion of the “moral versus economic” motives in his book, Managing the Nonprofit Organization (p. 111).
P. Traps to avoid in creating the content of measures.
A. Subsuming goals under each other.
There are 6 goals! Let them stand! Use them all!
B. Old wine in new bottles – re-labeling all existing system elements with the new terms.
This was probably the right thing to do in 1981 when we were trying to carry over existing activity into the new CSBG categories. This is probably the wrong thing to do now. It is time for reinvention.
C. Using only social indicators as measures.
This is a death trap. You will spend all your time trying to figure out a connection between the indicator and what you do.
D. Too many outcomes and performance measures, or not enough.
The average person can keep about 6 ideas going in their head at once. If you have 200 indicators most of us humans aren’t going to be able to track what you are talking about. If you only have one, you have too many eggs in one basket.
E. Data mania. “Does your child’s dog have all its teeth?”
Do we REALLY need to massive amounts of information about every single person we come into contact with? If there is no good reason to collect the information, don’t do it.
F. Doing the exact same thing with every single customer/ participant.
The two-hour intake process is not needed with most people. Only people going into intensive case management approach should have to divulge all this information. The main purpose of the interview is to establish the working relationship between the worker and the participant, not to collect “the facts.” The facts only reveal themselves over a period of months.
G. The Genuine Universal All Purpose All Encompassing Total Intake Form For All Our Programs and Maybe Other Agencies As Well.
This is an old favorite that comes around once about every 10 years. The St. Louis CAA tried this in 1967. We actually did this in New York City in 1970, and it covered AFDC, Food Stamps, employment and training, Head Start, and about 20 other programs. Unfortunately it was 17 pages long, and the front-line workers in any given program tended to collect only the info that was needed for their particular program. “We’ll get the rest of it later.” Hah hah. They tried this up in South Dakota about twenty years ago, too. SAARPA or something like that.
The problem this approach seeks to solve is the fact that Congress creates these programs with wildly different and conflicting assumptions, intentional variations in target groups and significant variations in benefits. These are legislated for political -- usually ideological -- reasons. If Congress placed a high priority on administrative efficiency they could legislate it or at least make it possible by removing the conflicting requirements they put in the legislation.
MY OBSERVATIONS: The perfect intake form takes thousands of person hours to invent. It might improve administrative efficiency by 1%. It does not change client outcomes or customer benefits at all.
Another perspective on this subject is that since computers can now be used to store and sort this basic information, and since people receive multiple services to achieve a goal (desired outcome) the standardization of information should (a) reduce application time for both staff and participants, increase resources made available to the family, make tracking possible and (d) reduce duplication of effort. A task force is working on this “multiple data base” issue. Contact NASCSP for information or if you want to get involved.
H. It’s all Computers (or, it’s all ANY one thing.)
Anytime you find this kind of “mind set” in an agency it is either because the management has dropped out of the process and let a subsystem take over, or there is a change in the internal power distribution in the agency in a way that some rising star has the bit between their teeth and is running with it.
Step 17. Add your position on these issues to your criteria for developing the content of measures.
Q. Transition from the old to the new
CROSSING OVER FROM THE OLD TO THE NEW
Some of the approaches to shifting to use of ROMA are given below.
A. Continue to run old system while developing the new. Begin use of new system on a date certain (e.g. at the start of a new program year).
B. Begin using “chunks” of the new system as they are developed, dropping the portions of the old system that these new chunks replace.
C. Create crossover points or transition bridges between old/new systems (Use the GPMS approach of creating several bridges that could be stepped across)
D. Use the TRI approach of prototyping and scale up. Start with one program.
E. Continue running the old system and develop and run the new system simultaneously. This does increase workload, but it assures accountability and that the new system is working before you drop the old.. And, extra work is all part of any change process!
F. Drop the old system and operate with no system at all while the new one is being developed (now this approach is probably the worst of the bunch!)
Conclusion. Don’t stop using any part of the old system until you consciously replace it with part of the new system.
Step 18. Develop a calendar that anticipates when various portions of the new system will be phased in and when training will take place beforehand.
R. Reporting system development.
Your program strategy and the locally determined outcomes approved by your staff, board and stakeholders -- that are derived from your purpose, vision, mission and values – should shape the categories in which you will be reporting. You may of course have to crosswalk between what you are really doing and the categories used on reports required by various funders – just like you do now!
Of course, every funder has their own version of reality and wants you to report that way.
Conclusion. After you have made it work on paper, then you can begin to make it work on a computer.
Step 19. Resist changing your computer system until you have designed your new program structure and measurement tools.
S. Linking your new outcomes measures to your financial system.
There is a bottom line, so to speak. The Government Accounting Standards Board is requiring additional changes in how you will have to link your outcomes to your financial system. Their standards will require auditors to link results measures with fiscal activity by the year 2002. Buy these beauties as gifts for your fiscal staff. G.A.S.B. has three publications describing SEA, or Service Effectiveness Accounting. See especially GASB’s Statement No 2 of Concepts Related to Service Efforts and Accomplishments Reporting.
By 2002, your auditor will have to certify any outcome measures that are included in their scope of audit as “acceptable.” The American Institute of Certified Public Accountants has adopted standards for certifying your measures.
They are given in the enclosed article by Frank Monti, Toward a Certified Statement of Outcomes, from Innovating Magazine. He describes the standards and the criteria that auditors will use when reviewing your statement of outcomes for certification. See Appendix C on page 65.
Give a copy of this appendix to your fiscal person.
Conclusions. Do not change your financial system until after you have developed your program structure.
Step 20. Use these standards and criteria to develop and review your local measures. And, you may as well use them when engaged in any state-level effort as well.
Two of the California CAAs that have done considerable work in trying to relate their new outcome measures to their financial systems are the CAA of San Mateo County, CAA Director Bill Parker at firstname.lastname@example.org and the Contra Costa County CAA, contact Steve Betz,
T. Computer software and ROMA related tools.
I do not think you can buy an off-the-shelf software program, plug it in and have it “solve your ROMA problem.” There is no substitute for YOU developing your own outcomes and results measures. These are what define your new program structure. Do this yourself, don’t let somebody else do it for you.
Every software vendor has major bucks invested in their off-the-shelf product and are hoping you can find a way to use it. Many vendors are developing templates for off-the-shelf software programs (Excel, etc) that you or they can modify to fit your specific needs. Most of these software programs cover only the direct service goals (1 and 6) and do not have much by way of content either for your community goals (2 and 3) or agency goals (4 and 5) -- so you have to invent desired results and outcomes measures for these goals anyhow.
Vendors with an installed base in CAAs (e.g. THO, GMS, CAP Systems and others) are all busily developing new modules for your consideration.
Some vendors even go so far as to offer you “free” software. Remember that the learning curve and staff training time, lost work time, and upkeep costs are all part of using any new software package.
Conclusion. There are no magic wands here.
Step 21. You will have to help your vendor adapt your software. You may be able to do this on a statewide basis.
U. Future development possibilities.
There are several opportunities and challenges on the horizon.
A. Return on Investment (R.O.I.)
We should continue to pursue models for measuring R.O.I. . As national expert Dennis Bensen says, “First, you have to have a program structure....” So – develop your new program structure, i.e., your new strategies, outcomes and measures -- then R.O.I. will come naturally. Although it is interesting to be aware of the methods for measuring R.O.I. [and for Internal Rate of Return (IRR) and Net Present Value (NPV) of dollars invested] and other business tools for measuring value of dollars spent as you design your measures and linkages to your financial system, the practical use will be several years in coming. It took the employment and training universe about ten years of hard work (with Dr. Bensen’s able assistance) to work out R.O.I. measures – and that was a far less complicated program structure than the CSBG. And, Congress just shuffled the deck on the E&T universe again – so R.O.I. is no insurance against random intervention.
If you want to get involved with the M.A.T.F. working group on R.O.I, contact Dennis Benson at Appropriate Solutions, 511 Garden Drive, Worthington, OH 43085. 614.840.0466, FAX 614.840.0467, e-mail: email@example.com
The American Society for Training and Development has a publication titled Level 5 Evaluation: R.O.I. The title tells the story – you have to do evaluation at levels one through four as the building blocks for evaluation at level five.
B. Community Service Network.
Some would like to see all the local agencies tied into the same management system or software system, with each service provider funded only for specific services. Would this best serve your community? If you would like to look into this, take a look at the systems and software produce by CSN, Inc, 1747 Pennsylvania Avenue, Washington, D.C. 20006. 202.775.6617. Joe Eaglin is one of the principals in this consulting group.
They have exhibited at NACAA conferences. Some city governments have adopted this software. It should be noted that neither OCS nor the NASCSP are requiring that this approach be used.
C. We have too much generic program activity under the CSBG, e.g. I&R, family development, case management, etc. With several other networks already doing the same or similar work, we run the risk of not having a distinctive approach. We need at least five new brand name strategies related to each goal that clearly belong to CAAs. Right now, we have two -- Head Start and the Dialogue on Poverty. We need a bunch more, e.g. like CAP Thrift Shops; Community Action Organizers; Community Action Family Development; Intergenerational Child Development; Willing Weatherizers; Lusty, Laughing, Lead Removers; YouthBuild (oops, already taken). You get the idea.
D. In the same way that devolution and program consolidation have swept through employment and training programs, the housing and community development programs and public assistance programs -- there could be a tidal wave that sweeps through other human services and development programs and that carry them into the rapids of consolidation and devolution. At the same time we revise and update the program strategies and management methods, the very existence of the programs must constantly be protected.
Wayne Thomas, one of our mentors and a former OEO/CSA official from Kansas City, says that “Sometimes people don’t get to where they want to go because they quit before they get there.”
Step 22. PERSIST!
Appendix A. The Big Picture.
This is a one-page comparison of family, community, and agency goals from the ROMA materials.
I think Eleanor Hunnemann wrote this 1997. Contact her at Positive Outcomes: e-mail is
If you are looking at this on-line, you can find this table as Figure 2-C in the ROMA Guide, page 2-5.
If you are looking at the printed copy, it is next.
Introductory Table-Contrasting Family, Community, & Agency Scales
Interventions, Strategies, and Activities produce change in:
1 & 6
1. Low-income people become more self-sufficient.
6. Low-income people, especially vulnerable populations, achieve their potential by strengthening family and other supportive systems
2 & 3
2. The conditions in which low-income people live are improved.
3. Low-income people own a stake in their community.
4 & 5
4. Partnerships among supporters and providers of services to low-income people are achieved.
5. Agencies increase their capacity to achieve results.
Unit of measure
One client or family at a time (consumers, individuals, enrollees, participants, households)
A community condition or system (wards, boroughs, counties, townships, parishes, SMSAs, territories, districts, neighborhoods)
An agency or program (agency, employee group, department, bargaining unit, project, program)
Scale Level & type of change
Family Level Scale measures change in levels of family functioning, attainment, and achievement
Community Level Scale measures change in levels of systems functioning, systems capacity, & community conditions
Agency Level Scale measures change in levels of capacity and performance
Dimensions of change
* family functioning
* basic needs
* community involvement
* substance abuse
* child care
* public policy
* civic capital
* service & support systems
* economic opportunity
* relationships & linkages compliance
* cultural sensitivity
* measurement and evaluation
* info management
* In Crisis
* In Crisis
* In Crisis
agency or program
agency or state
state or agency
[...] A working paper from the CSBG MATF Committee on Scales & Ladders to the CSBG Monitoring and Assessment Task Force, July 1, 1997.
Appendix B. Strategies that Might be Under Each Goal.
These are examples of strategies to illustrate where various TYPES of strategies might be associated with different goals. The TYPES are in caps.
Some strategies could be under more than one goal. Or, the strategy could start in relationship to one goal (community economic development under goal 2 to increase the number of jobs in the community) and work it way across into another goal (placement of people in jobs under goal 1 to help them become self-sufficient.)
It seems unlikely that the national groups will prescribe which types of strategies/programs or activities must be located under a specific goal. It is likely that either the national groups or your state funders will create reporting forms that have topics under one goal or the other. However, there is nothing to prevent you from planning and managing your programs using whatever framework you want. As long as you can identify your results/outcomes and cross-walk them to the reporting forms you should be O.K.
a. Get a j-o-b. Job clubs, work readiness, skill training, job placement.
Advanced education (basic ed is under goal 6)
b. Increase self-employment earnings. Microbusiness training, coaching.
c. Provide work through a venture owned by the agency.
d. Increase the value of work (minimum wage, living wage. This might also appear under goal #2.
e. Reward the social value of work as well as the economic value of work. EITC, other tax code approaches.
Goal 2. Conditions under which low-income people live are improved.
ECONOMIC OPPORTUNITY IS EXPANDED
Business expansion, retention.
INSTITUTIONS ARE MADE MORE FAIR, ACCESSIBLE
Community Reinvestment Act, Fair Housing, anti-discrimination, affirmative action, Americans with Disabilities Act.
EXISTING ORGANIZATIONS ARE IMPROVED
Persuaded to operate more effectively in relationship to low-income individual, or a community.
RULES OF THE GAME ARE CHANGED.
Penalties on immigrants are removed.
Regulations prohibiting street vending carts are eased.
Master plan update.
Affordable housing encouraged or rewarded
NEW INSTITUTIONS ARE CREATED.
Community dispute resolution service.
water/sewer, streets, signs, etc.
CLEAN UP FIX UP.
ALTERNATIVE FINANCIAL INSTITUTIONS ARE CREATED
CDFI, CDCU, Micro loan fund, local currency.
3. Own a stake in community.
COMMUNITY PARTICIPATION individually or collectively
civic engagement, volunteering, official role in organization or government, advisory committees, boards, elective offices. Voting, civic activities.
SOCIAL CAPITAL CREATED
Interconnections among groups. Trust levels go up, transaction costs decrease.
IDA’s (Could be under goal 1, self-sufficiency, as well.)
First time homeowner, buy public housing. Co-op formed. Community Land Trust.
According to the study conducted by NACAA (Fazzi and associates), the “typical” CAA has 62 formal agreements with other organizations, with the number ranging from about 10 to several hundred.
CREATE NEW PARTNERSHIPS
STRENGTHEN EXISTING PARTNERSHIPS. They have to be active partnerships -- just sitting the back of their room doesn’t count for much.
5. Agency Capacity.
Board and staff development fit here.
DEVELOP EXISTING STAFF/BOARD MEMBERS
6. Help families achieve their potential
Strengthen the family and other support systems, the human development, family functioning, and maintenance activities.
CHILD CARE (I put this here because of its child development component. Others might subsume this under their employment strategy.)
SUBSTANCE ABUSE PREVENTION AND TREATMENT
FOOD. Food stamp eligibility, commodity foods, food banks and pantries, kitchens.
SHELTER. Temporary, permanent.
CLOTHING. Thrift shops.
INCOME ASSISTANCE. SSI, TANF, etc.
Appendix C. Toward a Certified Statement of Outcomes
by Frank A. Monti
Financial reporting by not-for-profit organizations has always been different from financial reporting for profit-oriented business. While the balance sheet for each type is essentially the same, the statement of operations is very different. In the not-for-profit organization, the focus is not on the "bottom line," but rather on the sources and uses of funding.
The expenses of the not-for-profit are presented on a functional basis. The organization reports the total expenses for each of the different programs or initiatives it has undertaken. While this reporting, certified as being fairly stated by independent auditors, is essential to this sector of the economy, we believe that an additional Schedule of Program Service Accomplishments should also be published by these organizations. This Schedule would provide the reader (e.g., potential or current funding source) with statistical information regarding the outcomes achieved from the dollars expended in program efforts.
Reporting that ABC Agency spent $113,000 on counseling pregnant teenagers is not as useful to a reader as knowing that 400 teenagers participated in three counseling sessions. Even better reporting would be that the teenager's knowledge of prenatal care increased as a result of the counseling and that behavioral change followed new information. If this type of data were available, it could become part of the audited performance by the independent Certified Public Accountant. Further, it could be presented along with the traditional financial statements.
We are encouraging our clients to issue an annual report called Statistical and Financial Information. This report, bound in the auditor's report covers just as financial statements have been presented for years, would contain the Schedule of Program Service Accomplishments followed by the traditional financial statements. Preceding these reports would be the Independent Auditor's Report covering both the financial statements and the Schedule of Program Accomplishments. The auditors would provide their opinion on this Schedule in the following manner.
In our opinion, the Schedule of Program Service Accomplishments referred to above presents the program service accomplishments in conformity with (identification of established criteria or) the measurement and disclosure criteria set forth in Note 1.
Getting Your Auditor Interested.
Why would your auditor want to audit your Schedule of Program Service Accomplishments? The first and most obvious reason is that it represents additional professional services for which the auditor bills you. Secondly, this Schedule is a report to which their stamp of approval adds enormous credibility to your organization. An auditor wants to assist his or her client to become successful and grow in size. Finally, this audit function is so very different from standard certification of financial statements that many auditors might openly welcome the change of pace!
Reprinted with Permission from INNOVATING TM Vol.7, No. 3, Spring 1999. Copyright The Rensselaerville Institute, Rensselaerville, New York 12147.
Creating the Schedule of Program Service Accomplishments
Creating a Schedule of Program Service Accomplishments for audit by your independent CPA (Certified Public Accountant) is not a task to be left to the end of the year. A series of complex reporting standards apply to reporting program service accomplishments just as they do in financial reporting. It is these standards, promulgated by the American Institute of Certified Public Accountants (AICPA) that give your certified financial statements credibility with potential funding sources. Likewise, adhering to standards of reporting for program service accomplishments will impart similar credibility to those statistics.
The AICPA has established standards for certifying the information presented in a Schedule of Program Service Accomplishments. (Footnote 1) The AICPA indicates that a CPA may only certify an outcome if the following two conditions exist:
1. The outcome is capable of evaluation against reasonable criteria that either have been established by a recognized body or are stated in the presentation of the outcome in a sufficiently clear and comprehensive manner for a knowledgeable reader to be able to understand it.
2. The outcome is capable of reasonably consistent estimation or measurement using such criteria.
In guiding the CPA in his or her determination of whether reasonable criteria have been established by recognized bodies, the AICPA indicates that criteria issued by regulatory agencies and other bodies composed of experts that follow such procedures as including protocols for broad distribution of proposed criteria for public comment should be considered reasonable criteria for this purpose. Criteria established by industry associations or similar groups that do not follow due process or do not clearly represent the public interest, however, should be viewed more critically.
When reasonable outcome criteria from sources external to the reporting organization do not exist, the organization itself may have to develop its own outcome criteria. In such a situation, the CPA must determine if the established criteria are reasonable. Reasonable criteria are those that yield useful information. The usefulness of information depends on an appropriate balance between relevance and reliability.
Relevance and Reliability If your organization is formulating its own outcome criteria, the concepts of relevance and reliability must be understood and incorporated into those criteria.
* The capacity to make a difference in a decision. The outcomes are useful in confirming or correcting prior expectations.
* The ability to bear upon uncertainty. The outcome is useful in confirming or altering the degree of uncertainty about the result of the action or program.
Relevant outcomes are either the ultimate outcome or the performance targets that are recognized or hypothesized as leading to the achievement of the ultimate outcome. For example, when reduction of teen pregnancy is the ultimate outcome, and your organization is working with teens throughout their risk years, then the outcome of not becoming pregnant is relevant. However, if your organization's program distributes literature to 12 to 14-year-olds, then a relevant outcome is the assertion that they read the literature and retained some target percentage of the knowledge contained therein. In this latter situation, the organization's outcome is not to reduce teen pregnancy, but to provide knowledge which is believed to be useful in attaining the ultimate outcome of reduced teen pregnancy.
Currently, organizations involved in such a program are likely to be focused on the process of the literature distribution. To become outcome-oriented, they may have to devise a feedback mechanism to determine if the literature was read and whether it increased the knowledge of the readers. A longer-term feedback mechanism is needed to confirm the hypothesis that the knowledge gained results in incidence of pregnancy.
If this longer-term feedback is replaced by conventional wisdom or external studies, e.g., National Institutes for Health studies concluding that ignorance of the risks of certain behavior contributes to teenage pregnancy, then the organization may not have to create its own feedback mechanism. The important point is that the outcome is relevant because it is useful in confirming prior expectations.
To be relevant, the outcome must also be timely, complete and consistent over time.
* Timeliness. The outcome is available to decision-makers before they lose their capability to influence decisions.
* Completeness, The outcome does not omit information that could alter or confirm a decision
* Consistency. The outcomes are measured and presented in substantially the same manner in succeeding reports. If changes are necessary, they are disclosed, justified, and, where practical, reconciled to permit proper interpretations of sequential measurements.
* Representational faithfulness The outcomes correspond or agree with the phenomena they purport to represent.
* Absence of unwarranted inference or certainly or precision. The outcomes may sometimes be presented more appropriately though the use of ranges or indications of the probabilities attaching to different values rather than as single point estimates.
* Neutrality. The primary concern is the relevance and reliability of the outcomes rather than their potential effect on a particular interest.
* Freedom from bias. The measurements involved in the outcomes are equally likely to fall on either side of what they represent rather than more often on one side than the other.
If your organization is faced with the task of creating its own relevant and reliable outcomes for measurement purposes, do not underestimate the difficulty of the task. It could take many hours of discussion among knowledgeable people. The amount of lack of agreement and misunderstanding that exists at the beginning of this process could be frightening or depressing to an organization that thought it clearly understood its mission.
When we first started this initiative, many of our clients experienced difficulty in expressing their organization's mission in a relevant and reliable outcome-oriented manner. Part of the problem was that they saw themselves as contributing to a larger goal of society. Although this larger goal was usually expressed as an outcome-reduced hunger, increased literacy, etc.-their particular role in achieving the outcome was still expressed as a process that was part of the larger process designed to achieve the ultimate outcome. The odds are that your organization's program is involved in only one of the performance targets along the way to achieving the ultimate outcome desired by society. Your staff's focus on the ultimate outcome temporarily blinds them to the performance outcomes of your program.
We have a client that works to improve health conditions in the third world. The staff thought in terms of reducing infant mortality and intestinal viruses. The program built water systems to deliver clean, safe drinking water to the village. They could never document that their program reduced infant mortality or intestinal viruses because there were so many other factors that contributed to these conditions. When they realized that their program was actually to change the village peoples' water habits, then they also realized that their outcome-oriented goal was the creation of a water system sufficiently convenient to the village coupled with education of villagers, with the outcome that the villagers used clean, safe water 90% of the time.
Measuring Outcomes to be Certified
After you have finally developed relevant and reliable outcomes to measure, the next step is to design a measurement system. Just as your accounting system is designed with specific internal controls to assure accurate measurement of the resources received and used, the outcome measurement system also needs to have internal controls. The absence of these controls may render your system and your Schedule of Program Service Accomplishments incapable of audit and certification by your CPA. Therefore, you need to start this task at least a full year before you hope to publicly present your program accomplishments as independently certified.
For example, if your program outcome is to distribute literature to a target population and increase their knowledge as a result, you will need to design a way to measure with reasonable assurance the degree to which the literature is reaching the target population. In addition, you need some mechanism that shows statistically that your literature is increasing their knowledge. National advertisers obtain such feedback on the effectiveness of their advertising program as measured against a specific outcome. Not-for-profit organizations that want their auditors to certify the achievement of program outcomes need to consider similar methods of measure.
Statistics that are maintained internally need to be verifiable. The process of data collection must be free of weaknesses that may contribute to their mis-statement -- either by accident or intent. Staff may have to be trained in the accumulation of statistical information to ensure its proper reporting. Many believe that reporting outcomes can only be achieved with the addition of expensive computerized databases of information. However, just as manual accounting systems kept track of hundreds of thousands of dollars prior to the advent of the personal computer, reliable statistical information on program outcomes can also be accumulated manually. Be wary of those who hide behind the $100,000 computer system deemed to be essential to building to an outcome-oriented program.
Even without a computerized database, moving to an outcome-oriented program is going to result in added costs. However, if you can document the value-added to your program of obtaining a certification of the program's achievement of its outcome objectives, your search for funding will be much easier. Funders will soon be turning their backs on agencies seeking continued funding for process. The first agencies to pick from the outcome funding tree will enjoy the best fruit.
1. Attestation Standards by American Institute for Certified Public Accountants.
Frank A. Monti is a CPA in Providence, RI. He can be contacted at
387 Charles Street, Providence, RI, 02904-22249.
* * *
Accountability for Performance Measurement and Monitoring in Local Government. David Ammons, Editor. International City Management Association. 1995. 777 N Capitol St N.E., Washington, D.C., 20002. 800.745.8780.
Assessment of Performance Measures for Public Health, Substance Abuse and Mental Health. National Research Council. 1997. Edward Perrin and Jeffrey Koshel, Editors. National Academy Press. 2101 Constitution Avenue, N.W. Washington, D.C. 20418. This is THE book on performance measures for these subject areas.
Concepts Related to Service Efforts and Accomplishments Reporting. Concepts Statement No 2 of the Governmental Accounting Standards Board. April, 1994. PO Box 5116, Norwalk, CT 06856. 203.847.0700 ext. 10.
Developing Client Outcome Monitoring Systems. The Urban Institute. 2100 M Street, NW. Washington, D. C. 20037. PH 202..857.8729.
The Evaluation Exchange: Emerging Strategies in Evaluating Child and Family Services. Harvard Family Research Project, Harvard Graduate School of Education, 38 Concord Avenue, Cambridge, MA 02138. PH 617.495.9108, FAX 617.495.8594. Web page is And, by the way, on my computer that little ~ (tilde) is on the key that is in the upper left hand corner of the keyboard.
Field Guide to Outcome-Based Program Evaluation. $34.95 The Evaluation Forum. 1932 First Ave, Ste 403. Seattle, WA 98101. 206.269.0171 FAX 206.448.0977. e-mail is: EvalForum@aol.com
Goals and Success Indicators of Healthy and Self-Sufficient Communities, and Bay Area Partnership: Building Health and Self-Sufficient Communities for Economic Prosperity, Status Report. 50 California Street, Ste 200. San Francisco, CA 94111. 415.772.4430. FAX 415.391.9929. E-mail is:
Grant Programs: Design Features, Shape, Flexibility, Accountability and Performance Information. GAO/GGD-98-137. Single copies are free. Order by mail. General Accounting Office, PO Box 370950, Washington, D.C. 20013 Or, call 202.512.6000 or FAX 202.512.6061.
Head Start Program Performance Measures, Second Program Report. HAS/ACT/ACYF, Research Demonstration and Evaluation Branch, Log No. ACYF-IM-HS-98-19. 11/24/98.
How Effective Are Your Community Services? Procedures for Measuring Their Quality. Harry Hatry, et. al. The Urban Institute and ICMA. 1992. The Urban Institute is at 2100 M Street, N.W., Washington, D.C. 20037. 202.857.8729.
IMPLEMENTING RESULTS-ORIENTED MANAGEMENT AND ACCOUNTABILITY FOR THE COMMUNITY SERVICES BLOCK GRANT (CSBG): A Preliminary Report On State Efforts. By Julie Jakopic. 1997. National Association for State Community Services Programs. 444 North Capitol Street, NW. Suite 395. Washington, DC 20001
Innovating (TM). By the Innovation Group. The Rensselaerville Institute. 63 Huyck Road, Rensselaerville, NY 12147. 518.797.3783. A quarterly publication. Annual subscription $30.
Level 5 Evaluation: R.O.I. May, 1998, Issue # 9805. $10. American Society for Training and Development. 1640 King Street. Alexandria, VA 22313. PH 703.683.8100. FAX 703.683.8103.
Managing the Nonprofit Organization. Drucker, Peter. HarperColliins Publishers, New York, NY. 1990.
Measuring Program Outcomes: A Practical Approach, 1996. United Way of America. 701 North Fairfax St, Alexandria, VA 22314. 703.836.7100. Item Number 0989. To order, call Sales Service 800.772.0008
Working Toward Community Goals: Helping Communities Succeed. North Central Regional Center for Rural Development, Iowa State University, 108 Curtis Hall, Ames, IA 50011. Ph 515.294.8321, FAX 515.294.3480. $15.
Their web page is:
Managing the Transition to Outcome Based Planning Evaluation. The Evaluation Forum. 1932 First Ave, Ste 403. Seattle, WA 98101. 206.269.0171 FAX 206.448.0997. $14.95
National Partnership for Reinventing Government, (formerly National Performance Review) 750-17th Street, NW, Suite 200. Washington, DC 20006. Customer Service Desk (202) 694-0001 Voice Mail - (202) 632-0150. FAX - (202) 632-0390 Web page is at:http://www.npr.gov
Reinventing Government, David Osborne and Ted Gaebler, 1992. Addisom Wesley Publishing, Reading, MA 01867.
Toward a Certified Statement of Outcomes. Frank Monti. Innovating, Spring, 1999. Tpp 15 -- 24. The Rensselaerville Institute. 63 Huyck Road, Rensselaerville, NY 12147. 518.797.3783. A quarterly publication. Annual subscription $30.
* * *