sment Expert
Review Pilot

Employment Portfolio

Panel Summary Report

Table of Contents

 

Executive Summary

The National Institute on Disability and Rehabilitation
Research (NIDRR) conducted its Annual Portfolio Assessment Expert Review
(APAER) Process Pilot for the Employment Portfolio on September 29 and
30, 2005, in Washington, D.C.

The APAER process was developed
by NIDRR to assess the agencys progress in meeting Federal performance
requirements under the Government Performance and Results Act (GPRA)
and the Program Assessment Reporting Tool (PART). The PART is a systematic
method of assessing and improving program performance across the Federal
government, instituted by the Office of Management and Budget (OMB).
The APAER process was designed as an external expert review of NIDRR
grantee accomplishments using a three-year cycle, in which one-third
of the overall research agenda is reviewed each year. NIDRRs Employment
Portfolio was the first review completed under the 2005 pilot phase
of this new process. 

A panel of nine researchers,
consumers, clinicians, policy experts and administrators, reviewed NIDRRs
employment portfolio, based on reports from 16 grantees with active
awards in 2004, across three program mechanisms: Rehabilitation Research
and Training Centers (RRTCs), Disability and Rehabilitation Research
Projects (DRRPs), and Field Initiated Projects (FIPs). The 16 awards
that were reviewed were grouped into two topical clusters based on the
focus of the award: addressing systems level issues and addressing individual
level issues.

The review covered:

Objective
1:  The quality and relevance of NIDRR-funded research and
the extent to which outputs and outcomes are contributing to the agencys
long-term performance measures and strategic goals.
Objective
2: The strengths and weaknesses of the employment portfolio as
a whole, including recommendations to improve the portfolio. And,
Objective
3: The quality and relevance of the agencys management of research
directions and award decisions.

Since this was a pilot
process, data were also collected to assess the feasibility of the process
design and implementation.

Findings
and Recommendations

The panel provided frank evaluations and recommendations
to NIDRR in three areas: (1)  the Employment portfolio performance,
(2) NIDRR management, and (3) the APAER process. Overall, the panel
expressed concern about performance of the portfolio and the soundness
of the evaluation and review process. Given the pilot nature of the
2005 APAER process, care must be taken in interpreting and applying
the findings contained in this report on NIDRRs Employment portfolio. 

Portfolio Performance. Panelists identified a number of outputs
and outcomes that appeared to be important to improving employment services,
including projects that focused on employment for culturally diverse
populations and aging populations, and projects focused on telework
and other important accommodations.

Overall, the panel found
the research
ideas in the portfolio to be of very high quality. However, they
expressed concern about: (1) the apparent lack of scientific rigor behind
the identified outputs, (2) the lack of sufficient information on the
methodologies used by grantees; (3) the lack of evidence supporting
many of the claims made by grantees in their Supplemental Information
Reports, and (4) the lack of peer reviewed publications.

In the area of knowledge
translation, the panel was uncertain about the adequacy of consumer-oriented
outputs. Some panelists believed the number of outputs was good, but
could not rate the quality of these products based on the information
provided. Many reviewers felt that grant activities had the potential
to impact individuals with disabilities, but that these activities did
not seem connected to the research. 

The panel concluded, that
in terms of progress toward NIDRRs capacity-building performance measures,
the Employment portfolio was limited. Many awards were judged lacking
in the use of multidisciplinary teams and no controlled studies were
identified. The panel could not judge the adequacy of awards to diverse
individuals and institutions, nor the integration of research and training.

Management of the Portfolio. Because persons with disabilities
continue to be unemployed and underemployed in disproportionately high
numbers, the panel recommended that:

NIDRR should
consider whether this portfolio area is adequately funded.
NIDRR should
reconceptualize the role of the FIPs and how results from FIPs could
translate into other, larger, more comprehensive funding mechanisms
within NIDRR. They felt that even though the science was exploratory,
the push to disseminate often led to findings being presented as too
definitive.
NIDRR should
consider more cross-award and cross-agency work.
NIDRR should
strengthen its unique role in distributing information to consumers.
NIDRR should
provide more focus on employers and labor/demand side issues, as well
as on empowerment and school-to-work transition.
NIDRR should
consider using an advisory group to help determine priorities. 

The panel also suggested
that NIDRR examine the unintended bias to continue funding the same
centers vs. its role in funding new researchers.

Critique of the APAER Process. In general, the panel commended
NIDRR on its efforts to evaluate the Employment portfolio.  However,
panelists expressed many concerns about the process and their ability
to accurately evaluate the portfolio, including:  (1) the quality
of the grantee reports, suggesting that perhaps grantees did not understand
how to complete the online form, were confused by the terminology, or
simply did not put much effort into the process; (2) receipt of under-processed
data for review;  and (3) the general lack of clarity in the definition
of terms (multidisciplinary, publication, short-term outcome and intermediate
outcome). Panelists suggested that project officers might provide more
guidance to grantees in preparing reports.

 

Annual Portfolio Assessment Expert
Review Pilot

Employment Portfolio

Panel Summary Report

The National Institute
on Disability and Rehabilitation Research (NIDRR) conducted its Annual
Portfolio Assessment Expert Review (APAER) Process Pilot for the Employment
Portfolio on September 29 and 30, 2005, in Washington, D.C. This summary
report presents a brief description of the APAER process and the findings
and recommendations of the expert panel.

Section 1:
Overview of the APAER Process

The APAER process was developed by NIDRR to assess its progress in
meeting Federal performance requirements under the Government Performance
and Results Act (GPRA) and the Program Assessment Reporting Tool (PART)1, and to
capture progress of its grantees using a three-year cycle, with one-third
of NIDRRs portfolio reviewed every three years. A team of experts reviewed
NIDRRs Employment portfolio as a pilot of this new process. This first
year is a baseline year for the Employment portfolio. Through this pilot,
NIDRR intends to gather data to compare its baseline performance with
results at the end of a specified time period in 2013.

The challenges to NIDRR of APAER were numerous and included establishing
an integrated and methodologically sound portfolio assessment and independent
expert review process that would:

Be manageable with existing resources and consistent with emerging
standards of practice for performance assessment in Federal R&D
agencies,