Future Directions: Study Protocol for an Effectiveness- Implementation Hybrid Evaluation of a State-based Social Housing Strategy and Three Social Housing Programs

Background: In the Australian state of New South Wales nearly 60,000 approved applicants are waiting for social housing. Future Directions for Social Housing is a response to this challenge. This collection of housing programs aims to provide more social housing, support and incentives for leaving social housing and a better social housing experience. This document presents the protocol of the evaluation of these programs and the overarching Future Directions Strategy. Methods/Design: The evaluation will use a Type 1 effectiveness-implementation hybrid design, with an integrated, dual focus on assessing the effectiveness of Future Directions and better understanding the context for reform implementation. Program effectiveness will be examined using quasi-experimental techniques applied to linked administrative data. The implementation context will be examined via program level data, qualitative interviews and focus groups with stakeholders and tenants. Some quantitative survey and administrative data will also be used. Findings from the implementation evaluation will be used to inform and interpret the effectiveness evaluation. Economic evaluations will also be conducted. Discussion: This methodology will produce a high-quality evaluation of a large, complex government program which aims to facilitate rapid translational gains, real-time adoption of effective implementation strategies and generate actionable insights for policymakers.


Introduction of Future Directions
To respond to these challenges, in 2016 the NSW Government announced a new 10-year vision for social housing -Future Directions for Social Housing in NSW (Future Directions). This aims to transform the sector by providing: (1) more social housing; (2) more opportunities, support and incentives to avoid and/or leave social housing; and (3) a better social housing experience. Specifically, Future Directions aims to transform the NSW social housing system through the introduction of three new programs and six Service Improvement Initiatives (SIIs).
This protocol documents the proposed design and methods to evaluate the overarching Future Directions Strategy and the three new programs. The strategy evaluation will also utilise information provided by independent evaluations of the six SIIs. 2 The three programs are: • Social and Affordable Housing Fund (SAHF): This program uses the proceeds of a dedicated financing fund to provide additional social housing dwellings through outcomesfocused contracts with registered Community Housing Providers (CHPs). Contracted CHPs also provide access to services through tailored support coordination to assist tenants to build their capabilities and take advantage of opportunities to become more independent.
• Future Directions Projects implemented by the NSW Land and Housing Corporation 3 (LAHC FDI Projects): These projects deliver substantial expansion and redevelopment of housing stock aiming to deliver 23,000 new/redeveloped properties where social housing is blended with private and/or affordable housing via partnerships with private sector developers and finance. The social housing is owned by government with management by government or CHPs. There are four streams: i) Major Projects: Large scale projects on government-owned land that provide a mix of social (30%), affordable and private dwellings (70%) close to transport, education, jobs and services. These sites will be linked with education, training and local employment services; ii) New Communities: Existing social housing estates reconfigured into mixed-ownership with high quality community facilities and open space (both government-and CHP-managed); iii) Neighbourhood Projects: Small to medium-sized projects (20-250 dwellings) delivering social housing mixed with affordable and private housing; and iv) New Supply: small-scale renewal of dated social housing dwellings resulting in increased housing density.
• Social Housing Management Transfer (SHMT): SHMT transfers significant tenancy management responsibility for around 14,000 properties across entire specific areas to nine CHPs. CHPs are granted 20-year leases on the properties, compared to the usual 3-year leases. Tenants in CHP-managed properties become eligible to receive CRA from the federal government. This increases the revenue flowing to the CHPs as they receive social housing rental payments plus CRAincreasing their ability to provide additional services to tenants -while leaving tenant rental expenses unchanged. DCJ (formerly FACS) undertook extensive consultation and planning from 2016, developing the Future Directions Evaluation Framework to enable a high-quality robust evaluation in accordance with the NSW Government Program Evaluation Guidelines (2016). 4 The Framework ensures data collection is embedded in program delivery where possible, allowing outcomes to be measured over time, providing timely and relevant information to support decision making, service delivery and service improvement. This longitudinal approach is critical given the staged implementation of the programs and initiatives. It allows for the clear attribution of change in outcomes to inform more effective, outcomes-based performance monitoring. Cross-government consultation and governance has supported linkage of government administrative data to measure client outcomes across the seven domains of the NSW Government Human Services Outcomes Framework. Several components are now being implemented (LAHC FDI Projects and SAHF) or are completed (SHMT). The evaluation will examine implementation and outcomes up to late 2021.

Overview and key approaches
The Future Directions evaluation aims to answer two main questions: 1. Has the Future Directions Strategy achieved its objectives? 2. Which Future Directions Programs and Initiatives work well, for whom, and under which circumstances (including location-specific differences within a Program or Initiative)? The evaluation will use a Type 1 effectiveness-implementation hybrid design (Curran et al., 2012), with an integrated, dual focus on assessing the effectiveness of Future Directions and better understanding the context for reform implementation. Data from the implementation evaluation will be used to interpret the effectiveness evaluation by providing insight into contextual factors that may have activated or hindered mechanisms of change. The evaluation plans and all reports will be reviewed by independent expert advisors to ensure quality standards are met.

General approach to measuring effectiveness
For each program, we use the most rigorous approach that is feasible given what we know about the program, and the administrative and other data that are available. Quasi-experimental methods are used where possible. For each program a comparison group will be constructed whose outcomes will be compared with the outcomes of households participating in Future Directions (the treatment group). As the programs may affect different subpopulations differently, where possible, the effects for specific population subgroups --Aboriginal and Torres Strait Islander or Culturally and Linguistically Diverse tenants, or tenants targeted by particular programs e.g., older women, single parent familieswill be estimated.

General approach to assessing implementation context
Using primarily qualitative approaches tailored to program context, we will examine the development of the Future Directions policy, and barriers and enablers to policy and program implementation with service providers and government. We will also explore tenants' perspectives of the quality of Future Directions delivery including services, tenancies, and dwellings, and how those experiences may relate to any broader outcomes they have realised.

Data sources
We will mostly use administrative data from government departments to measure effectiveness outcomes. Some complementary information on specific locations will be collected at an aggregated level from a number of sources (e.g., NSW Police for local crime data, Australian Bureau of Statistics for local unemployment rates).
Data used to examine implementation context will mostly be collected from stakeholders (i.e., service providers and government) and tenants through qualitative interviews and focus groups. Some administrative data will also be used and some program-level data and quantitative survey data will be collected to report on the achievement of specific Future Directions targets, and assess the impact of barriers and enablers on implementation over time.

Data collection methods
The administrative data will be extracted from the respective systems in government departments, and linked by a Data Linking Authority, after receiving data custodian approval and ethics approval for this linkage.
Qualitative data will be collected from policymakers, program managers or service providers with significant involvement in the Future Directions reform, using validated implementation science interview guides administered via telephone or online. Tenant data will come from conversations conducted via telephone or in-person, via one-on-one interviews, focus groups, family interviews, or paired in-depth interviews with people who reside in social housing or, where appropriate, in affordable or private market accommodation. We will recruit tenants to represent the diversity (cultural, linguistic, Aboriginal and Torres Strait Islander, and other key dimensions) prevalent in the targeted housing communities, and tenants will be interviewed by people who match that diversity.
Recruitment approaches and specific data collection methods will be tailored to each housing program based on the suggestions of stakeholders, and in the case of tenants, CIRCA's Housing Program Evaluation -Aboriginal Reference Group.

Data analysis
Analysis of the administrative data for the effectiveness component of the evaluation will involve statistical tests of differences in outcomes across treatment and control groups, matching of treated (Future Directions) tenants to similar non-treated tenants, and differences-indifferences and regression analysis.
Analysis of data for the implementation component will in general take the following forms: • thematic or framework analysis (a technique which uses a priori codes) of transcribed qualitative data to describe policy and service models, identify barriers and enablers to implementation, and tenant experiences and perspectives of service delivery quality; • descriptive statistics to determine the frequency with which specific implementation barriers or enablers are experienced over time; and • content analysis of key Future Directions policy and strategy documents.

Social and Affordable Housing Fund (SAHF) program evaluation
All SAHF residents are in scope for the evaluation. Each SAHF dwelling has a 25-year timeframe, the overall program timeframe extends from 2017 to 2048, 25 years after the delivery of the first dwellings in 2017 to the final dwellings expected by the end of 2023. The first tenants moved into SAHF housing in May 2017, and by July 2019, 494 residents were living in SAHF housing. This will increase to around 3,400 by the end of 2022. The evaluation will thus establish relatively early findings.

Effectiveness component
A quasi-experimental approach will exploit the randomness of social housing applicants' allocation to a SAHF or other social housing dwelling, conditional on the allocation zone chosen by the applicant. This method compares the post-treatment outcomes of a treatment (SAHF) group and control (non-SAHF) group who were similar prior to the treatment. The quasi-random estimation approach relies on the assumption that the allocation to a specific dwelling is close to random (only reflecting the position on the Housing register when the SAHF dwelling becomes available). Thus, the characteristics of those allocated to the SAHF dwelling and those allocated Social Science Protocols, July 2021, 1-12. http://dx.doi.org/10.7565/ssp.v4.5712 6 to other social housing, or still on the waiting list, are similar. We will test this assumption and if it is incorrect, we will complement this quasi-random approach with a differences-in-differences strategy in which we additionally control for the differences between the treated and control groups at baseline.
We will conduct three comparisons: SAHF tenants to: 1) non-SAHF public housing tenants; 2) non-SAHF community housing tenants; and 3) households on the Housing Register. The first comparison identifies the benefit of SAHF over regular public housing. The second comparison allows us to isolate the impact of having a tailored support services coordinator who can link tenants to relevant local services for which they are eligiblean innovative element of the SAHF program not available in other community housing. The third comparison assesses the benefits of providing additional social housing (as opposed to delivering it in a certain way). This control group will be comprised of people who are as close to the top of the Register as possible. In each comparison, differences in characteristics of treatment and control groups will be examined with the control groups being further refined using propensity score matching if the two groups are not sufficiently similar.
We will examine outcomes for those in social housing and those who have exited social housing, including satisfaction with housing; subjective assessments of individuals' well-being; probability of exiting social housing and exit destinations; outcomes of social housing tenants who exit (e.g., homelessness within certain period after exits, re-entry into social housing). We will also examine utilisation of housing stock (e.g., quality of the match between dwelling and tenant, vacancy rate and duration of vacancies, exit rates); quality of housing stock (e.g., distance to public transport, shops, health care, education, employment and other amenities); dwelling quality (e.g., ventilation, heating, cooling) and regional outcomes (e.g., homelessness service usage rates, crime rates, housing prices and rents, employment and unemployment rates).

Implementation component
A mixed-methods approach will be used to assess the quality of the implementation of SAHF against the intended standards, to describe advantages and disadvantages of various SAHF service models, and to identify and track perceived and experienced barriers by CHPs and tenants. We will assess the relative nuances of each model using an online questionnaire hosted by Qualtrics focusing on, for example, number of staff, key roles and responsibilities, model performance and functioning, and adaptations made over time. We will measure barriers and enablers to SAHF implementation in two ways: 1. Semi-structured qualitative interviews, informed by the Consolidated Framework for Implementation Research (CFIR; Damschroder et al., 2009), with 21 stakeholders across the SAHF models to elicit information on the specific barriers and enablers to SAHF implementation (delivering new supply, and delivering tailored support coordination) across five implementation domains, and 2. An online questionnaire, based on the CFIR and insights gleaned through qualitative interviews, designed to quantitatively monitor what barriers and enablers are being experienced by each CHP over time and at different stages of implementation. In order to contextualise the quantitative findings on tenant outcomes in the effectiveness component, qualitative data will be collected from 60 tenants across three case study sites (20 per site) about their experiences with SAHF implementation and CHP services, and their outcomes. Tenants will be selected to represent the diversity (cultural, linguistic, Aboriginal and Torres Strait Islander, and other key status backgrounds) prevalent in the targeted housing communities. The focus of the qualitative component will be to explore how tenants define their satisfaction Social Science Protocols, July 2021, 1-12. http://dx.doi.org/10.7565/ssp.v4.5712 7 with services and dwellings, which aspects of their experience with the services and dwellings are most important to them and why, their perceptions of control and safety in their tenancies and the process with the CHPs, and how they feel SAHF may have affected any of their educational, health, economic, or social and community outcomes. The qualitative approach will also explore the general qualities of the process of engaging with the CHPs, from tenants' perspectiveswhat has worked, what has not, and whether patterns in the types of tenants who have different experiences are discernible.

Land and Housing Corporation Future Directions Implementation Projects (LAHC FDI Projects) program evaluation
In total, as at 30 June 2020, just over 1,700 LAHC FDI Projects dwellings had been completed since June 2016. Most Major Projects were still in the land use rezoning and development approval stages with local councils and can therefore not be evaluated yet. Three New Communities projects have delivered sufficient dwellings to be in scope for this evaluation. One (small) Neighbourhood project and New Supply dwellings amounting to approximately 1,500 social housing dwellings will also be included in the evaluation. The outcome evaluation will include all dwellings, while in-depth qualitative data will be collected in relation to at least 60 of these dwellings for the implementation component. Due to the timing of the evaluation only relatively short-term outcomes will be evaluated.

Effectiveness component
The evaluation will use a quasi-experimental design, similar to that described for SAHF, exploiting the randomness of allocation of social housing applicants to the new housing delivered by LAHC FDI Projects, conditional on the allocation zone chosen by applicants and their characteristics. The specific outcomes to be considered are the same as those listed above for SAHF.

Implementation component
We will focus on exploring the implementation process of the delivery of mixed community dwellings from the perspective of councils, developers and designers (e.g., architects), and explore tenant perspectives about the implementation process, as well as the outcomes they may have experienced.
We will use qualitative semi-structured interviews, informed by CFIR, with 21 stakeholders to explore the achievement of objectives and barriers and enablers to implementation from a range of perspectives. We will supplement this approach with a framework or checklist of known critical factors in mixed community delivery (identified from reviews of the literature) to explore which critical success factors were present, which were missing and why, and what impact this had on stakeholders' perceptions of implementation effectiveness.
In order to contextualise the quantitative findings on tenant outcomes in the effectiveness component, qualitative data about their experiences with LAHC FDI implementation, as well as their outcomes, will be collected from 60 social, affordable, and market rate tenants residing in New Communities and New Supply dwellings across a range of LAHC FDI Projects development sites. Tenants will be selected to represent the diversity prevalent in the targeted housing communities. The focus of the qualitative component will be to explore the extent to which and how LAHC FDI Projects provide housing and associated services that meet the needs of tenants. This component of the evaluation also seeks to explore if and how LAHC FDI Projects create Social Science Protocols, July 2021, 1-12. http://dx.doi.org/10.7565/ssp.v4.5712 8 opportunities for tenants to realise improved social, economic, educational, health, and wellbeing outcomes.

Social Housing Management Transfer (SHMT) program evaluation
The evaluation will assess SHMT at all nine locations and across all SHMT CHPs. It will seek to separately identify the total impact of the program and the impacts of its sub-components: 1) the transfer of the property and tenancy management and 2) the impacts of the anticipated increased wrap-around service provision. All management transfers were completed by the end of 2019, which allows an evaluation of the impact for all transfers up to at least two years after the introduction of SHMT.

Effectiveness component
Due to the whole-area-transfer of public housing approach taken in the SHMT program, a (quasi) random method is not feasible. Instead, we will apply individual-level matching methods to construct a control group of non-SHMT tenants with similar pre-transfer individual, household and dwelling characteristics to SHMT tenants (treatment group). A comparison of changes in outcomes for the SHMT tenants with similar public housing tenants in non-SHMT zones will identify the total impact of SHMT. Where feasible we will compare changes in the experiences of the treatment group with changes in the experiences of the control group (a difference-indifference methodology). This will not be possible when comparing the SHMT experience to other CHP-managed housing experiences; then a straight difference in outcomes will be compared.
Matched difference-in-difference analysis relies on the assumption that both the treatment and comparison group were on the same trajectory prior to program implementation. This assumption will be checked using data from the pre-program period.
Outcome variables are similar to those for SAHF, with the addition of the following SHMTspecific variables: composition of new tenants and rent paying structure (e.g., proportion paying market rent, proportion paying 25 per cent of income); and the following housing register outcome variables for the relevant allocation zones: number of applicants, composition of people on the housing register, incidence of skipping people on the housing register and reasons for skipping, and transfers of applicants to other zones.

Implementation component
We will assess the quality of SHMT implementation using qualitative semi-structured interviews with up to 72 CHP stakeholders involved in the delivery of SHMT services. The interview guide will be based on CFIR, to explore implementation barriers and enablers, and adapted versions of the Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM) and Feasibility of Intervention Measure (FIM; see Weiner et al. (2017)), which are validated, questionnaire tools designed to assess implementation outcomes. These implementation outcomesacceptability, appropriateness and feasibility -are often described as 'leading indicators' of implementation success (Bowen et al., 2009;Proctor et al., 2011;Weiner et al., 2017) and are used here as a measure of implementation quality. We will also access quantitative administrative data to determine what proportion of NSW social housing is being managed by CHPs versus the NSW Government.
In order to contextualise the quantitative findings on tenant outcomes in the effectiveness component, qualitative data will be collected from 60 tenants across three case study sites (20 per site) about their experiences with the SHMT implementation process and the tenancy services Social Science Protocols, July 2021, 1-12. http://dx.doi.org/10.7565/ssp.v4.5712 9 offered by the CHPs, as well as their outcomes. Tenants included in the sample will be a mix of people who experienced the transfer and tenants who moved into the property after the transfer. In addition, tenants will be selected to represent the diversity prevalent in the targeted housing communities. The focus will be to explore how tenants define their satisfaction with services and dwellings, which aspects of their experience with the services and dwellings are most important to them and why, their perceptions of control and safety in their tenancies and in the process of transfer to the CHPs, and how they feel the transfer of management may have affected any of their educational, health, economic, or social and community outcomes. The qualitative approach with tenants will also explore the general qualities of the process of engaging with DCJ and the SHMT CHPs during the transfer and afterward from tenants' perspectiveswhat worked, what did not, and whether patterns in the types of tenants who had different experiences are discernible.

Overall strategy evaluation
The strategy evaluation looks across the three Future Direction programs to examine the collective impact of the strategic reform for tenants. Implementation in this context focuses on policy and strategy development, and the inclusion of tenants in strategy design.

Effectiveness component
The Strategy evaluation will synthesise the effectiveness components of the program evaluations and the independent evaluations of the SIIs, comparing outcomes for similar populations/target groups to determine whether some programs work better for specific groups of clients, and whether some programs work better overall. Additional analysis that will be conducted includes comparing the control group of no Future Directions, with multiple treatment groups: • participation in one of the programs without participation in SII • participation in SII but none of the programs • participation in one of the programs with participation in SII.
We repeat the approaches above for sub-populations where relevant and feasible. We will also use quasi-experimental techniques to examine the impact of the (temporary) relocation process on tenant and community outcomes. (a) We will compare outcomes for residents of redeveloped sites who are relocated (treatment group) with social housing tenants residing in the same allocation zone as the relocated residents but who have not been relocated (control group). (b) To identify community impact (if relevant), we will compare average outcomes for individuals residing on estates where relocated residents have been temporarily housed (treatment communities) with average outcomes of individuals residing in the same allocation zone but on estates where no residents were relocated.

Implementation component
A mixed methods approach will be used to identify policy impact determinants and explore and describe the factors that influenced development and implementation of the Future Directions strategy. The approach will include collection and analysis of qualitative and quantitative data derived from 12 key policymaking and service staff closely involved with Future Directions strategy and implementation. We will use qualitative semi-structured interviews to: • identify and explore the nature of policy determinants in the development and implementation of the Future Directions strategy, developed using by the Analysis of Determinants of Policy Impact (ADEPT) determinants interview guide (Rütten, Röger, Abu-Omar and Frahsa, 2009), and • assess the use of evidence in the development of the Future Directions strategy using a validated tool based on the Supporting Policy In health with Research: an Intervention Trial (SPIRIT) Action Framework, called the Staff Assessment of enGagement with Evidence (SAGE) interview tool (Makkar et al., 2015).
We will also explore the use of tenant perspectives in the Future Directions Strategy development, in order to understand if and how the needs, interests, and perspectives of housing tenants who will be affected by the Future Directions Strategy were solicited and used in its development. This component of the evaluation will supplement the research on policy impact determinants and evidence use, to provide a holistic view of the factors affecting the development of the strategy. Drawing on documentation provided by DCJ, LAHC and any other relevant NSW Government departments, we will conduct a content analysis to identify the ways prospective and current tenants impacted by the Strategy are officially reported to have been engaged and what effects their input had. Following the content analysis, we will validate the findings with key informants within relevant NSW Government departments (e.g., DCJ and LAHC). This will provide important insight into the ways in which the Future Directions Strategy reflects tenants' perspectives, highlighting where beneficiary-centred policymaking processes were strong, and where there are opportunities for improvement and capacity building in the future. This insight will also inform the exploration of factors that relate to tenant perceptions of Future Directions program implementation and outcomes.
A qualitative approach using content analysis and follow-up interviews and informationseeking will be used to explore and describe the ways in which policymakers engaged with and used input from direct beneficiaries in the development of the Future Directions strategy, and to identify how this engagement and use relates to tenants' experiences with the implementation of each Future Directions Program.

Economic evaluation component
To determine whether the economic benefits of each program outweigh the costs, we will compare the overall Future Directions reform case to the base case of no reform and compare each program component to the relevant base case scenarios. We will calculate the net present value and the cost-benefit ratio from the viewpoint of the NSW Government (the implementer) and from the whole of society. We will conduct the analysis for the short-medium term (1-2 years) and longer term (projected) cost-benefit analyses.
Benefits will be monetised where possible, but the evaluation will also discuss the scope for benefits that are difficult to monetise, e.g., the psychological security of a permanent home and empowerment. Actual values will be used where possible. For projections of benefits into the future we will use the DCJ Benefits Database (FACSIAR, 2019) where possible. To calculate the overall costs of the reform we will use actual costs subject to data availability and be guided by the FACS CBA Guide and Unit Costing Manual. We will also use costing information from CHPs where available.
All (annualised) costs and benefit estimates will be converted to present values by accounting for the economic life of assets (where applicable) and applying a discount rate. Costs and benefits of the reform will be calculated for population sub-groups and sensitivity analysis will be undertaken to test for a range of assumptions. . Written consent will be obtained from all participants involved in the qualitative components prior to inclusion in this study. Permission to use administrative data has been received from relevant government departments.

Discussion
Future Directions is an ambitious program aiming to deliver more and better-quality social housing, while providing pathways to the private housing market. We aim to provide a highquality evaluation of a large, complex government program. By using a Type 1 effectivenessimplementation hybrid design approach, the evaluation aims to facilitate rapid translational gains and real-time adoption of more effective implementation strategies, as well as generating more actionable insights for policymakers.
The use of linked administrative data ensures social housing tenants are not burdened with numerous surveys, while ensuring a rigorous evaluation of this important policy can take place. Once the data extraction is in place, it can be repeated relatively easily, allowing for continued assessment of a broad range of outcomes to inform policymakers.
The focus on the tenant experience via qualitative and culturally grounded approaches means that tenants can talk about their experiences in a culturally appropriate and safe forum, which enhances our ability to get honest insights into policy impact.
Limitations of the evaluation include that the reliance on administrative data restricts the analysis to information routinely collected by government. A further challenge is turnover in NSW Government staff involved in program development and implementation.

Declarations
Funding: This evaluation is funded by the NSW Department of Communities and Justice.
Contribution Statement: Cameron, Etuk, Hateley-Browne, Kalb, Parker and Rose contributed to the conception and writing of this protocol. All authors contributed to the evaluation methodology and approved the final draft.
Competing Interests: The authors declare that they have no competing interests.

Financial Interests:
The authors declare that they have no financial interests in the interventions evaluated within the context of this study, nor will they financially benefit from the outcomes of this research.