Utah HPwES Online Self-Assessment vs. In-home Analysis

Detailed Accuracy Analysis

An important question to ask is how an online self-assessment compares in terms of consumption accuracy and realistic recommendations. While an online assessment cannot (and should not) replace the "sitting at the kitchen table" experience that an in-home analysis can provide for a homeowner, it is an important comparison to ensure that the pre-screening experience is disqualifying/qualifying accurately and that it is setting realistic expectations for savings. EnergySavvy's Online Audit was just as accurate at predicting consumption and savings potential as an in-home analysis.

Methodology

EnergySavvy and PECI conducted a comparative analysis of a randomly selected subset of 100 homes from the Utah HPwES program for which we had online self-assessment data (entered by the applicants themselves), in-home analysis results (collected by BPI Certified analysts using a RESNET Certified in-home analysis tool) and actual consumption data (based on 12 months of utility bills provided by the homeowner at the time of analysis).

We compared the overall consumption values (in Mbtu's) predicted by the in-home analysis software and EnergySavvy's online analysis tool to actual consumption values for the 100 home sample. For each software tool (online and in-home), a percent error was calculated between predicted consumption (by the tool) and actual consumption (from billing history). We also compared the savings percentage estimates predicted by the in-home analysis software and EnergySavvy's online analysis tool for the same 100-home sample. Both the mean percent error values and mean savings percentage estimates for each tool were compared using a T-test.

What did we learn?

No building science tool (in-home or online) is perfect in modeling actual consumption in all cases. As an industry, we all understand that. There are too many unique circumstances and variations in behavior to always get it right. However, we were surprised by some of the findings.

Overall, the EnergySavvy online analysis tool - filled out within a few minutes by homeowners themselves - had 22.2% less error, on average, in modeling consumption than the in-home analysis software did (p-value: 0.0012 - values less than 0.05 indicate a statistically significant difference).

EnergySavvy's average savings percentage estimate was higher, but within range of the in-home analysis' estimate (42.6% versus 31.5%; p-value: < 0.0001). However, it is important to note that the savings estimates from the in-home analysis software represent only the subset of measures that the homeowner was interested in completing; whereas, the EnergySavvy online tool's savings estimate was based on all possible relevant measures for a home.

Why an Online Self-Assessment Doesn't Replace an In-Home Analysis

From the standpoint of modeled accuracy alone, one might come to the conclusion that in-home analyses are not necessary. At a cost of three to four hours and hundreds of dollars (often partially paid by the homeowner and partially subsidized by a program), if there is no additional accuracy to be gained over EnergySavvy's three to five minute online self-assessment, why bother?

Programs should still "bother" with in-home assessments, but not merely as an expensive means to estimate overall savings potential when a simpler online assessment can do that just as well. An in-home analysis is important, instead for:

• Work Scope Actionability: An online self-analysis cannot produce an actionable list of specific measures and places for a contractor to focus on during a retrofit. A trained analyst needs to inspect the home in person to see the actual conditions in each part of the house and determine a clear and realistic work scope for the upgrade.

• Homeowner Engagement: An online self-assessment is an easy first step for homeowners to take at their own convenience, at home or even when they are away from home. (Statistically speaking, most web browsing occurs during business hours.) But since the ultimate goal of a retrofit program is to drive retrofits, a personable and trusted analyst serves a critical role as a sales person for the retrofit: explaining the costs and benefits and helping the homeowners make the right decisions given their situations.

• Savings Modeling by Measure: Detailed modeling inputs are necessary for programs that require a high level of pre- and post- condition data for measurement and verification purposes. And if a technically inclined homeowner is trying to decide, for example, between a vinyl 2-pane window and a triple-pane low e window with a composite frame, a detailed model-based comparison can help them make their decision.

Page 1 2 « Previous Page