A senior engineer for a large national property owner recently asked me about the accuracy of energy modeling while mentioning his team’s uneasiness with trusting energy model results. As a long time modeler, I find this a disheartening, but not uncommon question and sentiment. But there are steps a client can take to significantly increase the odds of obtaining accurate and detailed energy models that will be very useful to a project.

To answer the question of modeling accuracy, I can provide an analogy to consider. The accuracy of an energy model is like considering the tastiness of a cake. Whether a cake tastes good is entirely dependent upon the experience of the chef and the quality of the ingredients. A chocolate cake made by a famed pastry chef is going to be far from a cake my daughter makes for Father’s Day.  The same can be said for making – or creating – energy models. The accuracy of an energy model is entirely dependent upon the experience of the modeler and the quality of the information being incorporated into the model.

However, there is a unique struggle the energy modeling industry faces related to model accuracy. There is no clear, non-technical system for evaluating the accuracy and validity of work during the project. Currently, the only way a nontechnical person can evaluate an energy model is to wait a couple of months to evaluate whether building energy usage matches with the model predictions.  This method is just too slow for the fast pace of the building industry. One of the few universal methods to evaluate energy models is performed as a part of the LEED process.  Every energy model developed for LEED goes through review.  However, the accuracy of LEED models has been notoriously called into question (2008 New Buildings Study), giving skeptics fuel for their arguments against energy modeling.

Building energy consumption is very complicated, and the best energy modeling programs have to be complicated themselves to be accurate, meaning that specialized skills are needed to be able to understand and evaluate the model. This has led to a “black box” type of view of energy modeling. Unfortunately, this also has resulted in the industry being populated with professionals that “don’t know what they don’t know”. The majority of energy modeling professionals operate in a vacuum.  Most energy models are not competently reviewed or compared to actual resulting energy consumption. As a result, modelers rarely get actual feedback on whether or not they did a good job.

So how does a client obtain an accurate energy model when it can be difficult to properly evaluate the work? The simple answer is to focus on the same thing that makes a great cake: the chef and the ingredients. Following are some steps that can improve the accuracy energy model.

  • Hire a firm with extensive experience modeling existing buildings.
  • Require onsite measurements and data logging of critical assumptions (applicable when modeling existing buildings).
  • Require a model assumptions report.
  • Bring in a third party reviewer.

 

Hire a firm with extensive experience modeling existing buildings

This should be your first priority. An architectural or engineering design firm that is considered to be competent, may not a capable energy modeling firm.  Energy modeling is not identical to running design load calculations, even though design loads can often be calculated in the same program that is used for energy modeling. Many firms will sell energy models because they have licenses for programs that can build energy models, not because they have experience to do the work. In this same vein, having personnel on staff that have completed a training course or have obtained modeling certifications is not enough to ensure accurate results.

To obtain optimal results, you should locate a modeling team that has significant (100+ models) experience modeling existing buildings. The “existing buildings” expertise is key, and will disqualify 80% of the firms offering energy modeling as a service. One of the primary issues with modelers today is that many of them only understand how buildings are designed to operate. They don’t understand how buildings actually end up operating. Many modelers have only built energy models as part of the building design process (such as with LEED), and have never revisited their models to determine how their assumptions actually performed. Issues including people flows and actual occupant densities are often lost on designers. Also, buildings rarely operate HVAC systems exactly as the designer would assume. Human behavior in building operation is the most critical aspect of creating an accurate energy model, and is one with which most designers are not familiar.

The reason I recommend finding modelers that have considerable experience with existing buildings is the additional step of baseline calibration that typically factors in to modeling existing buildings. When creating a model of an existing building, it is necessary to compare and calibrate the model results to the actual historical energy consumption, often referred to as the baseline. This step brings a measure of validity to the model. It also happens to be, by far, the hardest process in energy modeling. It forces the modeler to consider the human aspect of the occupants, the real performance of equipment, and the controls sequences and settings being used to run the building, and how all of these things change with time.

Time and again, I have seen engineers that have only created new building energy models become completely baffled because their model results for an existing building are completely different from historical utility data. Based on my experience, a modeler needs to have completed at least 20 – 30 existing building models – under the instruction of a seasoned modeler – prior to being able to accurately complete this last step on their own. Even when modeling new buildings, the more experience a modeler has with the calibration process of existing building models, the better his or her results will be.

Require onsite measurements and data logging of critical assumptions (applicable when modeling existing buildings).

It’s a fact that few investment grade projects today will accept using results from an energy model if some sort of on-site data collection is not completed. There are several types of data collection that can be accomplished, whether through building control system trending or with standalone data loggers. Often, help from building operators is required to provide good data, access, and information to the modeler. It is critical to have building staff involved in the process, as they are the one most familiar with its operation. For example, if a model is being used to predict energy savings from adding VFDs to AHU fans, it is important to trend existing fan operation and take amp readings of fan power. This way correct power and run-times can be input into the baseline.

This step defines baseline variables that are important to the savings measures planned to evaluate. It also prevents overly aggressive assumptions used to inflate the savings potential of a specific energy savings measure. There are always more variables that can be trended or logged, but capturing the critical pieces is a significant step. It is not uncommon for an average engineering firm to trend 10-50 control points, amp larger HVAC motors, and place data loggers in a building before modeling it. A quality energy model will not only calibrate to the historical utility consumption, but also have measured and confirmed inputs for major energy consuming variables.

Require a model assumptions report.

While this may seem like a formality, in my experience, a detailed assumptions report offers many useful benefits. The first, and most obvious, is that this report can allow non-technical staff, owners, and technical personnel that may not be familiar with energy modeling, to validate critical model assumptions. While it can be difficult to evaluate an energy model directly in the modeling program, it can be relatively easy to evaluate operational assumptions and inputs.  I have been a part of a number of projects where incorrect assumptions made by modelers were ferreted out during review of the model reports by building operations staff.

The secondary effect of requiring this report is to put an onus on the modeler to justify assumptions to themselves. Because there is so much information that is input into an energy model, one can often unintentionally be inconsistent with assumptions or overlook verification of a particular assumption. When assembling a report, the modeler is forced to review and summarize their assumptions, and will often discover and correct issues like this prior to submission, hence these reports improving the quality of the model.

Finally, a model assumptions report can provide valuable documentation for the project. Energy models are expensive to build and delays in project scope changes may require the model to be modified or revisited; this can range anywhere from 3 months up to 2 years later. By requiring a model assumptions report, the framework for the model is documented and can make revisiting and revising the model a much smoother process.

Bring in a third party reviewer – and listen to their advice

A number of firms offer owner representation services for energy projects, and their assistance reviewing energy models can be very important when the model is utilized to make investment decisions.  Evaluating energy models is difficult even for seasoned engineers not familiar with the process. If you do not have someone on staff that is familiar with energy modeling, it can be a good idea to hire a third party that is familiar with the software and process to represent you and review the energy modeler’s work. The larger and complex the project, the more useful the service of a technical reviewer will be.

This step takes more importance when you have a modeling team that has incentive to inflate results, or when you question their experience. While most energy modelers are ethical, there are some that will inflate the results to sell you their services and/or equipment. It is becoming more commonplace to have equipment reps offer to calculate savings from installing their products. Many times I have seen these models use overly-aggressive assumptions, and have very little due diligence incorporated in the process.

Even when you trust the engineer or product representative to be ethical, having a third party reviewer can be good practice to keep all parties honest. In fact, many government energy projects require this step in the process. Having a qualified reviewer can save headaches and help to avoid making a decision based on a model created by someone who doesn’t know what they are doing.

John R. Nott, PE, CEM, Principal
John is responsible for the technical development of energy projects for Griffith Engineering, Inc., including creating detailed, year-round energy performance models. John is also responsible for developing and editing the firm’s library of Excel tools for simple energy calculations. Additionally, John leads the energy modeling team focused on EA credits for LEED certification.