The need to design resilient energy systems becomes ever more apparent as we face the challenge of decarbonising through reliance on non-dispatchable technologies and sectoral integration. Increasingly, modelling efforts focus on improving system resilience, but fail to quantify the improvements. In this paper, we propose a novel workflow that allows increases in resilience to be measured quantitatively. It incorporates out-of-sample testing following optimisation, and compares the impacts of demand and power interruption uncertainty on both risk-unaware and risk-aware district energy system models. To ensure we encompass the full range of impacts caused by uncertainty, we consider nine distinct objectives encompassing differences in: investment and operation costs, CO2 emissions, and aversion to risk. We apply the workflow in a case study in Bangalore, India, and demonstrate that scenario optimisation improves system resilience by one to two orders of magnitude. However, systems designed for resilience to demand uncertainty are not able to gracefully extend to managing risk from extreme shocks to the system, such as power interruptions. We show that shock-induced instability can be addressed by specific measures to reduce grid dependence. Finally, by studying out-of-sample test results, we identify an objective which balances cost, CO2 emissions, and system resilience; this balance is achieved by novel application of the Conditional Value at Risk measure. These results expose the need for out-of-sample testing whenever uncertainty is considered in energy system modelling, and we provide the framework with which it can be undertaken.