Source: EPC Reports

EPC – Energy Performance Certificates are available dating back to 2008.  The data contains a lot of information related to each property.  EPC certificates are actually quite easy to find on the internet. That said, if you want to enhance data like that from the Land Registry, then EPC data can help.  In my case I am specifically interested in floor space and EPC holds that, the Land Registry does not.

Source

You can find the data here; you have to log in before accessing any data. No charge, the data is open.

This data covers all domestic EPCs, that is, residential property.

https://epc.opendatacommunities.org/

This is provided under the Open Government License V3.0

Data

Available as a single download each month, but expensive at a 6.6GB compressed file, expanding into 2 main files:

1. The Certificates – 26.25GB csv – certificates.csv
2. Recommendations – 18.95GB csv – recommendations.csv

The download also includes a couple of useful references

• A list of each of the 100 fields, their name, data type and description. columns.csv
• A JSON Schema file, very useful for the first setup. schema.json

The recommendations file simply contains all the recommendations to improve energy efficiency, and the certificates file contains the property data.

This is a chunky dataset with just the certificates taking 26.25GB so depending on what you want it for, choose wisely!

There are 100 fields in the certificate file, and around 29 million certificates since 2008.

There are alternatives.  Files available monthly for the Local Authority only, and the lodgement date (year), so you can download smaller chunks.  

If you need non-domestic EPCs, they are available along with DECs.

Indexes and Joins

LMK_Key – unique identifier.
POSTCODE – The data contains a postcode, usefully formatted in a standard way, WR5 3EU, for example.  Consistent with other datasets. This makes a useful index and join option.

The address is not great, it’s split into ADDRESS1, ADDRESS2 and ADDRESS3.  The issue is with ADDRESS1, it merges house name, house number, flat number and street into a single field.  The land registry used PAON for house name or number, SAON for flat number.  So, accurate matching between the Land Registry and EPC tables is a bit more complex.  Refine down to the postcode level first, then use fuzzy searching if needed; otherwise, exact matching could and will fail on records.

I have chosen to leave ADDRESS1 as is and use a combined search.  Based on the Levenshtein ratio for near-match logic, and then use a heuristic weighting.  Not a pure mathematical distance measure, more hybrid heuristic.

As a standalone searchable dataset, indexes can be clean and simple.

Upload Handling and Maintenance

I prefer to upload the entire dataset, raw, no indexes, nothing.  staging_epc, fast to import, minutes rather than hours.  Then import from that table into live epc_certificates and create indexes, clean, well-thought-out schema for speed, only pulling in the fields that are required, but having everything available if needed.

Then, on the monthly update, I simply re-upload the full certificates.csv file into staging_epc and import from there into epc_certificates.  Clean, manageable, fast and means I can pull more fields over as needed should I need them.