Data wrangling

Data wrangling is a practice by which data is processed from an original, 'raw' source format or values, to a 'usable' form, more suitable for a given application or further applications. The process can include from relatively simple formatting or conversion from one format to another, to quality assessment, clean-up, gap filling, aggregation and even visualisation. For solar energy applications, in particular, solar radiation data is the primary source of information to be further processed and input to all kinds of models. Thus, in this chapter, we describe a basic but complete workflow for solar radiation data, and more specifically, solar radiation measurements at a surface level; satellite-derived solar radiation data is not explicitly included here, as users normally obtain these data from online services (free or paid) and the provided data have already been processed, with no further checks needed from the end user (with the possible exception of verification or site validation of the values, but this is a topic beyond the scope of this chapter); however, formatting of the files provided by these satellite data services may be needed, so parts of this chapter may be relevant for these cases too. A word of caution: the procedures described in this chapter will alter the contents of the data files, although (important to stress) not the measured values, but the format and handling/representation of missing or 'bad' data. For numerous reasons, from practical to traceability and even legal in some cases, it is not a good idea to modify the original ('raw') data files, so any and all changes that may be needed should be done in separate copies, never overwriting the original files, which should be kept safe.