I have (at last) managed to get hold of all the Mars Weather data that's been missing from my dataset. In fact, I've got hold of so much (over 110Gb) that it's going to take some processing - it bears little resemblance to the data I currently fetch. I need a plan.
Firstly, to avoid confusion, let's call the data I already had (and continue to fetch) as DATA-A, and this new data in its different form as DATA-B.
Having extracted DATA-B (and backed it up), I need to process data for days I already have in DATA-A. Let's call the resulting data, DATA-C. If DATA-C is the same as DATA-A, then my processing should replicate the data I'm missing.
Then for each sol that's missing from DATA-A, see if it exists in DATA-B and if so, process it and add the results to DATA-A. At the same time, I need to mark every record I'm adding as one I've processed myself, and therefore might not agree with NASA's equivalent data for that day. It should, but it's not guaranteed.
Then I need to highlight on the Mars Weather page, which data items in the tables I processed, and on the graphs (the rose graph and the Temperature Over Time graph, which elements were processed by me. This is not as straightforward as it sounds, especially with the TOT graph, but I'm writing a new graph script anyway to cope with data overflowing the graph boundaries, and the influx of all this new data will actually trigger that event early - so I can build it into the new graph rather than changing the existing one.
Which brings me neatly to the Temperature over Time graph. At the time of writing this article, each data point occupies 1.3 pixels (width). As more data is added, that shrinks, until each data point is exactly one pixel wide. As new data is added beyond that date, each data point must occupy less and less of a pixel - what's known as sub-pixel metrics - which is not physically possible. A new way to represent the data must be employed.
There are many options for this, but I think the one I have chosen is a good one. I did have some weeks to go before the data became sub-pixel, but the addition of all this new data (nearly 100 sols will be added to the beginning of the dataset) will force me to process the new data and replace the graph at the same time. I'd better get cracking!