Resending the Power PointJul 14 1 of 2View Source
Resending the Power Point
Hi there, This was interesting reading. I though it may be worth adding to this. Daniel Richman has recently rewritten the CUSF Predictor. We have been havingJul 14 1 of 2View SourceHi there,This was interesting reading. I though it may be worth adding to this. Daniel Richman has recently rewritten the CUSF Predictor. We have been having lots of issues requesting data sets from NOAA so we've decided just to download all of the GFS model four times daily. As a result the predictor is now pretty much instantaneous and it does work worldwide : http://predict.habhub.org/Daniels mail to UKHAS here :Hi everyone,
TL;DR: the predictor now works, and works very quickly at that!
http://predict.habhub.org (the old address redirects). While I've been
quite careful to make sure it behaves exactly the same as the old
predictor, it's a mix of old and relatively new software so should
perhaps be regarded as beta.
(It's also maybe worth reminding you that no guarantee is given for
the accuracy of any data included or produced by this program, use it
at your own risk.)
(not TL;DR - What follows is a brief tale of the problems of prediction:)
As you've no doubt aware, the predictor has been a little bit broken
over the last week.
This was, as almost every problem with the predictor is, due to the
NOAA wind data servers. (Though, we are of course very grateful for
them providing wind data for free :-).)
The predictor used to use their OPeNDAP servers. It's basically a Java
(and therefore slow) web application; we would ask it "can I have wind
data in this latitude/longitude range please". We've had some minor
issues with them (e.g., "blah is not an available dataset" due to its
domain resolving to two IP addresses with a very low TTL), but it was
generally OK. However, over the last week they've become too slow for
the predictor to get any wind data at all.
It turns out you can just wget the the entire forecast off a NOAA
HTTP/FTP server - so that's what we're doing now. Every 6 hours a
daemon downloads ~6-7GB - the entire forecast - and decompresses it
for the predictor to use. As a result, predictions complete
near-instantly; sometimes taking a second or two to bring the data
from disk into RAM, or if you're lucky and it's cached, about 20
I've upgraded the predictor and spacenearus' live predictions. I will
upgrade the hourly predictor soon (though am going on holiday
shortly...), however, the 8GB download may be prohibitive to people
running their own hourly predictors. Luckily, since we now have
worldwide data on the habhub servers, we can start running hourly
predictions for anyone that wants them ourselves. I'll email when this
It's worth noting that if you want a copy of the dataset yourself, it
might be better if we mirror it for you. Several benefits:
- the habhub.org servers (perhaps owing to us having gigabit, maybe
due to less load) are faster than the NOAA HTTP server
- only 25% of the 8GB download is actually useful, so we can mirror a
compressed 1.8G file.
- reduce load on the NOAA servers, because we're nice people
I'll have to double check that this is legal first. Let us know
(#habhub FreeNode, ukhas-tools@...) if you're interested.
This upgrade was actually part of larger CUSpaceflight work on
rewriting the predictor (replacing bits one at a time), which was
going to happen later; replacement of the wind downloading bit was
brought forward since the current one wasn't working.