NTP performance, events, and notes on using NTP (or alternative presentation)

	PC Feenix Intel Pentium 4 1.9GHz, 1GB Windows XP SP3 Main EUMETCast RX ntpd 4.2.6 GPS 18 LVC GPS/PPS reference
	PC Stamsund 2.8GHz Pentium 4 HT, 3GB Windows XP Pro SP3 EUMETCast backup ntpd 4.2.6 GPS 18x LVC GPS/PPS reference
	PC Bacchus Intel Pentium III 550MHz, 512MB Windows 2000 Server SP4 APT RX ntpd 4.2.6 Synched from PC Feenix (sometimes from GPS/PPS)
	PC Gemini AMD-64 X2 4400, 3GB Windows Vista Ultimate SP2 Test-bed ntpd 4.2.6 Synched from PC Feenix
	PC Hydra AMD 64 3200+, 2GB Windows-7 Ultimate RTM EUMETCast - test PC ntpd 4.2.4p6 (273) Synched from PC Feenix
	PC Narvik Intel E6600, 3GB Windows XP Pro SP3 Interactive ntpd 4.2.6 Synched from PC Feenix
	PC Puffin Intel E8300 dual-core, 3GB Windows Vista SP2 Interactive use ntpd 4.2.6 Wifi synched from PC Feenix
Click on a graph for weekly, monthly and yearly data

Please note, because I happen to use MRTG to gather and plot this data, and negative values aren't allowed, I needed to add a bias to the actual offset to derive the graphs above (100ms for Gemini, 3ms for most PCs, and 500µs for the much more accurate PCs Feenix and Stamsund).  An ideal timekeeper would therefore display a straight line at the mid-scale level.  The value plotted is the offset from the server clock that the NTP servers on each of the PCs reports when interrogated, every five minutes.  There appears to be an oddity with MRTG in that is looses small figures in the year data, so you may see a gradual drift upwards in some of the data, and that is probably just a recording artefact.
How I obtain this data.

Hardware and OS configuration

Normally, there are normally two stratum-1 NTP servers, peered together, each fed from a separate GPS receiver.  One of the GPS receivers had its RS-232 output parallel-connected so that a temporary test could be made, and this was most recently used with a serial-to-USB converter box for testing the feasibility of using a USB connection where the PC has no serial port.  The parallel connection is also used for checking test versions of ntpd.exe on PC Bacchus, so Bacchus may well be a stratum-1 server at any particular time.  The performance graph should be a good indicator!  Internet backup servers are configured for all PCs.

Active NTP Versions

Configuration changes and other events

Please note that some transients are caused by system reboots, e.g. after a security update, and these events are not usually individually recorded.

• 09:27 UTC, Thursday, 2010 Feb25, week 8.  Updated wireless PC Puffin from ntpd 4.2.4p6@DLH-QPC-o May 30 3:58:32.88 (UTC) 2009 (273) to ntpd 4.2.6-o Dec 09 11:48:30.27 (UTC-00:00) 2009 (1) as an experiment.  I hope that this will make little or no difference to the timekeeping, but knowing the issues with Vista I am not sure.  Noted that I tried this on Dec 07 last year, but with no results noted.  The idea later  is to make the polling interval less for the local stratum-1 server (reduce 64s to 32s), but to increase the Internet server poll right up to 1024s - only the later 4.2.6 allows this.  At the moment, the Internet servers are being polled every minute, and I would like to stop this.  14:24 - looks like the jitter is worse, but I have now set the stratum-1 local poll down to 32s (maxpoll 5) and upped the Internet poll to 1024s (minpoll 10).
• 06:45 UTC, Tuesday, 2009 Dec 22, week 52.  Noted that PC Hydra was getting its event log full with events where NTP was bouncing between the two stratum one servers, so I restored the "prefer" keyword for server Feenix.  This PC is running the older 4.2.4p6 (273).  The PCs running 4.2.6 do not show this clock-hopping.
• 08:35 UTC, Saturday, 2009 Dec 12, week 50.  After reboot of Feenix, noted that some of its client PCs followed its transient rather than changing to the still-accurate Stamsund.  I therefore removed the "prefer" from the Feenix entry on the three main client PCs (Gemini, Hydra and Narvik) and restarted NTP on those systems.  Now need to keep an eye open for clock-hopping!  Noted that on Hydra, at least, the averaged jitter seemed to be quite a lot lower, dropping from a variable 1.12-1.18ms to a steady 0.98us.  The instantaneous jitter was much steadier.  Was this simply the NTP restart, or the removal of the "prefer" keyword?  Narvik showed no average jitter change - 30-65µs.  Gemini showed no average jitter change - 5-13ms.
• 14:10 UTC, Wednesday, 2009 Dec 9, week 50.  Updated PCs to ntp 4.2.6 from Dave Hart - 14:10 Narvik, 14:15 Stamsund, 14:20 Gemini, 14:25 Bacchus, 14:30 Feenix.  This leaves one Windows Vista PC (wireless LAN sync) and one Winodws-7 PC (direct connection LAN sync) still running 4.2.4 because of unresolved performance problems.
• 10:15 UTC, Sunday, 2009 Dec 06, week 49.  Updated Windows XP GPS/PSS-synced PC Feenix from ntpd 4.2.5p231-RC to ntpd 4.2.5p250-RC.
• 07:30 UTC, Sunday, 2009 Dec 06, week 49.  Started monitoring NTP on PC Puffin with ntpd 4.2.4p6@DLH-QPC (230).  At 15:48 UTC changed to ntpd 4.2.4p6@DLH-QPC (273), with the thought of trying 4.2.5 later.  20:00 UTC installed ntpd 4.2.5p250-RC.  06:00 UTC Monday, re-installed ntpd 4.2.4p6@DLH-QPC (273), but noted that the 4.2.5p250 ntpq -p running on Puffin returned "no such file or directory" when run against "puffin", but worked against other PCs, and a 4.2.5p250 running on another PC correctly showed the data from PC Puffin.  Wonder if it's because IPv6 may be enabled by default on Puffin?  Yes, running the local ntpq against the IPv4 address works correctly.
• 11:28 UTC, Saturday, 2009 Dec 05, week 49.  Updated Windows Vista, LAN-synced PC Gemini from ntpd 4.2.5p246-RC to ntpd 4.2.5p250-RC.  Appears OK.  11:38 - updated Windows 2000 Server, GPS/PSS ref-clock synced PC Bacchus from ntpd 4.2.5p231-RC to ntpd 4.2.5p250-RC.  Also appears OK.
• ~08:59 UTC, Friday, 2009 Dec 04, week 49.  PCs Feenix and Stamsund lost PPS signal from the roof-mounted GPS puck.  Powered the puck down and back up, and the PPS signal restarted OK.  Those PCs NTP picked up the returned PPS signal right away.
• 07:05 UTC, Tuesday, 2009 Dec 01, week 49.  Updated NTP on Narvik from ntpd 4.2.5p231-RC to ntpd 4.2.5p250-RC.  At 07:20, updated Stamsund from ntpd 4.2.5p246 to ntpd 4.2.5p250-RC, so that version 250 gets a good checkout.  This introduced a 11ms step, from which it will take a little while to recover.
• 06:26 UTC, Saturday, 2009 November 28, week 48.  Reduced LAN poll rate on Narvik from 64s to 32s.  Has a noticeable affect on reducing the offset.
• 06:30 UTC, Thursday, 2009 November 26, week 48.  Reduced LAN poll rate on Hydra from 64s to 32s for an experiment.  If there is no ill effect from these faster polling rates for Gemini and Hydra I'm likely to leave them.  This has reduced the offsets seen on Hydra.
• 07:45 UTC, Wednesday, 2009 November 25, week 48.  As an experiment on Gemini, left 4.2.5p246 running, but increased the polling rate from maxpoll=6 to maxpoll=5 (i.e. from 64 seconds down to 32 seconds maximum interval).  The idea is to see what reduction in visible offset jitter might be achieved.  I also want to see whether there is an visible effect on the network traffic to and from Gemini.  No noticeable effect on network traffic.  No noticeable effect on the network traffic.  Results don't seem to be a lot different, but perhaps I should to back to 4.2.4p6 (273)?  Tried this, but 4.2.4 won't accept maxpoll=5.  Another odd thought - should I kill the unused IPv6 connectivity I have?  As NTP doesn't seem to report using servers on IPv^, I suppose the connectivity doesn't matter?
• 10:15 UTC, Tuesday, 2009 November 24, week 48.  Updated Vista PC Gemini to 4.2.5p246 following a request for testing from Dave Hart.  Updated Windows-7 PC Stamsund at 10:25.  Both appear to work correctly, although Stamsund had a 50ms step which will spoil the graphs!
• 07:09 UTC, Monday, 2009 October 26, week 44.  Step of NTP frequency on Stamsund from 3.4 to 6.4ppm from 07:09 to 07:15 UTC, followed by a recovery over the next five hours or so.  Nothing in the event logs suggesting why - it could have been a bad set of lost interrupts. Instead of the typical 0.45ms steps in jitter there was a 5.9ms step, offset showed a +16.6ms step.
• 12:40 UTC, Monday, 2009 Oct 19, week 43.  Broken GPS 18x LVC was replaced by Garmin (no comment as to fault), so that unit was used on PC Stamsund (no restart required) and the parallel feed from the external GPS was fed to PC Bacchus (restart of NTP required after editing the ntp.conf, Windows 2000).  It seems that there may be a known issue with these units, when the firmware is earlier than 3.10.
• 06:37 UTC, Saturday, 2009 Oct 17, week 42.  Noted that since the Vista PC Gemini's reboot for security updates on Wednesday 14, NTP seemed much less stable.  The jitter figures confirmed this.  Tried restarting NTP in case it had not picked up something at the reboot (although it did say "using native clock directly").  NTP seemed to have been rather stable between about October 3/4 and Oct 14.  Noted that after the reboot, the average CPU for the tc-recv.exe process had increased from 1.5% to 2.0%, and that the working set for FlightDisplay had increased by a couple of MB.
• 14:00 UTC, Saturday, 2009 Oct 10, week 41.  Updated Narvik (Win XP Pro) from ntpd 4.2.5p212 to ntp-4.2.5p231-RC.  14:09 updated Stamsund (Win-7 Ultimate).  14:17 updated Feenix (Win-XP Home) from ntpd 4.2.5p186 to ntp-4.2.5p231-RC.  15:23 updated Bacchus (Windows 2000 Server) from ntpd 4.2.5p181 to ntp-4.2.5p231-RC.  This version appears to work OK at first glance.
• Early on Wednesday, Oct 07, week 41, the new GPS 18x LVC appeared to fail, so the parallel feed from the main roof-mounted GPS 18 LVC was removed from Bacchus and connected to Stamsund.  Note that NTP on Bacchus failed when the serial source was disconnected, requiring an edit of the ntp.conf to comment out the GPS serial port and ATOM driver (PPS) lines.
• Sometime on either Tuesday, 2009 Sep 22 or Wednesday, 2009 Sep 23, week 39, NTP became very unstable on the Vista PC Gemini.  As I was away at a conference at the time, it wasn't anything I did!  There was a Windows Defender automated update and run at 04:00 UTC on the Wednesday morning, so perhaps that was the cause.  Offset on the weekly graph changed from small peaks of about 10ms to variations peaking at 100ms.  Averaged jitter increased from around 3ms to around 6.5ms, but the offset    The frequency offset increased from a stable 5ppm to a rather instable 55ppm, and there were a number of "no servers available" message in the event log, suggesting that NTP couldn't contact the servers, or there was some sort of network error.  No other applications seemed to complain about a network problem.  Restarted NTP at 07:25 UTC on Tuesday, 2009 Sep 29.  Decided to reboot, as both the TelliCast had restarted at 08:37 UTC, and I got a disk full error message from Plane Plotter when drive F: had plenty of space.
• 07:20 UTC, Thursday, 2009 Sep 17, week 38.  Updated Stamsund NTP from ntpd 4.2.5p186 to ntpd-4.2.5p212, to check a current version with the serial GPS/ATOM drivers.
• 05:10 UTC, Wednesday, 2009 Sep 16, week 38.  Updated Narvik NTP from ntpd 4.2.5p186 to ntpd-4.2.5p212, 05:15 updated Gemini NTP from ntpd 4.2.4p6 to ntpd-4.2.5p212, 05:20 updated Hydra NTP from ntpd 4.2.4p6 to ntpd-4.2.5p212.  The behaviour of 4.2.5p212 seems little or no different to other 4.2.5 versions on Vista and Windows-7.  It seems no worse on Windows XP, so I'll likely leave it running there, but I've reverted top 4.2.4p6 in the Vista and Windows-7 system from 05:05 UTC on Sep 17.
• 00:50 UTC, Monday, 2009 Sep 07, week 37.  Changed the ntp.conf on Gemini (Vista) to have "tick 1", and changed from ntpd 4.2.4p6(273) to ntpd 4.2.5p186, following a request from Dave Hart.  PC Vista is badly behaved for timekeeping in any case.  Offset looks noisier and jitter is higher, but it's workable.  00:55 changed Windows-7 PC Hydra as well, but discovered that it had found the wrong value for "Windows clock precision", so restarted again just after 05:00 UTC.  18:30 UTC tried changing tick to 0.3 on Gemini - not much difference.  05:00 UTC Sep 08, tried tick as 0.1 on Gemini.  On the Windows-7 PC Hydra, this just doesn't seem to have worked at all, but in the spirit of experimentation I tried changing tick to 10 at 05:23 on Sep 08, but it made no obvious difference.  This seems to have made little or no change to the jitter, so Hydra was returned to the older 4.2.4p6(273) NTP at 07:54 UTC, and as the offset variation and jitter on Gemini was also made worse by the change, Gemini was reverted at 08:00 UTC.
• 09:38 UTC, 2009 Aug 24, week 35.  Updated Stamsund to use the slightly later 4.2.5p186 replacing 4.2.5p181, which had been the version is use when this PC had been running Windows XP SP3.
• 2009 Aug 23, week 34.  Rebooted Stamsund around 08:00 UTC, and got good timekeeping restored.  Talked with Dave Hart who suggested setting the environment variable NTPD_USE_INTERP_DANGEROUS=1 to force NTP to use the interpolation code in any case.  This also worked nicely.  Why Windows-7 behaves like this on this particular PC is unknown as yet.
• 2009 Aug 22, week 34.  Installed different network card on Stamsund at about 11:10 UTC, and NTP went well until I restarted it at about 13:30 rather than rebooting.  It dropped the interpolation code and the timekeeping went wild.
• 2009 Aug 18, week 34.  Installed Windows-7 RTM on Stamsund.  NTP seemed surprisingly good running as a stratum-1 server, with an increased jitter compared to Windows XP, but a much reduced offset.  Used version [ntpd 4.2.5p181-o Jun 06 15:21:08.65 (UTC-00:00) 2009 (1)] by accident.
• 0730-0800 UTC, Thursday, 2009 Jul 08, week 29.  Updated Narvik, Stamsund and Feenix to "ntpd 4.2.5p186-o Jul 09 3:58:51.88 (UTC-00:00) 2009 (2)ntpd 4.2.5p186-o Jul 09 3:58:51.88 (UTC-00:00) 2009 (2)" - the latest from Dave Hart.  Tried to update Bacchus but it gave an unknown software exception 0xC0000417 at location 0x00444D99, so I reported it to Dave.  Left the Vista and WIndows-7 PCs alone.
• 0525 UTC, Monday, 2009 Jul 06, week 28.  Updated NTPD on Gemini to 4.2.4 (273) as it is the most recent 4.2.4 that I have [ntpd 4.2.4p6@DLH-QPC-o May 30 3:58:32.88 (UTC) 2009 (273)].  It will then be the same as on the Windows-7 PC.
• 1415 UTC, Thursday, 2009 Jul 02, week 27.  Some disturbance which caused NTP on Hydra to loose some network connectivity.  Restarted NTP, and as it was still version 230, it discovered the wrong system clock precision and performed poorly.  Neither a soft reboot or a power-down reboot helped, until I recalled that I still had version (230).  Updated to version (273) and normal operation was restored.
• 0940 UTC, Sunday, 2009 Jun 14, week 25. Updated Hydra to ntpd-QPC-20090614-0900.zip (ntpd 4.2.4p6@DLH-QPC-o May 30 3:58:32.88 (UTC) 2009 (273)) to see whether the initial detection worked any better.  It may do.  Replaced version (230) to see what happens at the next reboot.  Yes, it failed again.
• 1230 UTC, Saturday, 2009 Jun 13, week 25.  After a reboot of Hydra, noted that NTP was performing much worse than expected.  Hadn't settled after several hours, so stopped and restarted the NTP service, and all was well.  Seems that NTP may not have recognised the need to switch off its interpolation code when running under Windows-7.
• 1053 UTC, Tuesday, 2009 Jun 09, week 25.  As version 4.2.4p6(230) works well on the Windows-7 PC Hydra, tried in on the Vista PC Gemini just to see how well it faired.  Nicely, thanks very much.
• 0841 UTC, Monday, 2009 Jun 08, week 24.  PC Hydra still giving poor performance with ntp 4.2.5.  Noticed that it had the -U 3 command-line parameter which would cause it to scan the network interfaces every three seconds, which seems excessive on that PC compared to the default of every 5 minutes, so removed that parameter and value to see whether it made any changes.  1848 UTC - reverted to 4.2.4p6 (230) on Hydra - couldn't stand the poor performance any longer!
• 1530 UTC, Sunday, 2009 Jun 07, week 23.  Updated PCs in sequence to "ntpd 4.2.5p181-o Jun 06 15:21:08.65 (UTC-00:00) 2009 (1)".  Narvik (1528), Stamsund (1533), Gemini (1554), Hydra (1605), Bacchus (1615), Feenix (1621).  Hydra (Windows-7 RC) is not as good as it was with "ntpd 4.2.4p6@DLH-QPC-o (230)" - see note from Sunday 2009 May 17.  However, I did set the NTP account to have the Increase Scheduling Priority right, in case that helps, and restarted at 1730 UTC.
• 0605 UTC, Sunday, 2009 Jun 07, week 23.  Updated ntp on Stamsund to the (13) version to see if the jitter was reduced by using the kernel-mode timestamping in the new DLL-based provider.  Needed to update to the serialpps.sys driver as well, of course, which needed a reboot.
• 0953 UTC, Saturday, 2009 Jun 06, week 23.  Updated "ntpd 4.2.5p175@1.1858-o May 12 15:20:06.38 (UTC-00:00) 2009 (1)" on Narvik to "ntpd 4.2.5p180@1.1877-o Jun 05 10:01:52.09 (UTC-00:00) 2009 (13)" to test the higher numbered serial ports.  Ports 4, 15, 101, and 255 work OK.  Port 256 does not, but that is outside the range allowed.
• 0510 UTC, Saturday, 2009 Jun 06, week 23.  On PC Feenix, updated "ntpd 4.2.5p175@1.1858-o May 12 15:20:06.38 (UTC-00:00) 2009 (1)" to "ntpd 4.2.5p180@1.1877-o Jun 05 10:01:52.09 (UTC-00:00) 2009 (13)" to confirm correct operation.  Appears OK, so at 0935 UTC enabled the new DLL-based atom/pps provider.  On the average jitter plots, you could clearly see where the kernel-mode PPS support was enabled, and the average jitter was reduced from about 3.5 µs to under 2.5 µs.
• 1300 UTC, Friday, 2009 Jun 05, week 23.  Restored duplicate serial-GPS feed to PC Bacchus and confirmed operation.  Updated to "ntpd 4.2.5p180@1.1877-o Jun 05 10:01:52.09 (UTC-00:00) 2009 (13)" and confirmed correct operation.  Later, tried adding the serialpps support DLL, but needed to alter the server order to put the Atom driver before the NMEA driver to get it to show in the ntpq -p display.  Kernel-mode does not appear to be working yet, at least on this PC.  Later that evening, updated the DLL to one which produced fewer error messages.  The interpolation is no so good on this 100Hz system, and you could not clearly see where the kernel-mode PPS support was enabled..
• 0537 UTC, Saturday, 2009 May 30, week 22.  Updated NTP on Bacchus from: ntpd 4.2.5p161@1.1825-o Apr 02 11:57:05.06 (UTC-00:00) 2009 (9) to: ntpd 4.2.5p180@1.1877-o May 29 16:02:39.34 (UTC-00:00) 2009 (8).
• 0500 UTC, Saturday, 2009 May 23, week 21.  Noted that PC Narvik was synced to Stamsund, rather than the lower-offset Feenix, so made Feenix "prefer" and restarted NTP.  Interesting to see whether there is any difference - at least clock hopping should stop.
• 1655 UTC, Friday, 2009 May 22, week 21.  Removed parallel RS232 feed from from roof-mounted GPS 18LVC to PC Stamsund, and replaced it with an indoor GPS 18x LVC (should be more sensitive) and USB-powered +5V.  2110 UTC - made Feenix and Stamsund a peered pair.  Initially they rejected each other ("x" in ntpq -p), but later changed to candidate ("+").
• 1620 UTC, Thursday, 2009 May 21, week 21.  Removed the GPS-serial-USB source from PC Narvik, as testing was complete.
• 1520 UTC, Sunday, 2009 May 17, week 20.  Switched PC Bacchus back to 4.2.5p161, once I had worked with Dave Hart to resolve the problem in 4.2.5p175 (which would also have been a problem in the stable branch).  Without the improved interpolation, the performance on Bacchus was noticeable poorer (10+ms vs. 3ms offset, 0.6ms average jitter vs. 0.1ms).  1526 UTC switched PC Hydra back to (230) with the better support for Vista/Windows-7, by /not/ using the 1ms interpolation when the system clock is already running at 1ms intervals.  (20ms "noisy" offset vs. 2ms "cleaner", 8-13ms average jitter vs. 1.1-19ms).
• 0805 UTC, Friday, 2009 May 15, week 20.  Updated PC Stamsund to 4.2.5p175@1.858-o.  0813 updated PC Gemini.  Didn't work on PC Bacchus (seemed to corrupt the Event Log, so re-installed ntpd 4.2.5p161@1.1825-o.  08:43 updated PC Hydra, running Windows-7 RC (7100).  0905 updated PC Feenix, but here we seem to have lost access to the ATOM driver (was version ntpd 4.2.4p6@DLH-QPC-o Mar 15 21:30:20.47 (UTC) 2009 (239) before).
• 1545 UTC, Saturday, 2009 May 09, week 19.  Added serial-over-USB ref-clock to PC Narvik.  May 12 updated to 4.2.5p175@1.858-o to check that the new COM4 support worked - it does.
• 0747 UTC, Saturday, 2009 May 09, week 19.  Noted that NTP wasn't running at real-time priority on Stamsund, so used secpol.msc to allow this, and restarted NTP.  I don't expect it will make a lot of difference - it will be interesting to see whether it makes any difference either to the NTP jitter, or to the TelliCast performance.  No significant change to NTP performance.
• 1330 UTC, Wednesday, 2009 May 06, week 19.  Changed PC Feenix to use an indoor GPS 18x LVC rather than the outdoor unit.  The 18x is more sensitive, but indoor coverage will be worse.  So this is just a test to see how good or how bad the performance might be.  My estimate is that the basic timekeeping accuracy should be unchanged.  There may be a transient at the changeover time.  Let's see!
• 0615 UTC, Tuesday, 2009 May 05, week 19.  Changed PC Hydra (Windows 7) from Meinberg VegasV2 to Dave Hart's v.230, so that it is the same version as Gemini, and we may be able to compare Vista with Windows 7, albeit on two different PCs.  They have the same 64s poll of local ref-clocks and 1024s poll on Internet backup servers.  I will be looking to see if the spikes seen on PC Gemini are also seen on PC Hydra.
• Sunday, 2009 May 03, week 18.  Moved PC Hydra onto Windows 7, initially using Meinberg VegasV2.
• 0815 UTC, Friday, May 01, week 18.  Had noticed increased EUMETCast packet loss on Gemini since April 21/23 so, while I don't think this is due to NTP changes, reset Gemini to use Dave Hart's v.230 (March 10).  Seemed to be the most recent DLH version (apart from p163).  I don't seem to have a DLH v.239 to hand on that PC.
• 1100 UTC, Wednesday, 2009 Apr 29, week 18.  Edited Perl MRTG monitoring script to limit just below 6000µs so that limiting values on Bacchus, Hydra and Narvik can be more accurately represented.
• 1536 UTC, Wednesday, 2009 Apr 22, week 17.  Tried setting the MMCSS service on the Vista PC Gemini to disabled.  It might affect the errors on the DVB system (by not prioritising the multi-media network traffic), or improve the NTP performance by not delaying its network I/O.  Or it may have no effect!  Seemed to have no effect, so restored the service at about 10:30 UTC on Thursday 23rd.
• 0800 UTC, Tuesday, 2009 Apr 14, week 16.  It seems I have been comparing the wrong versions to check bug 1030, I should compare p162 or later versus e.g. Meinberg VegasV2.  So replaced p163 with the VegasV2 on Gemini to see what happens....
• 0645 UTC, Sunday, 2009 Apr 12, week 15.  Updated PC Gemini from Dave Hart's (239) of March 15 to test build "4.2.5p161@1.1828-o Apr 09 9:56:38.95" to see whether the the jitter stays normally low and asymmetrical (as in recent -stable builds), or whether it reverts to much higher levels but symmetrical as in a recent -dev test version (as noted here).  Suspect this should have been build 163 rather than 161, but it's OK, it will establish how 161 performed.  14:20 UTC, updated to true test build 163.  "ntpd 4.2.5p163@1.1831-o Apr 12 12:22:16.13 (UTC-00:00) 2009 (2)"
• 0630 UTC, Saturday, 2009 Apr 11, week 15.  Started new graphs for Bacchus, Hydra and Narvik with a +/-3ms range rather than a +/- 100ms range.  The old data is still being collected and can be seen here.
• 1300 UTC, Thursday, 2009 Apr 02, week 14.  Updated PCs Bacchus, Narvik and Stamsund to Dave Hart's latest -dev release 4.2.5p161@1.1825 (9) for testing.  (About 1100 had updated PCs Bacchus, Narvik and Stamsund to have maxpoll 6 on the local ref-clock servers, and minpoll 10 on the Internet servers.  Had been running Hydra that way for some time.
• 1445 UTC, Tuesday, 2009 Mar 31, week 14.  Disappointed with the very poor offset and five times worse jitter of the -dev branch of ntpd on Gemini, reverted to the (239) build of Dave Hart's 4.2.4 -stable version.
• 0900-0910 UTC, Saturday, 2009 Mar 28, week 13.  Changed NTPD to Dave Hart's 4.2.5p158@1.1823-o (5) version on PCs Narvik, Stamsund (ref-clock) and Gemini (Vista).  This is the "dev" rather than the "stable" version, and does include the leap-second majority vote code.  0940 UTC, updated Windows 2000 PC Bacchus as well, since Dave confirmed this is the 1-receive-per-socket coding.
  1025 UTC - changed Hydra to have minpoll=10 for the Internet servers, maxpoll=6 for the local GPS/PPS servers.
• 0900 UTC, Wednesday, 2009 Mar 25, week 13.  Restarted NTPD as a regular service on Bacchus with Dave Hart's 1-receive socket (254).  When tested after 24 hours earlier this morning, this version did not crash, so testing for a longer spell as a service is next.
• 0600 UTC, Tuesday, 2009 Mar 24, week 13.  Retired PC Pixie as the prime GPS reference PC, to be replaced by PC Feenix.  Added PC Stamsund as a second NMEA/PPS system, as a backup.
• 1820 UTC, Friday, 2009 Mar 20, week 12.  Changed Bacchus to Dave Hart (239) running in debug mode to see why I couldn't stop NTPD cleanly earlier.  1835 updated Feenix to (239) for its better leap-second handling.  1842 updated Gemini.  1847 Updated Hydra.  1855 updated Stamsund.  1940 Bacchus NTP shut down cleanly with Ctrl-C from the command line, so restarted it for a longer run.
• 0645 UTC, Monday, 2009 Mar 16, week 12.  Changed PC Bacchus back to Vegas.V2 to compare the jitter on this 100Hz system.  Changed Narvik to Dave Hart's (239) to check for the leap-second majority code.  At his suggestion, updated the other programs as well (e.g. ntpq).
• ~1900 UTC, Saturday, 2009 Mar 14, week 11.  Reset all PCs (except Feenix) to use the (230) release of Dave Hart's version.  PC Bacchus was reluctant to let the .exe go after stopping the service, so needed a reset-reboot.
• 0705 UTC, Friday, 2009 Mar 13, week 11.  Reboot Feenix, twice, to install the serialpps.sys device driver, and see how NTP performed with the kernel-mode rather than user-mode time-stamping.  This didn't seem to alter the mode, so I added the ATOM driver as well, which worked.  The lack of a stratum-1 server did upset Bacchus which was syncing with a fixed 64s maxpoll.  0951 restarted NTP on Gemini to pick up second stratum-1 server and Internet servers (emergency use only).  1145 restarted NTP on Bacchus and Stamsund to pick up the new configuration files (back to 1024s polling).
• 0830 UTC, Thursday, 2009 Mar 12, week 11.  Restarted Bacchus without the NTPD_PCC_FREQ = 551240000, to force it to use QueryPerformanceCounter rather than RDTSC.  22:46 restarted Bacchus with maxpoll 6 (64s) rather than maxpoll 4 (16s), as this is closer to the intended final configuration.  Internet servers still at minpoll 10 (1024s).
• 1726-1745 UTC, Tuesday, 2009 Mar 10, week 11.  Tried to restart Feenix with the NTPD_PCC_FREQ set, but this not sync even after 15 minutes, so reverted to new version but without NTPD_PCC_FREQ.  This seems to have slightly less jitter.
• 0707 UTC, Monday, 2009 Mar 09, week 11.  Finally restarted Gemini's NTP with just three local servers (one stratum 1) and a maxpoll of 6 (64 seconds).
• 2115-2135 UTC, Sunday, 2009 Mar 08, week 10.  Restarted NTP on Vista PC Gemini to minpoll 4 maxpoll 4, single server (pixie) for a final comparison.  
• 0615 UTC, Sunday, 2009 Mar 08, week 10.  Restarted NTP on Vista PC Gemini back to maxpoll 10, for a comparison.  0620, restarted NTP on PC Feenix to change the priority from high to real-time.  I would have just used the Task Manager, but I didn't have the priority, so I chose to add NTP to the Administrators local group and restart.  Restarted Feenix NTP at 0834 and 0906 to pick up updates from Dave Hart.
• 0925 UTC, Saturday, 2009 Mar 07, week 10.  Switched the Vista PC Gemini down to a maxpoll=6 (64s) just to show how flat the graph can be - or not.
• 1930 UTC, Friday, 2009 Mar 06, week 10.  Downloaded Dave Hart's v.204 and installed same on all Windows PCs.  No configuration changes.
• 1015 UTC, Friday, 2009 Mar 06, week 10.  Rôle reversal for Bacchus and Feenix.  Transferred serial port from Bacchus to Feenix, to see whether the 64Hz clock and much higher specification processor would perform better.  If this works, will most likely keep Feenix as the reference server for me, so that PC Pixie and perhaps PC Bacchus can be retired.
• 1025 UTC, Thursday, 2009 Mar 05, week 10.  Continued experiments on Bacchus by adding the atom ref-clock back in, as it will eventually be required when serialpps support is removed from Dave's ntpd.exe.  1225 UTC, moved Narvik, Stamsund and Gemini onto V.177, and added Bacchus as a server, making each of these PCs have two PPS devices.  Thinking towards when Pixie may be switched off.  1400 UTC, also installed on Hydra, but it failed because the NTP account wasn't a member of the Administrators group.  Fixed with:
    net localgroup administrators ntp /add
• 1000 UTC Wednesday, 2009 Mar 04, week 10.  Updated Bacchus to Dave Hart's V.177.  May have reduced jitter after 45 minutes operating.  1400-1430 - tests with INT=26 to INT=32.  Left running with INT=27 which seemed best.
• 0600 UTC, Monday, 2009 Mar 02, week 10.  Continuing experiments with Gemini, I restored the GPS-server Bacchus, and changed to maxpoll to 9.  This is one combination we haven't yet tried.  Changed to maxpoll from 4 to 6 on Bacchus (16s -> 64s).  This triggered another 20msec step, positive this time.  0756 - Dave Hart noted that Bacchus is a 100Hz system, where his interpolation code requires a slightly different sample interval to the 43msec default..  So I set environment variable NTPD_INT_INT=26 and restarted Bacchus.  There was a -120ms transient at the restart, which was taking a long time to settle down, so at 16:44 I changed the minpoll to 4, and restarted ntpd on Bacchus.  Tuesday 0312, restarted on Bacchus  with with NTPD_INT_INT=28 - this looks to work a little better, and has reduced the amplitude of jitter in the offset while increasing its frequency.  0610 added ATOM driver: server 127.127.22.1 minpoll 4.
• 1130 UTC, Sunday, 2009 Mar 01, week 10.  Cut the GPS/PPS feed to Pixie for few minutes while I soldered in a parallel RS-232 feed for tests on using a Windows PC with PPS (Bacchus).  This cut resulted in a transient, but that had recovered by about 12:30 UTC.  The graphs for Bacchus (with a serial GPS/PPS ref-clock) starting at about 17:40 on Sunday don't look too good.  An unexplained 20msec negative step at around 19:15.
• 2135 UTC, Friday, 2009 Feb 27, week 09.  As an experiment, changed the Vista PC Gemini to sync purely from local sources (the stratum 1 GPS/PPS and a couple of others as backup), removed the Internet servers, and set maxpoll to 6, thus checking every 64 seconds.  This looks good, so let's change it to maxpoll 8 from the next boot (due at 06:55 UTC, Saturday, 2009 Feb 28, week 09).  This may, or may not, still prevent NTP from going into that zone where "the frequency graph matches the offset graph", which I feel is not a correct way to operate.  Surprisingly, NTP on Gemini seems to have gone crazy.  It's running at real-time priority (presumably as the account is now a function of the admins group), so at 11:30 UTC I tried changing the affinity to a single processor.  No difference, so I just stopped and restarted NTP to see what happens.  Ah, my own fault.  I discovered that the -M had been omitted from the command-line for the NTP service, and that's why the jitter had been awful.  With Dave Hart's version the -M is essential (at least on Vista).  Found that with maxpoll=8 the typical offset was -7.5ms with transient offset peaks up to +0..+2.5ms with a couple up to +17..18ms.  0600 Sunday, just as another experiment, removed the local GPS/PPS server (Pixie) and left just four Internet servers, and will allow the max poll to reach 10 (1024s) - transients after 24 hours were in excess of 50ms, so reverted to using the local GPS/PPS server.  With maxpoll=9, the typical offset was 0, with most of the jitter being positive excursions of 20-30ms.
• 0840 UTC, Thursday, 2009 Feb 26, week 09.  As an experiment, changed Narvik so that just the local GPS/PPS server was polled at 128s intervals (maxpoll 7), in the hope that the Internet servers would eventually be polled at the standard 1024s intervals.  Change undone at 0900 UTC, Friday, 2009 Feb 27, as all servers were polled at 128s intervals, which is not normally acceptable for outside servers.
• 0725 UTC, Thursday, 2009 Feb 26, week 09.  Installed Dave Hart's latest update (157) on the Windows XP SP3 PC Stamsund, and at 07:45 on the Vista SP1 PC Gemini.  On Gemini it stayed stable for several hours, but then reverted to oscillating.  The short-term jitter on Stamsund was reduced, but it's too early to judge whether the longer-term performance has been improved.
• 1125 UTC, Monday, 2009 Feb 16, week 08.  As an experiment, I restarted ntp on PC Gemini, having altered the config file to exclude all the Internet servers and just leave the local LAN GPS-based server Pixie.  See how that does (although I suspect I've tried this before).  Seems to have reduced both the amplitude of the oscillations, and the tendency to limit.  So at 21:20 I added the Internet servers back in, a restarted NTP on PC Gemini.  Having the Internet servers may have made things a little worse, but not back to the extreme swings.  It's almost as if the act of stopping and restarting the NTP service improves its stability.  I checked, and the ntpd.drift file does exist, is being updated, and does contain a sensible-looking value.
• 0930 UTC, Monday, 2009 Feb 16, week 08.  Changed the priority of ntpd.exe on PC Gemini from "high" to "realtime" at Dave Hart's suggestion, but this seemed to make little difference.
• 0655 UTC, Sunday, 2009 Feb 15, week 07.  Rebooted PC Gemini after a security update.  NTP performance seems worse - back to the old ways with extreme oscillations.  Thought that perhaps I had the old version back, but "ntpq -rcv gemini" still shows version (69).
• 0755 UTC, Monday, 2009 Feb 09, week 07.  Updated XP PC Stamsund to Dave Hart's version (69).
• 0615 UTC, Monday, 2009 Feb 09, week 07.  Updated Vista PC Gemini to Dave Hart's version (69).
• 1623 UTC, Sunday, 2009 Feb 08, week 06.  Updated Vista PC Gemini to experimental ntpd from Dave Hart - version "ntpd 4.2.4p6@DLH-QPC-o Feb 08 12:16:40.85 (UTC-00:00) 2009 (68)".
• 2130 UTC, Saturday, 2009 Feb 07, week 06.  Updated Vista PC Gemini to experimental ntpd from Dave Hart.  Left the -M parameter present for the moment.  Although stable to start with, this version eventually had no better stability than the "vegas" release.
• 1000 UTC, Tuesday, 2009 Jan 13, week 02.  Updated all Windows PCs to Meinberg's "vegas-v2" release (ntp 4.2.4p6).
• 1150 UTC, Sunday, 2009 Jan 11, week 02.  Updated Narvik and Gemini to Meinberg's "vegas" release (ntp 4.2.4p6).  Security update only.
• 1000 UTC, Friday, 2008 Nov 14, week 46.  PC Feenix replaced PC Hermes as the main EUMETCast receiver.  For continuity's sake, I carried over the old MRTG data from Hermes.
• 0820 UTC, Sunday, 2008 Sep 28, week 40.  Just for interest, tried setting the priority of NTPD.exe to "real-time" on the Vista PC, Gemini.  Whilst I don't expect it to make any difference, it will be interesting to observe any changes or side-effects (e.g. on the satellite data capture software)!
• 0950 UTC, Thursday, 2008 Sep 25, week 39.  At the suggestion of Mike Hughes, I tried setting CPU affinity for the NTPD process on the Vista PC, Gemini, to just a single CPU.  I happened to choose CPU 1.  It seems to have helped Mike get better timekeeping, so let's see if it helps me.  Mike may force CPU on the ntpd.exe with the ImgeCfg.exe program, if the improvement continues.
  Didn't seem to help, so tried stopping ntp service, marking the image uni-processor only, and CPU-affinity 0x1, and restarted.
  See: http://www16.brinkster.com/salvage/thief/darkengine.htm
• 1230 UTC, Tuesday, 2008 Sep 02, week 36.  Updated Hermes, Narvik and Stamsund to Meinberg build "ntpd 4.2.4p5@beijing-o Sep 01".
• 0940 UTC, Tuesday, 2008 Sep 02, week 36.  Turned of the -M flag for NTP on the Vista system (Gemini).
• 1430 UTC, Monday, 2008 Sep 01, week 36.  Updated NTP on Bacchus and Hydra to Meinberg build "ntpd 4.2.4p5@beijing-o Sep 01".  1525 updated Vista PC Gemini to "beijing" build..
• 2110 UTC, Thursday, 2008 Aug 07, week 32.  EUMETCast uplink glitch caused timing glitch on Stamsund.
• 0255 UTC, Monday, 2008 Aug 04, week 32.  EUMETCast uplink glitch caused timing glitch on Stamsund, and about an hour later on Hermes.
• 1750 UTC, Saturday, 2008 Jul 19, week 29.  The uplink of the EUMETCast service suffered a brief rain outage, and whilst this didn't trigger the automatic restart scripts, it did trigger a timing glitch on two out of three PCs (Hermes and Stamsund).
• 0545 UTC, Thursday, 2008 Jun 19, week 25.  The Vista system downloaded service pack 1 yesterday, and invited me to install it this morning.  So far, NTP seems as good now on Vista as it was on XP, but that's only a three hour spell.  It didn't last, though, and NTP is now as bad as ever on Vista.
• 0756 UTC, Wednesday, 2008 Jun 18, week 25.  On Vista, set system environment variable RANDFILE=C:\Tools\NTP\etc\.rnd, so that NTP might be able to access the file.  Used this directory as it already exists on my system, and it's where I write the NTP statistics.  This is in the hope that NTP time keeping on Vista might improve, but I am not optimistic about this!  No significant change noted.
• 0530 UTC, Tuesday, 2008 May 27, week 22.  Discovered that the W32Time service was still running on Gemini, despite me asking the Meinberg installer to disable it.  Probably needed more privilege. Disabled manually, and lets see how it goes.....
• 1330 UTC, Thursday, 2008 Apr 03, week 14.  Changed the LED in the Pixie GPS interface, and it will take a little more current (around 2mA), but be a lot brighter.  Seems to have caused some oscillation in the timekeeping, but settled down OK after a few hours.
• 1730 UTC, Tuesday, 2008 Feb 26, week 9.  EUMETCast signal loss and NTP restarts will cause transients on PCs Hermes, Hydra and Stamsund.
• Thursday, 2007 Dec 20, week 51.  Only discovered on Sunday Dec 23 that the PSU fan on Bacchus had stopped, so PC replaced with a spare one on Monday, Dec 24.
• 2200 UTC, Tuesday, 2007 Nov 13, week 46.  Restart Gemini with Windows Vista.  Some issues with Vista, currently under investigation.
• 1725-1737 and 2012-2034 UTC, Tuesday, 2007 Nov 13, week 46. power outage.
• 2345 UTC, Monday, 2007 Oct 22, week 43.  Break in EUMETCast from 2345-2359, and 0029-0042 (Tuesday 23).  On Hermes and Hydra, the timekeeping suffered a glitch.  Stamsund OK.
• 0020 UTC, Monday, 2007 Oct 01, week 40.  Seems to have been some sort of transient causing Bacchus to go wild, leaving a large positive value (494.739) in the drift file.  Restarted NTP OK at 0525 with a drift value which was approximately correct (-72.717).  Gemini and Hermes also showed a small glitch around that time, and Hydra nearer 0100.  Unsure what this was.  Spurious leap-second flag on Bacchus?  Yes, it inserted a leap-second at 0100 clock.  None of the currently selected servers show a +1 second offset.
• 0520 UTC, Tuesday, 2007 Aug 21, week 34.  Rebooted Bacchus, which seemed to cause quite a transient!
• 1845 UTC, Sunday, 2007 Aug 19, week 33.  EUMETCast drop-out triggered NTP transients on Hermes and Hydra.  PC Stamsund, with its older V2.3 card, had a TelliCast restart and therefore an NTP restart.
• 0600 UTC, Sunday, 2007 Aug 12, week 32.  Discovered that NTP had fallen over on Bacchus with an access violation.  Message logged at 02:37 on Aug 09 "no servers reachable".  Restarted OK but with a visible transient.
• 1510 UTC, Tuesday Jul 31, week 31.  Updated NTP on the Windows 2000 system to "ntpd 4.2.4p3@1.1502-foehr-o".  A new installer has resolved the LIBEAY32.dll error, and there were other problems due to having an earlier installation living in the \WinNT\ directory.  Resolved by not doing a files-only upgrade, and editing the ntp.conf appropriately.
• 0820 UTC, Friday Jul 27, week 30.  Updated NTP on the XP machines to "ntpd 4.2.4p3@1.1502-foehr-o".  The Windows 2000 system failed with "The ordinal 3837 could not be located in the dynamic link library LIBEAY32.dll".  No upgrade attempted on the one Windows NT4 system.
• 1146-1149 UTC, Friday, Jul 20, week 29,  EUMETCast drop-out triggered NTP transients on Hermes and Hydra.
• 0400 - 0430 UTC, Friday Jun 29, week 26.  Upgraded PCs Hydra, Narvik & Stamsund to Meinberg NTP V1.1499 (V4.2.4p3-RC1).  This does not appear to work correctly on PC Bacchus which runs Windows NT4 SP6, and hence the glitch while reverting to the older Terje Matheson version.
• 0546 UTC, Thursday Jun 28, week 26.  Upgraded PC Gemini to Meinberg NTP V1.1499 (V4.2.4p3-RC1) as a test. Seems to be OK.
• 0317-0325 UTC and 1141 - 1146 UTC, Thursday, Jun 21, week 25.  EUMETCast outages caused timekeeping glitches on Hermes and Hydra.
• 1700-1730 UTC, Sunday Jun 10, week 23, EUMETCast outages caused timekeeping glitches on Hermes and Hydra.
• 1330 UTC, Saturday Jun 09, week 23, EUMETCast outage caused timekeeping glitches on Hermes and Hydra.
• 2100 UTC, Friday Jun 01, week 22.  Experiments on Gemini setting the time zone between UTC (Casablanca) and GMT/BST (Edinburgh) seem to have upset NTP, so restarted it from scratch at about 06:10UTC on Saturday, June 02.  Still not settled by 07:00 UTC, and the drift value showed 481, so created a small reset file to stop, set the correct drift value, and restart NTP.  Ran at 07:03 UTC.
• 0015 and 1926 UTC, Saturday May 26, week 21.  EUMETCast drop-out triggered auto-restart and hence timekeeping glitches.
• 2145 UTC, Monday, 2007 May 07, week 19.  EUMETCast drop-out triggered auto-restart and hence a timekeeping glitch on Hermes, Hydra but not Stamsund.
• 0530 UTC, Saturday, 2007 Mar 31, week 13.  Discovered that NTP had fallen over on Bacchus with an access violation.  Restarted OK but caused a transient lasting to ??.
• 1930 UTC, Sunday, 2007 Mar 18, week 11.  Discovered that NTP had fallen over on Bacchus with an access violation.  Restarted OK but caused a transient lasting to 0300 Monday.
• 0807 UTC, Monday, 2007 Mar 05, week 10.  Discovered that NTP had fallen over on Bacchus with an access violation.  Only logged event was Mar 02 at 1825 "no servers reachable".  Restarted OK but caused a transient.  Seems same as Jan 05 event?
• 0820 UTC, Thursday, 2007 Feb 01, week 5.  There was another glitch which caused a timekeeping step on the Test PC, and required an NTP restart on the main PC.
• 07:50 UTC, Thursday, 2007 Feb 01, week 5.  Test PC (Hydra) lost EUMETCast connection for more than 90 seconds, the automatic restart causing a slight glitch.
• 0424 UTC, Friday, 2007 Jan 19, week 3.  EUMETCast drop-out triggered restart event on Hermes.  Hydra timekeeping affected.
• 1750 - 1820 UTC, Thursday, 2007 Jan 18, week 3.  EUMETCast drop-out on all three PCs affected timekeeping.
• 0945 UTC, Thursday, 2007 Jan 11, week 2.  Rebooted Bacchus after monitor change caused the system to hang (may have disturbed mouse connection at the same time).  Bacchus seems to have ignored the ntp.drift file, or somehow entered a wrong value (-20) there.  Reset to near-correct value of -72 and restarted at 1415 UTC.
• 0923 UTC, Friday, 2007 Jan 05, week 1.  Discovered that NTP had fallen over on Bacchus with an access violation.  Only logged event was Jan 02 at 1733 "no servers reachable".  Restarted OK but caused a transient.
• ~2200 UTC, Friday, 2006 Dec 22, week 51.  Added PC Narvik as replacement for Gemini.
• 0700 UTC, Sunday, 2006 Dec 17, week 50.  PC Gemini rebooted after monthly security update and Zone Alarm update.
• 0200 UTC, Friday, 2006 Dec 15, week 50.  EUMETCast seemed to drop out on Hermes and Hydra, but OK on Stamsund (older V2.3 card).  NTP consequently affected.  Two more drops at 2047 and 2153, handled by automatic restart on Hydra.
• 0200 UTC, Wednesday, 2006 Dec 13, week 50.  EUMETCast seemed to drop out on Hermes and Hydra, but OK on Stamsund (older V2.3 card).  Hydra also showed a drop from 1000 to 1100 (from about 9dB to 7.5dB).  NTP also affected.
• ~2230 UTC, 2006 Nov 29, Wednesday, week 48.  Glitch on EUMETCast caused PC Hermes and Hydra to stop collecting, so I restarted NTP as a precaution.
• 0155 UTC,2006 Nov 27, Monday, week 48.  Glitch on EUMETCast stopped Hydra and caused timekeeping transient.  NTP restarted at 1850.
• 0130 UTC, 2006 Nov19, Sunday, week 46.  Timekeeping glitch on Hydra suggests EUMETCast transient.
• 1440 UTC, 2006 Nov 17, Friday, week 46.  Stamsund rebooted after security updates.
• 1625 UTC, 2006 Nov 15, Wednesday, week 46.  Noted that Bacchus timekeeping had been a straight line so looked rather good.  Further checks showed that NTP had crashed with a memory access violation, with "no servers reachable" in the event log at 20:04:55 on Sunday Nov 12.  Restarted.
• 0140 UTC, 2006 Nov 02, Thursday, week 44.  Seemed to be a glitch on both Hydra and Hermes, so I suspect a EUMETCast glitch.  Another smaller glitch at 0605 UTC.
• 1500, 2006 Oct 18, Wednesday, week 42.  Rebooted Gemini after hardware hang.
• 1500, 2006 Oct 12, Thursday, week 41.  Large tree at the house front cut down, which should have the incidental effect of improving the GPS coverage slightly.
• 0910, 2006 Oct 10, Tuesday, week 41.  Stamsund spontaneously reboot during a period of high network activity, and I took the opportunity to update the Zone Alarm.  Hence the glitch.
• 0843, 2006 Oct 04, Wednesday, week 40.  Rebooted Bacchus, because the CMOS clock will likely have been well of when the system restarted, so making the 16-bit sub-system time incorrect..
• 0500 - 0630, 2006 Oct 04, Wednesday, week 40.  Power failure will have upset timekeeping.
• 1030, 2006 Sep 22, Friday, week 38.  Repositioned GPS antenna for Pixie onto the roof.  This should give it a good vertical view and remove the 180 degrees restriction due to the house, but may restrict the view near to the horizon.
• 1140, 2006 Sep 21, Thursday, week 38.  Due to building maintenance, temporary loss of EUMETCast signal caused Hermes NTP to go wild.  I wonder if the TCP/IP stack is getting overloaded by repeated "where-is-the-signal" activity and delaying NTP requests?  Restarted NTP on Hermes around 1200.
• 0715, 2006 Aug 26, Saturday, week 34.  Rebooted Hydra to correct EUMETCast signal strength reporting.  0730 rebooted Stamsund for the same reason (this made no difference).
• 0900, 2006 Aug 25, Friday, week 34. Loss of EUMETCast signal caused Hermes and Stamsund to loose proper timekeeping (they were increasingly slow while the red satellite icon was displayed), so restarted NTP.  Both Hydra and Stamsund were rebooted after security updates were applied.  NTP again failed to pick up the UK pool servers on the rebooted PCs, so added one other external stratum 2 server into the configuration.
• 0245, 2006 Aug 18, Friday, week 33.  Loss of EUMETCast signal (or high error rate on the Usingen upload), caused NTP on Hermes and Hydra to go wild.  Restarted with a near-correct drift file.  Stamsund appears to have survived OK.
• 1130, 2006 Aug 14, Monday, week 33.  Restarted NTP on Hydra to enable statistics collection.
• 0030 - 0300, 2006 Aug 13, Sunday, week 32.  Local power cut, so all PCs restarted manually (except the older Pixie and Bacchus which automatically restarted).  Bacchus seems to have -10 instead of -75 for drift, so copied the more correct drift file and restarted NTP.
• 1600, 2006 Aug 08, Tuesday, week 32.  GPS source lost sync altogether and had to be rebooted (switched off and on again).  No obvious reason, unless it's too few satellites.  Only noticed this at 0820 on Wednesday 9th!
• 1000 - 1900, 2006 Aug 07, Monday, week 32.  PC Stamsund rebuilt wit Windows XP Pro.
• 1855 - 1940, 2006 Aug 03, Thursday, week 31.  PC Gemini froze and had to be manually rebooted.
• 1456 - 1522, 2006 Aug 03, Thursday, week 31.  PC Stamsund was down for maintenance.  Memory upgraded from 1 to 3GB, and thoroughly cleaned!
• 1100, 2006 Aug 03, Thursday, week 31.  Discovered that Hydra's NTP had stopped, because Hydra was not connected to the network when booted.  Restarted NTP.  It would be helpful if NTP attempted to repair connections itself rather than just giving up. 
• ~0900, 2006 July 17, Monday, week 29.  Gemini rebooted after a hang, and then security upgrades.
• 1903, 2006 July 15, Saturday, week 28.  Discovered that NTP had stopped on Bacchus, perhaps at 0748 on Friday, with a "No servers reachable" event log message.  Restarted, and the initial offset was +74msec.
• 2000, 2006 July 10, Monday, week 28.  GPS source lost sync, and the PPS signal did not re-appear.  Power cycled at 0500 Tuesday.  What is happening here?
• 1830, 2006 Jul 08, Saturday, week 27.  Pixie seemed to loose sync with its GPS/PPS source.  Could have been 2030.  Sunday 0710 discovered this.  No flashing PPS pulses from the GPS, so powered down GPS and restarted it.  PPS pulses OK within less than a minute, and Pixie regained sync.
• 1640, 2006 Jul 05, Wednesday, week 27.  A thunderstorm with lightning (helped by the rain?) may have stopped the GPS 18 LVC working.  Power off and on restored operation, although it took a few minutes after power on before the PPS pulses appeared.  EUMETCast loss of signal at the same time.  This seems to have started as a lack of satellite signals (?) around 1415 UTC, although the EUMETCast outage was between about 1500 and 1530 UTC.  Perhaps these events were not related?  The EUMETCast outage caused NTP to go wild on Hermes and Hydra, but I detected this (by ear) and manually restarted NTP on those systems.
• 0715, 2006 Jul 01, Saturday week 26.  Timekeeping on both Bacchus and Stamsund had gone wild since about midnight UTC, with incorrect very large drift values (over 400) being recorded in ntp.drift.  Both these PCs are 100% reliable.  Reset drift file to correct content and restarted ntp.  This is just like what happened at the new year, with the leap second.  Could it be that some of the Internet servers these PCs were seeing were still sending the leap second flag?  See: ntp-events.
• 1900, 2006 Jun 25, Sunday, week 25.  Restarted NTP on Hermes as the DVB reception system restarted (due to satellite signal loss - bad weather over Germany), and the automatic restart caused a transient.
• 1300, 2006 Jun 18, Sunday, week 24.  Restarted NTP on Hermes with -94.557 as the drift value.
• ~1100, 2006 Jun 17, Saturday, week 23.  Transient on Hermes due to signal loss and re-acquisition on DVB system.  Although signal was back OK, the NTP software seemed to "go wild".
• 2006 Jun 11, Sunday, week 23.  Some sort of transient or issue on Pixie seemed to stop its precision lock for a period, but it recovered.
• 1146, 2006 Jun 07, Wednesday, week 23.  Checked that the GPS 18 LVC really was in 2D, and it was.  No improvement seen, so probably the glitches are not loss of satellites.  Ah, well!
• 0955, 2006 Jun 07, Wednesday, week 23.  Reboot Gemini to get network cards back to standard IDs for MRTG.
• 0810, 2006 Jun 06, Tuesday, week 23.  Attempted to set GPS 18 LVC to 2D mode, so that fewer satellites are required to get a time "fix", but I'm not sure if this worked.
• 0750, 2006 May 31, Wednesday, week 22.  Gemini rebooted after a security update.
• 2006 May 19 - 22, week 20-21.  Gemini hang followed by a reboot late Monday 22.
• 0700, 2006 May 12, Friday, week 19.  Stamsund rebooted after security update.
• 1645, 2006 May 11, Thursday, week 19.  Hermes rebooted after update of TelliCast client software.
• 1530, 2006 May 09, Tuesday, week 19.  Found that the ntp.drift file on Gemini contained 9 0x00 bytes, and that ntp.drift.TEMP had the correct value.  Could not delete ntp.drift, and the permissions were correct.  Stopped NTP, and copied ntp.drift.TEMP to ntp.drift.  Restarted NTP OK.
• 0000 (clock) 2006 May 06, Saturday, week 18.  Hermes had about a +80ms time step, and gradually recovered back to zero.  No obvious cause.  There was a EUMETCast issue at almost exactly the same time - a co-incidence?  Had been up for about 80 days 1h 45m, and is still running without a reboot.
• 1120, 2006 Apr 09, Sunday, week 14.  Noted that Gemini USB WiFi hadn't restarted after the reboot, so unplugged and re-plugged the adaptor.  This seemed to kill NTP, and it was manually restarted at 1435 when it showed a 30ms offset.
• 1530, 2006 Apr 05, Wednesday, week 14, took Gemini down for about an hour to add a fan to the hard disk bays.  Will be interesting to see any effect or not on the diurnal offset.  Disk temperatures about 17°C lower.
• 1600, 2006 Feb 10, Saturday, week 6, removed Odin after 4 years service, it will be replaced by Gemini.
• 1110, 2006 Feb 01, Wednesday, week 5.  Rebooted Pixie following a request for a test.
• 1400, 2006 Jan 30, Monday, week 5.  Made all four PCs sync to Pixie as a preferred server.  Slight glitch resulted.
• 1100, 2006 Jan 30, Monday, week 5.  Finally got kernel recompiled on FreeBSD PC (pixie - it took 5 hours!) and enabled the PPS stuff.  Will keep an eye on this before changing the other servers.
• 0730, 2006 Jan 30, Monday, week 5.  Some sort of glitch on Hermes seeming to stop DVB reception.  DVB restored at 0745, but there was a timekeeping transient just after 0800 with a step of approximately 75ms.  Why?  There is nothing in the event log.
• 1700, 2006 Jan 29, Sunday, week 4.  Add new FreeBSD system with GPS NMEA - Pixie.  Discovered that you need to re-compile the FreeBSD Kernel to enable PPS support, so this is in progress.
• 1600, 2006 Jan 28, Saturday, week 4.  Add GPS NMEA-only reference clock to Odin for testing.  Looks like the 4800 baud serial output is about 200msec earlier than the (unused) PPS signal, but this can be fudged.  Jitter is bad (up to 10ms), so it will be interesting to see how this shows on the plots.  Lots of errors "clock GPS_NMEA(1) event 'clk_fault' (0x03)" so I've abandoned this at 1715.
• 0800, 2006 Jan 14, Saturday, week 2.  Daily drift on Stamsund and Odin is more than I like, so reset the maxpoll to the local servers to 9 (512 seconds).
• 1100 - 1500, 2006 Jan 13, Friday, week 2.  Odin down and rebooted following peculiar hang.
• 2030, 2006 Jan 11, Wednesday, week 2.  Removed all MaxPoll lines from Odin and Stamsund to see how they get on in a "default" configuration.  This should make all servers MaxPoll 1024 seconds.
• 1000, 2006 Jan 11, Wednesday, week 2.  Removed the "prefer hermes" line from Odin and Stamsund in case yesterday's events should happen again.  At least with Bacchus remaining stable, perhaps these clients would not have behaved as badly.
• 2100, 2006 Jan 10, Tuesday, week 2.  For some reason, Hermes NTP went wild.  About 1 hour later, Odin started going wild, and nearly three hours later Stamsund joined the wild bunch!  PC Bacchus showed no problems, perhaps showing the Internet connection stayed OK (the traffic looked normal).  PCs Odin and Stamsund only showed changes of selected server. At 2119 Hermes changed to server 130.88.200.6 (utserv.mcc.ac.uk) and suffered an immediate time reset of +0.194392s.  It continued seeing time resets until 06:43:15 on Jan 11, and seems to have been able to recover without further resets since then.  There's nothing in the event log to suggest any other problems around the time when NTP on Hermes went wild, however, at 2105 there was a burst of wind which caused momentary loss of a satellite signal.  Perhaps a mains glitch?
• 1350, 2006 Jan 09, Monday, week 2.  Rebooted Hermes after addition of RAMdisk and Windows security updates.
• 0910, 2006 Jan 06, Friday, week 1.  Rebooted Odin after official WMF security update.
• 1000, 2006 Jan 04, Wednesday, week 1.  Rebooted Odin and Stamsund following WMF security patch update.
• 1500, 2006 Jan 02, Monday, week 1.  Decided to replace older NTP software on Hermes and Bacchus by the more leap-second complaint version.  Perhaps should have done this before the leap-second rather than after!
• 0740, 2006 Jan 01, Sunday, week 1.  Hermes and Odin had not settled down after the leap second, and the drift files values were very big (over 400), so stopped ntpd on those systems, deleted the drift files, and restarted.  Ntpd seems to have some instability judging by the offset graphs. Bacchus (drift -55.3) and Stamsund (drift -11.5) did seem to be sorting themselves out, albeit slowly.  See: ntp-events.
• 2010, 2005 Dec 26, Monday, week 52.  Advised that having a large difference between MaxPoll and MinPoll was not a good idea.  "You have forced the poll interval and time constant to 64 s.  Should that source fail and the host switch to another source with poll interval clamped to 1024 s, the discipline loop is seriously undersampled and could well become unstable.", so reset Odin and Stamsund to MaxPoll=8 as a compromise.
• 0735, 2005 Dec 24, Saturday, week 51.  Updated Odin and Stamsund to have MaxPoll 6 on internal LAN servers and MinPoll 10 for their Internet servers.  This provides a fast response while minimising the load on the backup servers.
• 1530, 2005 Dec 22, Thursday, week 51.  Updated Odin and Stamsund Meinberg ntp-4.2.0b@1.1436mbg-xmas version.  This accepts the -M parameter to enable the MM timer while ntpd is running, so I could now disable this frig in my own program on Odin.  SSL library updated on Odin.  [Didn't use the Meinberg installer].
• 1125, 2005 Dec 20, Tuesday, week 51.  Changed Odin and Stamsund from 128s poll to default 1024s poll.  Since the steps are gone, there is no need for the shorter poll interval.  Noted that Bacchus seems to have rather more drift than Hermes, even though both are nominally synced the same.
• 0639, 2005 Dec 13, Tuesday, week 50.  Restarted NTP on Stamsund to collect loop statistics.  Noted that this caused an initial 20ms offset.
• 1855, 2005 Dec 12, Monday, week 50.  Installed experimental NTP version on Stamsund.  Version: ntpd 4.2.0b@1.1432-o Dec 12 17:03:03 (UTC+01:00) 2005 (43) from Meinberg.  Needed to edit the Image path in the registry to include a "-M" parameter to force the MM timer to 1ms resolution.  Having this parameter present caused previous NTP versions to fail.  Needed to update the SSL libraries as well.
• 0910, 2005 Dec 09, Friday, week 49.  Tried the same "force MM timer to 1ms" experiment on Stamsund, as a test.
• 1700, 2005 Dec 07, Wednesday, week 49.  Started running QuickTime player permanently on Odin to force the MM timer to run with a 1ms period.  Seems to have stopped the violent swings.  Looks as if the MM timer was permanently running on Odin until Friday, Dec 05.
• 1030, 2005 Dec 06, Tuesday, week 49, updated all systems to V4.2.0b from Terje Matheson ("ntpd 4.2.0b@1.1431-o Dec 06 10:23:21 (UTC+01:00) 2005 (1)")
• 0941, 2005 Dec 05, Monday, week 49, replaced older 2003 V4.2.0 ntpd on Odin by V4.2.0a from Meinberg 1370 download.  The jitter still seems high.  Version numbers were:  old: ntpd 4.2.0 fr Oct 17 8:49:33 2003 (1), new: ntpd 4.2.0a@1.1370-o Apr 12 14:10:45 (UTC+02:00) 2005 (12).  Also, at 1310 replaced Stamsund's ntpd.exe with the Meinberg 1370, as a test to see if the extreme swings were any different.
• 1500, 2005 Dec 02, Friday, week 48, something has upset the otherwise excellent timekeeping on Odin.  Tried to get rid of this on Saturday by reboots at 1100, 1800 and 2100.  Can't discover what the problem is.  See: ntp-events.
• 1830, 2005 Oct 27, Thursday, week 43, reverted to maxpoll 10 on Bacchus as no permanent change noted.
• 0800, 2005 Oct 04, Tuesday, week 40, changed maxpoll interval on Bacchus from 10 down to 9 to reduce diurnal variation.
• 0820, 2005 Sep 09, Friday, week 36, found that local ISP head-end was no longer offering NTP services so switched to a different external server.
• 2230, 2005 Sep 06, Tuesday, week 36, power glitch takes down systems from about 1600 to 2230
• 0830, 2005 Aug 16, Tuesday, week 33, was moved by ISP onto a different upstream connection to the cable modem service.  Watch and see if there is any improvement.  Yes (two days later), it's much better.  Thanks for an understanding ISP!
• 1510, 2005 Aug 11, Thursday, week 32, noted more noise over the last couple of weeks or so, so added another two NTP pool servers to Hermes as a test.
• 1510, 2005 May 28. Saturday, week 21, remove NPL servers from Bacchus - too noisy because too many servers?
• 1830, 2005 May 27, Friday, week 21, remove NPL servers from Hermes - too noisy?
• 2205, 2005 May 25, Wednesday, week 21, added NPL servers to Hermes & Bacchus.
• 0000, 2005 Mar 25, Friday, week 12, Blueyonder network maintenance causes glitch.
• 0930, 2005 Feb 04, Friday, week 05, moved Bacchus and Hermes onto NTP Pool Internet servers.
• 2230, 2005 Feb 03, Thursday, week 05, a Blueyonder network problem in Scotland causes another glitch.
• 1200, 2005 Jan24, Monday, week 04, reboot of Odin following internal problem.
• 2300, 2005 Jan 15, Saturday week 02, a Blueyonder network problem in Scotland causes another glitch.
• 1900, 2005 Jan 03, Tuesday week 1, some sort of disruption of level 3 links at Blueyonder cause transient glitch in timekeeping.
• 1945, 2004 Nov 16, Tuesday week 46, add 'prefer' option to Hermes entry in Odin & Stamsund's configuration
• 1200, 2004 Nov 16, Tuesday week 46, major configuration change, Bacchus & Hermes sync to the Internet, Odin and Stamsund sync to Bacchus and Hermes.  This change because Hermes is really an always-on and lightly loaded server.
• 0930, 2004 Nov 10, Wednesday week 45, made Stamsund not have Tinker Step 0.050 to see if the extreme swings were handled any better.  Not sure what is causing the swings.
• 2004 Nov 06, Saturday week 44, multiple reboots of Stamsund to update DVB card drivers
• 2004 Nov 05, Friday week 44, multiple reboots of Hermes to update DVB card drivers
• 1830, 2004 Sep 27, Sun week 39, power glitch required reboots of all PCs
• 2004 Sep 15, Wed week 37, lost Internet connection for 4.5 hours
• 1400, 2004 Mar 08, Mon week 9, tested one PC as an Internet server.  As the bandwidth throttling appears not to work, it eats all the upstream bandwidth and therefore clobbers timekeeping.
• 0800, 2004 Mar 05, noted that on Odin & Hermes there were some 10ms steps, and that the Internet servers were 1024s maxpoll (10).  Altered the configuration to make them maxpoll 7 and restarted.
• 2030, 2004 Mar 04, increased maxpoll to 7 (from 6) on Odin & Hermes.  Hermes & Odin not seeing the external servers, although they had 6 hours and 42? hours ago!
• 1200, 2004 Feb 26, changed external server, rebooted Odin & Stamsund.  Hermes & Stamsund not seeing the external server.  Why not?
• 1705, 2004 Feb 24, removed Stamsund as peer for Hermes and Odin.  Hermes still not seeing the external server.
• 0037, 2004 Feb 21, Sat, changed the external network server.  Hermes still not seeing the external server, Odin got a 30ms step during service restart sequence.
• 1638, 2004 Feb 14, Sat, as an experiment, added one more external network server to Hermes, Odin and Stamsund.  Note that Hermes is in the state of not talking to external servers.
• 1230, 2004 Feb 03, Tue, a power cut at 1150 resulted in all PCs being rebooted at 1235.
• 1145, 2003 Dec 05, Fri, a reboot of Odin again gave a large step and slow recovery cf. mid-November.
• 1205, 2003 Nov 18, Tue, added tinker step 0.050 to try and avoid these large steps on reboot (Hermes, Odin, Stamsund).
• 0915, 2003 Nov 16, Sun, Bacchus performance not so good, nor the rest of the network, so reverted to no "prefer".
• 1740, 2003 Nov 15, Sat, gave Bacchus a "prefer" server as it was clock-hopping.
• 1415, 2003 Nov 09, Sun, restored Hermes/Odin/Stamsund peering, in view of the number of reboots that Hermes is requiring.
• 1900, 2003 Nov 08, Sat, replaced Hermes with a different PC (actually old Wotan).
• 1000, 2003 Nov 07, Fri,  Hermes rebooted at 1000, DVB card added to Stamsund.
• 2003 Nov 04, for operational reasons, Hermes required a number of reboots, and seems to have lost accurate drift information.....
• 1745, 2003 Nov 01, Sat, noticed that Stamsund is noisy (doing large panoramas!), so remove it from peering with Hermes and Odin.
• 1000, 2003 Oct 31, Fri, added one more server to Bacchus.
• 2115, 2003 Oct 30, Thu, noticed that Bacchus was more noisy, so tried a "burst" option on one server.
• 1834, 2003 Oct 28, Tue, PC Wotan replaced by PC Stamsund.
• 0815, 2003 Oct 26, Sun, reset Bacchus to 1024s Internet poll, Hermes, Odin and Wotan to 64s poll.  So Bacchus may get a bigger offset, but should be lower jitter...
• 1700, 2003 Oct 25, Sat, left just Bacchus connected to the Internet servers, with Hermes, Odin and Wotan peering with each other and being served from Bacchus with one Internet as backup.
• 1050, 2003 Oct 25, Sat, made least-used interactive PCs the main peered Internet connected ones (Hermes and Bacchus) and the most heavily used PCs (Odin and Wotan) the LAN served peers.  Resulted in more drift than I liked on Hermes.
• 0915, 2003 Oct 25, Sat, changed LAN PCs Odin and Hermes to 128 second poll
• 1000, 2003 Oct 18, Sat, removed ntp.blueyonder.co.uk from Odin and Hermes as it seemed to pull Hermes during the evening and night, and Odin wouldn't talk to it in any case (permanently in INIT).
• 1315, 2003 Oct 17, Fri, updated to NTP release V4.2.0, added ntp.blueyonder.co.uk to Odin and Hermes as it has now had the large offset removed
• 0830, 2003 Oct 04, Sat, set Internet facing servers to 512s poll to reduce Bacchus offset
• 1530, 2003 Sep 05, Fri, restored Odin as peer for Hermes - graphs showed no change.
• 2100, 2003 Aug 21, Thu, removed Odin as peer for Hermes - likely to be unreliable due to animations.
• 1010, 2003 Jul 23, Wed, updated to V4.1.80-rc1 from Terje Mathisen.
• 1330, 2003 Apr 24, added Odin/Hermes as peer pair, and Bacchus/Wotan likewise.
• 1700, 2003 Mar 20, Thu, installed DVB card in Hermes, timekeeping goes to pot!
• 1000, 2003 Mar 18, Tue, updated to V4.1.74 with the CMOS code present.
• 1550, 2003 Feb 18, Tue, updated all to V4.1.74.
• 1615, 2003 Feb 07, Fri, revert to 1024s poll on Internet facing servers for a test.
• 0840, 2003 Jan 29, Wed, V4.1.73 (dev) installed on PCs Odin and Hermes, 0945 installed on Bacchus and Wotan.
• 0845, 2002 Dec 19, Thu, Bacchus and Wotan changed down to 512s maximum polling (to reduce Bacchus' drift).
• 1000, 20002 Dec 17, Tue, Bacchus and Wotan synced to Internet servers, PC Hermes  and Odin synced to Bacchus/Wotan with 64s maximum polling.
• 2340, 2002 Dec 12, power cut, all PCs rebooted.  Power back at 0020 Dec 13
• 1345, 2002 Nov 08, Fri, all PCs synced to Internet servers.
• 1100, 2002 Nov 07, Thu, Bacchus and Wotan synced to Internet servers, and PC Hermes and Odin synced to Bacchus/Wotan instead of Bacchus alone.
• 1445, 2002 Nov 06, Wed, only Bacchus was left synced to Internet servers, and PC Hermes, Odin and Wotan were synched to Bacchus instead of Internet NTP servers.

NTP Notes

For my own system

• Building a precise time source using FreeBSD and a GPS receiver
• Using Windows with a GPS/PPS source
• Using NTP on Windows Vista
• Using NTP in a cable-modem environment
• NTP 4.2.4 compared with NTP 4.2.5
• Mechanising NTP performance recording with MRTG
• My Windows multi-PC NTP monitoring program
• My Windows loopstats plotter
• My daily network traffic analysis, an example of using MRTG
• Some events in more detail
  • Leap-second December 2008
  • Using NTP on Windows Vista, 2007-2009
  • False leap-second July 2006
  • Leap second at the end of 2005
  • MMtimer issues uncovered December 2005
• Loopstats data - only updated from time to time
  • PC Bacchus
  • PC Feenix
  • PC Gemini
  • PC Hydra
  • PC Narvik
  • PC Stamsund

External links

• Official NTP documentation
• Where to find NTP software
• Meinberg Windows NTP package download (includes Windows installer which can be automated for hands-off install)
• Meinberg free NTP Time Server Monitor (und auf Deutsch)
• The NTP Windows port from Terje Mathisen which I have used in the past

Hardware suppliers - in alphabetical order - vendor-supplied descriptions

• Galleon Systems - Providers of NTP Servers for Time Synchronisation of computer networks
• Meinberg - NTP Time Servers and innovative solutions for Time and Frequency Synchronization using GPS, DCF77 and IRIG
• Spectracom - NetClock GPS time servers for vital communications networks
• TimeTools - TimeTools provides accurate MSF DCF-77 and GPS NTP Time Servers for accurate network timing.