Windows Serial Port GPS/PPS reference clock for NTP
I took renewed interest in using a Windows box instead of my FreeBSD box as a local stratum 1 reference clock when I became aware of some work done by Dave Hart. The Windows port of NTP has long supported using an RS-232 GPS device with just a NMEA serial data, but the software has not supported a PPS (pulse per second) feed to the CD (carrier detect) pin. The old timeout-based serial data handling gave an unreasonably large jitter, meaning that a local serial clock could be less accurate than an Internet source! Windows has had the problem of poor timestamp resolution preventing its use as a highly accurate reference clock. However, Dave has done some work on the interpolation algorithm used, which has enabled much better timekeeping from Windows. Ntpd long ago overcame the OS clock precision using interpolation, Dave Hart's changes have been to improve that interpolation, most substantially removing about 1 ms systemic jitter.
He has also done work on an update to the serial device driver (serial.sys) with a view to making the driver provide the timestamps rather than the user program, which should enable even greater precision to be obtained. It's not clear to me at the moment how this driver will fit in with NTP - perhaps it will be supplied with the NTP distribution and will be used by the common NTP PPSAPI Atom driver?
Dave Hart writes: The official NTP distribution is source-only.
The patch or the entire source for serialpps.sys might become part of the distribution if Microsoft's DDK license allows, but the binary wouldn't be.
On the other hand, I wouldn't be surprised to see serialpps.sys included by Meinberg with their binary distribution, though the quality of installer
(i.e. Dave Hart's installer) is laughable by comparison, so they might opt to let people download it separately.
Either way, I would expect the PPSAPI implementation for serialpps.sys to be included in the ntp distribution sources and Meinberg binaries.
For the impatient - brief details
There is a step-by-step
guide here from Jeremy W. Langston who followed the guide on this
page. I would recommend that you consider using the latest
version available from Dave Hart rather than the specific version listed by
The hardware is essentially unchanged from my earlier FreeBSD stratum-1 clock, with a small box to interface a Garmin GPS 18 LVC puck to a serial port on a PC. I did build a second version of the hardware where I use the more recent GPS 18x LVC puck which is much more sensitive and allows indoor operation. This version used a spare USB port for the 5V power required by the puck. Please note some issues with the 18x firmware, which mean that you are only using the 18x for precise within-the-second sync, and using another source (e.g. Internet) for the nearest-second sync.
NTP can use a number of system environment variables to control its operation. For a typical end-user, client-only install of NTP, none of these settings will be required. For a stratum-1 NTP server install, you may only need the PPSAPI_DLLs variable, and even then only if you want the lower jitter of the kernel-mode PPS driver. Note: you will need to restart the NTP service for these changes to take effect.
Note from Dave Hart - How to update to local ref-clock
[With slight edits from DJT] Windows 2000 should work based on reports from testers (DJT - it does).
server 127.127.20.1 minpoll 4 prefer
If your GPS is sending $GPRMC each second at 4800 baud it should just work. If it's configured to send only a different NMEA sentence, we'll need a mode option on that server line. You could check Pixie's ntp.conf, if it's using a mode option on the server 127.127.20.x line copy it. [DJT: in my case, the GPS was already sending the correct $GPRMC sentences.]
For potentially greater precision, you can allow ntpd.exe to raise itself to the real-time priority level. You can do this in one of two ways. If security is not of the greatest concern, use method 2, otherwise use method 1. Dave Hart writes:
Look in Task Manager on the Processes tab for ntpd.exe. If you don't see it, make sure "Show processes for all users" is enabled. Show the priority by choosing View / Select columns... from the menu, and in that box turn on the check mark next to "Base priority". Click OK and you should see ntpd.exe at Realtime base priority. If it's High, the privilege wasn't available.
If you're running one of my test releases, a message is written to the event log or configured log file at startup indicating whether High or Realtime priority is being used.
Real-time priority in practice
Please look at the graphs below which show the value of jitter as reported by "ntpq -c rv" versus time. The priority of the ntpd.exe process was changed from high to real-time during the period of these graphs. Can you see when?
It was actually at 06:20 on Sunday, just to the left of centre in the top graph. It may have made a slight difference, but not a lot. On this system, at least, the offset is much greater than the jitter (possibly due to the relatively large temperature variations to which the system is subject). It's in a room in a domestic environment, and not in a temperature regulated computer room.
The gotcha is making sure you expect to lose the serial port on next reboot to the BallPoint Serial Mouse driver. Avoid a nasty surprise the next time Microsoft pushes out an off-cycle automatic update by planning to reboot, disable the BallPoint Serial Mouse device in Device Manager, and reboot again before too long after you move the GPS to the Windows box. But it doesn't have to be right away, it's a gamble you'll win most nights :)
[DJT] My Windows 2000 Server system also saw a Microsoft Serial Mouse, which I also had to disable and reboot yet again. Seeing the reach at 200 (meaning that just one value had been read from the serial port) was the clue. You disable the spurious mouse by going to Device Manager and selecting the Serial BallPoint Mouse. Right-click, and select Disable. Don't uninstall.
[DJT] I received an e-mail noting that Microsoft had a COMDisable tool available. See: KB819036.
As I don't have a second GPS receiver, I simply made up a lead with a parallel connection to the data and carrier detect signals from the GPS. It seems to work in my environment, with signal levels still around 4 volts even with one "RS-232" transmitter driving two "RS-232" receivers. I saw no detrimental effect to the accuracy of the existing FreeBSD system on PC Pixie. Of course, this is outside the "official" RS-232 standards, and for a proper installation multiple line drivers should be used. I later added the more sensitive GPS 18x LVC for testing, powered off the 5V USB port rather than a separate power supply.
In the early stages of testing, you can use my Serial Port LEDs monitor program to check that you are seeing the DCD pulse one per second. You need to stop NTP while using this program as access to the serial port cannot be shared. Please note: If you are using Windows for NTP with a PPS signal, the DCD line must flash briefly on, not be mostly on flashing briefly off as inverting the PPS signal is not supported by the Windows NTP port (i.e. fudge <ref-clock> flag2 1 isn't supported).
A new version of ntpd.exe is required to support GPS/PPS operation over a serial line with Windows. I installed this version (actually just overwriting the existing ntpd.exe file was sufficient in my case), and looking at the Event Log I found the following entry, which shows that the PPS line on the CD pin is being detected:
Using user-mode PPS timestamp
When I first changed to the new ref-clock I got an unexpectedly large amount of jitter, and a few 20ms steps in the offset. I had seen no steps when operating this PC just with Internet and LAN-based servers. Dave Hart noticed from the Event Log entry:
HZ 99.856 using 43 msec timer 23.256 Hz 64 deep
that I had a system with a 100Hz clock (quite unusual), and following a brief test, he provided me with an ntpd.exe which had a 26 millisecond sampling interval for his additions rather than the default 43ms. To check that the correct 26msec interval has been used, check for the line:
HZ 99.856 using 26 msec timer 38.462 Hz 64 deep
The initial results from this were much better, with jitter from ntpd -c rv being reported in the 200Ásec region, so I therefore tried his serialpps.sys driver which adds timestamps to the received data (better timestamps, timestamps for more events?). To know that the new driver is working, look for this line in the event log:
Using serialpps.sys timestamps
Please note that later versions of the drivers from Dave Hart change this, so that the serialpps.sys talks to the ATOM ref-clock driver, and not the GPS/NMEA ref-clock, which reverts to using user-mode timestamps. You need to run both ref-clocks to get the kernel-mode timestamps.
The results were not as good as Dave Hart had expected, although they were certainly better than just using Internet sources. There was an issue with this particular PC, as it was discovered to be using a 100Hz timing interrupt (10msec), and the code which Dave had written to enhance the interpolation does not work as well with 100Hz. We did try playing the value of the sampling, and found that 26 or 27 were better values for a 100Hz system. However, we could not get the low jitter values which Dave expected.
To set the sampling value, you can set the environment variable NTPD_INT_INT to the value 26, using the Control Panel, System, Advanced page. Be sure to set the System variables, and not the User variables!
One Windows 2000 issue which we encountered was that just using restart in the Services control application was that the environment variables weren't read afresh when a Restart operation was requested. To overcome this, I made a small batch file to run NTPD.exe interactively. For example:
net stop ntp SET NTPD_INT_INT=27 CD C:\Program Files\NTP\bin\ ntpd.exe -n -g -c "C:\WINNT\system32\drivers\etc\ntp.conf" -M
To stop ntpd, issue a Ctrl-C into the command window.
Results from different values of NTPD_INT_INT
Using the interactive start of ntpd.exe, I ran for a couple of minutes first (waiting for a 377 in the Reach field), then sampled the reported Jitter values from an "ntpq -p" at about half-minute intervals, with each INT value. Jitter values are in milliseconds.
We decided to leave further experiments on the 100Hz system, as I was considering scrapping that system in any case. I then moved the serial/CD line to another PC (Feenix) which has a 1.9GHz single-core processor (compared with 550MHz) and which runs Windows XP.
Sample NTP configuration file:
Results from the second system
It is likely that PC Feenix will replace PC Pixie as my main stratum-1 server, so I am interested to know how it compares with the existing stratum-1 PC, Pixie, which runs a homebuilt FreeBSD system. I wrote a simple data collection process in Perl which collects the output from an "ntpq -c rv" command, and filters the data of interest into a text file, which can hen be analysed offline with a program written in Delphi. The main criterion, for me, was how close to UTC the system would run - the offset field.
With the data plotted on the same scale, the Windows system appears a lot worse than the FreeBSD system, although it still offers performance typically well inside 200Ás, and any system syncing to Feenix will know about the offset and allow for that when setting its own time.
[Dave Hart comments: Actually no, ntpd tries to simply bring its time into alignment with the selected source (actually with the average of the survivors) but does not attempt to compensate for the current offsets the sources have to their sources.]
It's rather revealing, though, to take what is basically the same data and scale the axes differently, as I do on the MRTG plot below:
Although there is more relative variation on the FreeBSD system (the jitter is higher relative to the offset), the general shape of the offset plots is the same, being consistently lower early thing in the morning when the room temperature is at its lowest. Although it's difficult to judge, the Windows system appears to have some 20 - 50 times as great a variation. (This is running with the user-mode PPS, though). It's also interesting to look at the frequency error (in parts per million [ppm]) as reported by NTP. In the plot below, only the change in frequency offset from the mean value is shown, to allow for the fact that we may be looking for differences less than 1 ppm whereas the two systems could have a difference in mean error of several hundred ppm.
Here, the similarity shape of the two curves is even more striking, albeit that the daily deviation of the Dell 4400 (Feenix) is twice that of the older Viglen PC (Pixie). Two completely different PCs, so it's hardly surprising that they have two different temperature coefficients. The glitches on the green line (Feenix) are due to the NTP on that PC being stopped and restarted to change the ntpd.exe process priority. There seems to be slightly more "noise" on the FreeBSD system, but as that's using a version of NTP which is some four years older, it's not surprising that there may have been algorithm or accuracy improvements in the intervening time.
Here is another offset comparison showing March 20 and 21 of 2009. I notice that there are fewer transients in the Windows offset compared with the FreeBSD offset. Perhaps there were improvements in the GPS algorithm between the older 4.2.0 working on the FreeBSD system and the newer 4.2.4 on the Windows system?
Some current monitoring graphs
One option which has been investigated during these tests, but which has unfortunately not made it into either the release or development branches in the use of kernel-mode rather than user-mode for timestamping the transition of the PPS line. This has the unfortunate effect of making the jitter quite a lot worse, and subject to the effects of other activity on the system. However, the effects on frequency error and offset are not too great.
On 2009 June 05 Dave Hart wrote to me:
[After version 4.52.5p180] there's a new .dll there which ntpd currently finds by way of
environment variable PPSAPI_DLLS. If you put it in the same directory as ntpd.exe, this should work
Keeping it out of the ntpd.exe directory avoids having it stepped on by running the Meinberg installer.
He also wrote a fuller explanation on comp.protocols.time.ntp on 2009 Dec 01:
To have full PPSAPI capability with ntpd on Windows you need serialpps.sys, serialpps-ppsapi-provider.dll, and a recent-enough ntpd.exe. You need a serial port which can be driven by the stock serial.sys (which includes a huge variety including some simple multiport designs). serialpps.sys must be "installed" (if you can call it that, the file must be in place and pointed to by the serial.sys image path registry entry). serialpps-ppsapi-provider.dll must be accessible and pointed to by environment variable PPSAPI_DLLS visible to ntpd.exe (typically set system-wide), such as:
You can find both serialpps.sys and serialpps-ppsapi-provider.dll in:
There are more releases of serialpps .zip files in that directory than
there are different versions of serialpps.sys within. The most recent changes, adding serialpps-ppsapi-provider.dll, simply shuffled code
previously hard-wired into ntpd.exe off into a per-provider DLL, but did not change the ioctl interface or serialpps.sys.
Following a discussion on the comp.protocols.time.ntp
newsgroup, it became apparent that the current release of NTP (4.2.6) logs that
it is using user-mode timestamps, even though kernel-mode timestamps are
actually in use. This caused some head-scratching! It seems that
because one policy is minimal messages, and because of the architecture of the
Windows solution with a device driver and a DLL, and because of the timing of
the startup, it is not easy to log that the kernel-mode is in use.
C:> ntpq -p feenix remote refid st t when poll reach delay offset jitter ============================================================================== oPPS(1) .PPS. 0 l 15 16 377 0.000 0.087 0.003 *GPS_NMEA(1) .GPS. 0 l 14 16 377 0.000 0.052 0.002 +stamsund .PPS. 1 u 46 64 377 0.555 -0.038 1.203 -paddington.ragg 188.8.131.52 2 u 1008 1024 377 23.190 3.053 0.504 -dnscache-london 184.108.40.206 2 u 1047 1024 377 20.872 2.892 1.258 -utserv.mcc.ac.u 220.127.116.11 2 u 955 1024 377 28.712 4.269 1.949 +linnaeus.inf.ed 18.104.22.168 3 u 338 1024 377 35.373 1.102 1.066
Of course, your server names will be different, but the two
lines with the tally-codes (first column) * and o should both be
present. The * tally-code indicates the normal NTPD "I am
using this server" state, and in the above example it shows that it is
using the GPS_NMEA local reference clock, so the basic serial-port functionality
is present and correct. The o tally-code indicates that the
kernel-mode functionality has been detected by the Atom driver, and that it is
in use. The (1) indicates the COM port number being used for the
How well does it work?
Here is a graph of switching from a test release of NTP with the kernel-mode code to a development snapshot with user-mode, and therefore without the benefits of the ATOM driver.
Frequency variation range
Any side effects of using serialpps.sys?
Dave Hart writes: The effect (on CPU etc.) compared to using serial.sys is quite similar.
serialpps.sys adds a new "ioctl" function which software other than ntpd won't
know about or use. Until the first time that ioctl function is called by a
given process, serial.sys behaves identically. After the first call to that
ioctl function, the interrupt handler for "modem status" line changes (including DCD) squirrels away a system timestamp and cycle counter
value at each DCD clear->asserted transition, retrievable via the same ioctl
And that "User-mode" message?
Dave Hart writes: I should have said the fact that a message about user-mode PPS is logged does not imply PPSAPI is not available or even being used. You could see the user-mode PPS message (which is only seen with the Windows ntpd, for those who are baffled by this user-mode PPS discussion) during startup of NMEA configured to use PPSAPI directly (which implies not configuring PPS/atom as a separate source). The user-mode PPS message is coming from the common reference clock serial support code, and occurs if a PPS signal is detected on the DCD line of a serial port being read by a refclock. In that case, the timestamp of each PPS event is saved and replaces the normal end-of-line timestamp of the following line of serial input read by the clock driver, which is the reason you must configure only one sentence (line of serial input) per second for the user-mode PPS hack to work as intended.
There is an issue with the messages coming from entirely separate
components. Moreover, as I have just said, the presence of user-mode PPS timestamps (replacing end-of-line timestamps on serial input) does
not imply the lack of PPSAPI timestamps on the same port. If I configure the NMEA driver to use PPSAPI directly, user-mode PPS
timestamps remain active and functional from the perspective of the refclock serial support in ntpd on Windows. The fact that higher-level code in the driver only uses those timestamps briefly during
startup before switching to PPSAPI-provided timestamps is invisible to the code that logs the "using user-mode PPS" message.
Using a Serial - USB converter
Many PCs today do not have a serial port, but they have plenty
of USB ports. I wanted to try NTP via a USB port, and I already had
a serial converter which gave my PC COM4. This converter supports the DCD
line, and this is critical to getting good results, as the NMEA output from many
GPS devices is sent at low priority, resulting in drifts on 100 ms or more. However, it appears that NTP
will only support COM ports up to COM3, so I was initially unable to test.
I have a built-in modem on COM3, and these are the results from trying to use a
serial NMEA reference clock (GPS 18 LVC).
After setting the USB device to COM2 in the Device Manager, the first restart of NTP gave an error, and subsequent restarts gave "NT_COM: Device COM2: CreateFile error: Input/output error". Time for a reboot? Working OK now (had to disable the Microsoft Serial BallPoint device in the Device Manager, as expected). Postscript: NTP has since been updated and can now support serial port numbers up to COM255.
Performance - approximate only: compare Narvik (direct connect USB/NMEA/DCD) with Feenix & Stamsund (direct-connect serial GPS reference clock) Bacchus (ref-clock over LAN) here. Please note the different offset scales on these graphs! PC Narvik was changed from a LAN connection to USB ref-clock just before 16:00 in the middle day graphs below, where you can see the transient caused by the multiple reboots.
And just for interest, here is the transition back from using the USB source to using a LAN connection to a GPS-fed stratum-1 server at about 1600 UTC on the middle day:
showing that a direct-connected GPS-serial-USB reference clock is better than LAN-connected stratum-1 server, at least in my setup where the serial-USB adapter carries the DCD signal. Jitter increased from about 45Ás to about 160Ás when reverting to the LAN-alone references. However, it may also be that the jitter reduction was due in part to the shorter polling interval used for the ref-clock than for LAN operation.
I occasionally update the NTP loopstats for the various PCs here. Ask me should you need more recent data. These files may be several megabytes in size.
Although Meinberg's NTP Time Server Monitor is a very handy and comprehensive program, it does not plot jitter, so I wrote a simple NTP Plotter program. This program will take one or more loopstats files on its command-line, or you can drag-and-drop one or more loopstats files for display. In addition, you can have either on the command-line or via drag-and-drop a directory specification and all the loopstats files in that directory will be processed. This is a free and unsupported tool, but your comments are welcome.
On the plot above, you can see the effect of changing from the user-mode serial-port time-stamping with the default serial driver, to the kernel-mode time-stamping offered by the serialpps.sys port driver (provided by Dave Hart) and the ATOM ref-clock driver (part of NTPD). The jitter has been more than halved. The large glitch was caused by having to reboot (twice) to install the new serial driver and remove the phantom Microsoft Ball Mouse which is how the GPS serial stream is detected.
Frequency variation range
The two graphs of frequency error (i.e. the variation in the frequency from its mean value), and the offset graph, show quite clearly the effects of temperature variation. During the night (0000-0600 UTC and local) the room cools down, until the heating comes on automatically at 0600. At that time, the offset starts to increase and the frequency increases to try and reduce the offset. By about 1100 the offset has been reduced to near zero, and the frequency stabilises. At 2300, the cycle repeats. There are three days in the above plots, and the three cycles are clear to see. Interestingly, there is little diurnal variation in the jitter value, and while using kernel-mode time-stamps reduces the jitter, it does nothing to reduce the offset value, this being much more a function of the NTP algorithms and their interaction with the temperature-induced oscillator drift.
Early comments from Dave Hart
1. The latest binary is now actually 20090301 with
refclock_atom support, but if you're not using refclock_atom 20090226 should
11 April 2009 - comments from Dave Hart
David, feel free to quote anything here as you feel appropriate in email to others or on your web pages. As David indicated, the disabling of the code which reported "using serialpps.sys timestamps" was intentional. That was part of the process of preparing my "QPC" ntp-stable source code tree, loaded with all sorts of fixes, to have related changes pulled over into either both ntp-stable and ntp-dev (for bugs) or into only ntp-dev (for enhancements). There are three major pieces I was thinking about:
Keep in mind the need to decide whether something goes into -stable and -dev at the same time, normally bugs only, and depending on how close the next new -stable release is, only more serious bugs would be put in -stable because a new -stable version means throwing away the old -stable code and making the current -dev (for an instant) the new -stable. Changes which can't be classified as bugs, or aren't considered serious enough, go in -dev only and don't make it to -stable until the next non-patch stable release.
Point 1 above (serial I/O improvements) is clearly a bug and as a result was the first part I managed to peel off and integrate into the official NTP release, as of ntp-stable 4.2.4p7RC1 and ntp-dev 4.2.5p160. That includes the "user-mode PPS timestamp" part but not the "serialpps.sys timestamps" part disabled in the latest QPC private release of mine.
Point 2 above (synthetic timeline or "interpolation" 1 ms error) is harder to call a bug. The old code has more error, but either approach is an estimate with some inherent error expected compared to the preferred approach of the OS presenting a smoothly-increasing timescale. Add into consideration that it has been over two years since the last non-patch ntp-stable release (4.2.4) and two years of divergence has accumulated between the ntp-dev and ntp-stable trees, far too much for everyone's comfort. That means a 4.2.6 release should be coming very soon, and everyone believes it will happen any month now (sigh). On balance I decided to only attempt to integrate the new interpolation scheme into ntp-dev, and that has only just happened in the last few days with the release of 4.2.5p162. 4.2.5p163 is identical to 4.2.5p162, the only reason for the bump was a missed step of pulling 4.2.4p7RC2 ntp-stable final version number changes into ntp-dev, where they were then replaced as usual with the ntp-dev version number, bumped without changing anything else, to 4.2.5p163. This was a big relief to me as these two pieces are the real meat of my private releases' patches.
Point 3 above (serialpps.sys support) is a bit of a can of worms by comparison, and the benefit (while real) isn't huge compared to using only NMEA + user-mode timestamps possible with any Windows serial port, using serial.sys or not. It is not forgotten, but to be honest I'm taking a bit of a breather and helping with some other NTP project issues and haven't done anything to move that forward into the official NTP code yet. The biggest issue in my mind is licensing. As I read the Windows Driver Kit license agreement and related documentation, I have no right to distribute serial.sys object code (binary file such as serialpps.sys). You may have noticed serialpps.sys contains only a patch file showing my additions to the WDK serial.sys source code, and doesn't actually include the source code, you have to get it from the WDK. That was motivated by the same licensing concern. I have not pulled down my serialpps.sys binaries (in fact I cross-compiled x64 and Itanium versions and posted them as well since I dug into the licensing) because I can't imagine Microsoft would even bother to send me a cease and desist letter, let alone take more serious legal action. Largely, of course, the binaries I provide are only useful to those using Windows. Here it's important to note that the NTP distribution is as close to public domain software as you could hope. The license is a BSD-style do what you will and don't hold us liable or use our name to promote your product boilerplate. Dr. David L. Mills, the major author, is adamant about keeping NTP freely usable by anyone any way, in particular including in commercial products. There are at least potential emotional, if not legal, conflicts in including support for a serialpps.sys that we can't include in official distributions in source or binary form, and would have be obtained, arguably illegally, from me or others who have mirrored serialpps .zip files from my website.
After all that damned detail and history, I still haven't explained why
I disabled the "using serialpps.sys timestamps" code path from the Windows serial I/O code in ntpd in my latest binaries.
Basically, knowing the serialpps.sys changes would be the last major bit I'd try to move into the official code, I looked at my two parallel implementations in the Windows serial I/O code (providing timestamps via NMEA or another serial-based refclock driver) and in PPSAPI (providing timestamps via the atom/PPS refclock driver) and decided the earlier implementation in the serial I/O code definitely wouldn't make it in with the rest of the serial I/O changes going into both -stable and -dev, and also were contrary to Dr. Mills' expressed desire to concentrate PPS functionality in the atom/PPS driver.
Also, I did not want to risk my sneaky slipping of the user-mode PPS support into the Windows serial I/O code as part of a general redesign and cleanup of that code to be potentially derailed by the original serialpps.sys support code and its licensing and Mills issues, when the same precision serialpps.sys timekeeping works fine via Mills' preferred PPSAPI/atom solution.
So while you are free to keep it alive in local patches, I consider the serialpps.sys support in the Windows I/O
code-path a fun bit of history that I have no plans to integrate into the official code at this point (because of no benefit vs. PPSAPI approach and the concerns).
Even with the ugly licensing situation, I think the PPSAPI/refclock_atom approach to serialpps.sys support will make it into ntp-dev soon. That may not be before 4.2.6 is released, however. In the meantime you can continue using my QPC releases and if there's good reason, I can always fix something in that tree and make another release.
[My sincere thanks to Dave Hart for his work in Windows, and for his permission to include this most illuminating commentary of his significant enhancements to NTP, and the development processes involved in an open-source project].