MSG Data Manager
Home Page Up ATOVS Reader AVHRR Manager BUFR Viewer CHLO Viewer CMA Viewer DDE/SDR# add-on DWDsat HRPT Viewer GeoSatSignal GRIB Viewer Ground Map HDF Viewer HRPT Reader HRPT samples Kepler Manager Metop LSA Viewer Metop Manager MODIS L1 viewer MSG Data Manager NOAA HDF Viewer SatSignal Sea-Ice & SST Viewer WXtrack

 

The MSG Data Manager

What is MSG?

Meteosat-8 (or MSG-1 as it was known during trials) is the first of the new European geostationary weather satellites.  MSG is an acronym for Meteosat Second Generation.  Meteosat-8 served from 2003 to 2007, and is now the standby and Rapid Scan satellite, with Meteosat-9 as the prime satellite since April 2007, and Metoesat-10 (MSG-3) since January 2013.  Meteosat-8/9/10 has twelve sensor channels rather than three, higher spatial resolution, much higher visible channel resolution, better thermal accuracy, and double the frequency of image delivery than previous Meteosat satellites; all of which necessitates a change to the data transmission method.  Meteosat-10 (MSG-3) has replaced Meteosat 7, and the WEFAX and PDUS transmissions have now ceased.  The data from MSG is purely digital, and is delivered as a set of a number of compressed files per image.  As there are now twelve channels delivered four times an hour, the total data rate is some 1.5GB per hour, 15000 files per day, requiring significant management on your computer.  

Please note that you will need a licence from EUMETSAT to receive some of this data, but UK amateurs should see the GEO Quarterly for more details.  They are unlikely to have to pay licence charges.  In addition to Meteosat-9 data and other geostationary weather satellite data, you will get access to the upper-atmosphere ATOVS data and Sea-Ice data.  These are processed from polar orbiting satellites and added to the data stream.

 

What is the MSG Data Manager?

Download  Download the MSG Data Manager

The MSG Data Manager is a Windows program that will process the files received from Meteosat-8/9/10.  These files might be received over the Internet in the future, but today they are easily received with a Digital Video Broadcasting (DVB) card installed in your PC, and a standard domestic satellite antenna, LNB, and cable.  A description of my own system is here.  The program can also process data received from the EUMETSAT FTP service (binary transfers).  The data manager will automatically process the files into images, and optionally delete the large amounts of source data that would otherwise accumulate.  The program allows you to choose which of the twelve channels you process to images, and also allows you to keep some raw data in the processing PC should you wish.  
Please note that the MSG Data Manager and MSG Animator are original pieces of software developed independently by SatSignal Software, Edinburgh specifically for users such as you, and there is a lot of your input in the programs.  These are not some re-branded programs originally developed by someone else.  You will always get a fast response to the frequent changes made to EUMETCast data, and not be stuck waiting for the "next version".
More information about the data flow with MSG.

An Award!

My demonstration of Receiving and Processing Meteosat-8 Data with the MSG Toolset Plus suite (which comprises the MSG Data Manager, MSG Animator and GeoSatSignal) was voted the best of conference at the 2004 EUMETSAT Meteorological Satellite Conference in Prague.  The judgement was made based upon scientific content, presentation and overall interest and relevance.

Who uses the MSG Data Manager?

This software has been supplied to the Finnish Meteorological Institute, Meteo-Swiss, Universities, Museums, and other professional and amateur users throughout Europe.  

  • "We are very happy with MSG Data Manager: it does what we want, in a perfect stability" - Michel Albuisson, ╔cole des Mines de Paris.
  • "We use the MSG software 24/7 for operational forecasting information and it allows us to run animations of several channels simultaneously in different windows on dual displays" - Alan Hisscott, Senior Meteorological Officer, Isle of Man Meteorological Office.
  • "I'm very impressed with the software. When the system was running too low on hard drive space the data manager stopped updating but nothing crashed, and the animator continued with old data - very nice!  I am happy to recommend your applications." - Jeff Lashley, UK National Space Centre.
  • "My registration has just arrived for upgrade to MSG Data Manager and also AVHRR manager.  I must admit I'm blown away by it and have just sat down again after dancing around the room!  What have I been missing ah?  Awesome." - Rob Denton
  • "Following the recent need for users to activate the MSG-2 data flow, I checked with Ops and they confirmed that since the 'failure' of MSG-1, all users had MSG-2 data enabled.  With David's software MSG Data Manager, all I had to do was change over one setting in the advanced section in order to resume full operations.  David's software has again proved invaluable and exceptionally easy to configure. My grateful thanks, David." - Lawrence Harris, freelance journalist.

 

What PC configurations are supported?

You can use a single PC for receiving and processing data if the PC is powerful enough, or a two-PC network is supported, with the lower-spec PC as the Reception PC with the DVB card, and the higher-specification PC as the Processing PC, grabbing files off the Receiver PC over the network.  You can ask about any particular configuration on the MSG-1 Yahoo Group, where hundreds users with working systems can help, and there is a discussion about PC requirements here.  Windows-7 or 8 is required for support - older versions are not supported.  I would expect XP/32 and Vista/32 still to work but I cannot support these versions as I no longer have PCs running XP or Vista.  

Data Volumes

EUMETSAT state that the expected data volume is around 7000 files per hour (I think that's too high) totalling 540MBytes.  If you take EARS data and RSS data that increases by 44MB.  Therefore a full day's data would be about 15GB.  If you keep all 12 MSG channels, but save the images as JPEG not PNG, the saved images amount to 1.1GB per day in 1152 files.

 

What functions does the MSG Data Manager have?

  • Can work on the same PC as data reception or on a different networked PC
  • Automatic monitoring of the Reception PC for the presence of new files.
  • Automatic copying of those files from Reception PC to Processing PC
  • Optional deleting of raw data files on the Reception PC
  • Selection of which channels to process, reducing the subsequent processing and storage requirements
  • Processing includes both HRIT and LRIT Meteosat SEVIRI data
  • Processing includes both Meteosat-5 and Meteosat-7 data
  • Processing includes MPEF, MDD, GTS and DCP data
  • Decompressing of the data files coded in JPEG and Wavelet Transform formats
  • Data reduction from 10- and 12-bit source data to 8-bit display data
  • Fully calibrated 10-bit data saving (PRO version only)
  • Real-time thumbnail display of selected channels as data is received
  • Optional gamma-correction, gain and offset processing for visible channels
  • Temperature calibration of infra-red channels
  • Saving of processed data in standard image formats
  • Choice of PNG for full precision or JPEG for disk economy
  • Management of HRV region selection (offset and northern-area)
  • Local display of multiple detail images at full resolution 
  • Histogram equalisation and false-colour palettes for detail images
  • Histogram display of 10-bit received data (PRO version only)
  • Management of received data - older data can be automatically deleted
  • Display of Admin messages sent over the data stream, also saved as text files
  • Copying of Sea-Ice data for further processing
  • Copying of [Data Channel 1] ATOVS and RSS data for further processing
  • Optional program priority drop when CPU-intensive operations are underway
  • Optional sound announcements at start and end of scan
  • Optional alert on data loss - you can send yourself an e-mail!
  • Graphical display of scan progress

For an extra charge, you can also get the MSG Data Manager PRO version.  This is aimed primarily at those doing research, where it is important to be able to access the full 10-bit data, rather than have the data converted to 8-bit images.  If you are writing your own software for processing MSG data, the PRO version is probably what you need.  Extra functions in the PRO version include:

  • The ability to save raw 10-bit data in PGM files from Meteosat-8/9, and full-precision data other satellites
  • Calibration data is saved in the PGM files as well, allowing the recovery of radiance values
  • Eight "Detail Image" displays (versus four in the standard version)
  • A right-click, Histogram Display function for the thumbnail images
  • A right-click, Histogram Display function for the detail images
  • The ability to choose the save format (default, JPEG or PNG) on a per-channel basis

If you wish to process the images further, for example to make false-colour combinations, remap to standard map projections, or to animate the images, you could consider my GeoSatSignal program.  For purpose-built real-time animations, without colour or rectification to standard mapping, my MSG Animator program is the tool of choice.

 

What data does the MSG Data Manager process and display?

Image data - processed and displayed
Meteosat-8/9/10 HRIT image data 11 channels at 3km resolution
1 HRV channel at 1km
Displayed in MSG Data Manager
Meteosat-8/9/10 LRIT image data 5 channels at 3km resolution Displayed in MSG Data Manager
GOES-West image data 4 channels: visible, 2 x IR, WV Displayed in MSG Data Manager
GOES-East image data 4 channels: visible, 2 x IR, WV Displayed in MSG Data Manager
Meteosat-7 image data 3 channels: visible, IR, WV Displayed in MSG Data Manager
GOMS1 image data 4 channels: visible, 3 x IR Thumbnail displayed in MSG Data Manager
FY-2D/E data 6 channels: 2 x visible, 3 x IR, WV
1 visible channel at 1 km
Displayed in MSG Data Manager
MTSAT image data 5 channels: visible, 3 x IR, WV Displayed in MSG Data Manager

Non-image data - processed only
Reception of some of this data may require an extra EUMETSAT licence
MPEF data Multiple data types Some data viewable with the GRIB Viewer
DCP data Multiple data types
DCP/WMO data Multiple data types
MDD data Multiple data types
MDD/GTS data Multiple data types
OSI SAF Ocean and Sea-Ice data Multiple regions GRIB data viewable with GRIB Viewer software
Specialised Sea-Ice and SST viewer available
LSA SAF Land Surface Analysis data Multiple regions Some data viewable with the HDF Viewer
ERS/SCAT scatterometer data Wind data view with BUFR Viewer
ATOVS data From polar orbiters L1C and L1D visualised with the ATOVS Viewer
BMD data Multiple data Some data viewable with the GRIB Viewer
DWDSAT data Multiple data Some data viewable with the GRIB Viewer
MODIS Wind data Wind data view with BUFR Viewer

 

MSG Data Manager Operation

The program first copies data received by the DVB system from the Receiver PC to the Processing PC, and optionally deletes it from the receiver PC. You can select which channels are to be deleted, and which copied for processing. As the satellite scan proceeds, you will see the images build up from south to north, and once an image is complete it will be saved to disk. Visual channel images are JPEG compressed, and thermal channel images PNG compressed (lossless).

You can use the Windows Control Panel to select sounds to be played at the start and end of a scan (specifically when the Prologue and Epilogue files are processed).

The program is intended to be left running continuously, copying and processing the received data.

 

MSG Data Manager Screen Shots

Here are some screen grabs from a recent version of the MSG Data Manager.  

The first thing you need to do on getting the program is to tell it where to look for the files from EUMETCast, where on the processing PC the raw data is to be stored, and where the processed images should live.  Note that both of these latter directories will use an addition year/month/day naming convention for the folder names so that the number of files per directory is kept at a reasonable level.
You can then set a number of other options, in particular which channels are to be processed and what should happen to the raw data.

Once setup is complete, you press the Run button and switch to the MSG HRIT tab, where data from all the selected channels builds up.  During the operation of the program, you will see the scan build from South to North.  Any gaps at the top of the images are there because that part of the earth has not yet been scanned.  Watching the pictures build like this is fun!  The channel spectral region will pop-up as your mouse moves over the thumbnail.  Note that the last image, that of channel 12, is not square like the others, but has a 2:1 aspect ratio.  It is over 11000 pixels high and over 5000 pixels wide!  The imager on the satellite can be adjusted to cover different parts of the Earth on the north and south parts of the scan.  Here you see it almost centred on Europe in the northern part of the scan, but centred on Africa for the majority of its scan.  The program centres all the data to make the composite picture.  The scan position is altered from time to time, as shown in this 2MB animation from EUMETSAT.
Click on the image below to see an animation of the MSG Data Manager at work!

MSG Data Manager HRIT screen-shot - click for an animation
(screen-shot from more recent program version)
Click for an animation

The FSD (Foreign Satellite Data) tab shows the LRIT data relayed from satellites around the globe.

MSG Data Manager - Foreign Satellite Data screen shot
(screen shot from beta program version)

On 2004 May 27 at 08:15 UTC I selected channel 12 by clicking on the image, and then moved the view to the are around Sicily and mount Etna.  The snow-covered peak of Etna and the darker volcanic soil around Etna are both easily visible.  The islands of Gozo and Malta are visible to the south of Sicily.  All these images are from EUMETSAT data.

More recent versions of the MSG Data Manager allow multiple Detail Image pages, and allow the detail images to be coloured with your choice of visible or thermal palettes (Colour Lookup Tables or CLUTs).  On the two images below, note how the low fog-like cloud off Norway and the southern coast of the UK stands out well in the visible channel, but not in the thermal channel with the particular LUTs in use.  (I show the PRO version of the MSG Data Manager below.  This has eight detailed displays versus four with the Standard version).


Channel 9 Meteosat-8 data with ZW colour enhancement


Channel 2 Meteosat-8 data with pseudo WEFAX colour enhancement

 

On May 18th, 2004, much the UK and Ireland was clear of cloud.  Here is an HRV (channel 12) image from Meteosat-8, rectified to orthographic projection in GeoSatSignal.  False-colour is achieved by using data from the 10.8Ám thermal channel.

 

Rapid Scanning Service

In May 2008 EUMETSAT started a Rapid Scanning Service which uses the MSG-1 (now MSG-2) satellite now located at 9.5░ East to provide a new image of North Africa and Europe every five minutes.  This provides considerable improvements for "nowcasting" and of course can produce superbly smooth animations!  To achieve the fast scan, only the northern-most part of the globe is scanned, resulting in just three image segments being transmitted (and nine for channel 12 HRV).  To distinguish the rapid scan mode of operation, and to provide a slightly more detailed view of Europe, the MSG Data Manager will provide a stretched display of the thumbnails when operating in rapid Scan mode.  You can read about running two copies of the MSG Data Manager, one for normal scan and one for rapid scan, in the FAQ.  The screen display will appear as shown below.  Do you like this mode?  If not, please come and discuss it in the SatSignal Yahoo group.  Of course, the Detail Images and the saved images retain their correct shape and can be processed as normal - it is only the thumbnail display which has this distinctive appearance to show that RSS is being received.

MSG Data Manager - HRIT thumbnails appearance
The special appearance of thumbnails in HRIT/Rapid-Scan mode

 

How do I get the MSG Data Manager?

  • You can purchase a licence for the MSG Data Manager online.
  • You can try the program before purchase by downloading the program below, as it has a built-in 30-day trial licence.  At the end of your trial period, you will need to either stop using the software or purchase a licence.
  • Please note that Windows 98, 98SE and ME, NT 4, and 2000 are not supported, and whist the program may function, some error messages may occur because these operating systems have trouble supporting the very large image sizes.
  • There is a bundled offer of the MSG Data Manager, the MSG Animator and my GeoSatSignal software in the MSG Toolset Plus package, which is a very economical way to purchase.

This software has been supplied to the Finnish Meteorological Institute, Meteo-Swiss, Swiss, Italian and Spanish TV, Weather Services, Airports, Universities, Museums, and professional and amateur users throughout Europe.

Downloads

V2.6.10   Make the default satellite MSG-3, update Met-7 Spring eclipse dates, add FY-2C/D/E 1 km visible as a Detail Image choice (need to start with standard resolution image), add support for parallel-scan rapid-scan data (data is saved with RSS in the file name, but in the images\MSG-PAR tree), support h04/h05 SAF data, reporting updates for PAR/RSS data.
V2.6.12   Support for h12 and h13 hydrology HSAF file names, support SARAL/AltiKa data on Channel 10, support O3M SAF high-resolution data (files named starting W_NL-KNMI-DEBILT,SOUNDING+SATELLITE,METOPB+GOME2, treated as SAF, GOME-2 O3+NO2), start MPEF OCAE support including reporting, omit "GOES-12 from GOES-10" in RSS or PAR report, layout update for MPEF check-list boxes, correct MTSAT direct visible save name to "mt1-vis" (the "mt1" prefix covers both MTSAT-1R and MTSAT-2, add raw calibration data to JPEG and PNG for MTSAT direct, will refer to 0..1023 scale, but our data is just 0..255, honour JPEG/PNG save option for MTSAT direct), update Met-7 & GOES-E/W autumn eclipse dates.
V2.6.14   Better land/sea mask on HRV (channel 12) detail image (needs files from GeoSatSignal 7.4.2, MaskMet8HRV.png & MaskMet8HRV-9.5E.png, also included in the MSG Data Manager beta archive), preliminary GOMS support, four channels (EUMETSAT says IR 9.7 Ám & 10.7 Ám channels are "blurry", saved files named: 201310080700-goms1-vis3.jpg,  201310080700-goms1-ir6.jpg, 201310080700-goms1-ir8.jpg,  201310080700-goms1-ir9.jpg, just vis3 and ir9 used GeoSatSignal), reporting updates (change TOZ reporting from 3-hourly to 1-hourly, show computer name in saved report).
V2.6.16 Experimental version allowing more memory in 64-bit (Windows-7/8 systems with more than 4 GB RAM, program not supported on XP/64 or Vista/64, should work on all 32-bit Windows - XP SP3 and later), add 3.8 Ám and 11.9 Ám GOMS-1 support, add second FSD tab showing main GOMS-1 and FY2D data, add Detailed View support for GOMS-1 and FY-2D (NB. displaying multiple 1 km resolution images could cause the program to run out of memory and stop working. Be very careful - hence the experimental 64-bit memory support added), scan both Data Channels 11 and 12 for CMA data, extend Advanced Setup for newer channels, correct retention purge of W_NL-KNMI-DEBILT SAF files.
V2.6.18 Add nowcasting NWC SAF support data saved in SAF directory, open to negotiation!), add support for new format EUMETCast News messages (file names starting: EUMETCast_Meteosat, plain text files, honour text message retention times), add support for Global NOAA ATOVS data (files in Data Channel 4), add support for NCEP_forecast for Africa data (received directory ..\WMO-RA-I\ will be scanned, file names like: ncep_forecast_20140223_gfs.tar, will be copied to the ..\images\DWDSAT\ tree), use the file date & time for the slot for AMESD file names which can't otherwise be decoded, add support for changed FY-2D/FY-2E data as the HDF format was changed without notice, remove support for a couple of DevCoCast-1 products as this project has now ended, and the product  names conflicted with other products, update Met-7 Spring eclipse dates, restore LRIT direct reception function, avoid reporting misleading singleton in missing slots.

Please note: Windows-XP and Vista support has been dropped in versions after 2.6.16, although the software should continue to work.

Important: This program includes a 30-day trial licence.  Please wait until you have data to test before installing the program, as the 30-day countdown starts from the first time the program is run!

This program requires a display size of at least 800 x 700 pixels.

If you are interested in MSG data, did you know that with the same EUMETCast reception hardware you can now get high resolution AVHRR data from the polar orbiting satellites such as NOAA-18 and Metop-A and Metop-B  I have software for these EUMETSAT options in my AVHRR Manager, DWDSAT HRPT Viewer, and my Metop Manager.

 

MSG Animator

The MSG Animator provides real-time animation of images from Meteosat Second Generation and other weather satellites received with the MSG Data Manager.  It is dedicated to EUMETCast data.  If you want false-colour animations with geographic reprojection and weather chart overlay, please look at my GeoSatSignal program

Download Download the MSG Animator

Program highlights

  • Multiple simultaneous, independent animations, updated in real time as new data becomes available
  • Selection of channel and graphical selection of region subset, with country boundary overlay
  • Animation of Meteosat-7, 8, 9 & 10, MTSAT-1R, & GOES-E & GOES-W data providing world-wide coverage
  • Looping and smoothing of animations, with optional histogram and contrast stretch image-processing functions
  • False-colour option with land/sea separation and dark-cloud tops
  • Control over animation length with option to choose visible hours only
  • Save an animation in standard Windows AVI format
  • Kiosk-mode operation for interactive public displays and museums
  • Automated animation switching for reception areas etc.
  • Can work remotely from the MSG Data Manager over a network (requires the runtime Library Bundle to be installed on the PC where the MSG Animator is to operate).
  • Windows-7 or Windows-8/8.1 is required.  The earlier Windows XP/32 and Vista/32 versions are not supported.

Sample animations

Here are some sample still screen-shots showing the MSG Animator in action.  You can use thermal channel, water-vapour channel or visible channel data as the source of the animation and, by using colour look-up tables, this date can be made easier to understand.

This data is from Meteosat-8, channel 9, which is the main thermal channel.  The data has been converted to temperatures, and used with a land/sea palette where the higher temperatures are shown as green (land) or blue (sea), but the lower temperatures are shown as increasingly dark cloud colours.  The darkest areas show the most likely areas for rain.
This data is from the Rapid Scanning service and this time the palette used shows the colder temperatures in yellow and red.  This satellite scans every ten minutes, enabling a very smooth animation to be produced, but it does not scan as far north, hence the coloured band at the top of the image.
This data is from the Meteosat-8 HRV (high resolution visible) channel and is also used in conjunction with a land/sea palette just working on the brightness values in the data.  The city data is held in a text list, so you can easily add your own towns.

Getting the MSG Animator

Downloads

V2.6.6 Add "PAR" parallel option (should work for both SEVIRI and MPEF data, designed for parallel running operation - e.g. early MSG-3), new convection LUTs.
V2.6.8 Add new "Winter stretch" video processing option for visible channel images.
V2.6.10 Accept rapid-scan and parallel options together.
V2.6.12 Start work on mini- and micro- full disk (press Shift or Control for mini and micro when changing from normal to full-disk, may need to be repeated if other setup options changed, acknowledged by a beep when changing).
 

Visit the SatSignal Yahoo group to discuss this software with other users

 

Specialised software

In addition to the software listed on this page, I have a number of other software packages for processing Meteosat-8/9 data including MSG to GeoTIFF for converting the data to radiance, brightness-temperature and reflectance and saving the results as GeoTIFF files, and the MSG Fire Tracker for detecting fire hot-spots using a multi-channel algorithm and reporting the results up to an FTP server.  Please contact me if you are interested in this software.  I can also supply readers for the EARS ATOVS data, the Sea-Ice and Sea-Surface Temperature (SST) data, and the GRIB data from EUMETCast MPEF and DWDSAT data broadcast over the DWDSAT service.

Fire detection with Meteosat-8/9/10

For example, here is a sample screenshot from the MSG Fire Tracker program.  Using Meteosat-8 data from 07 August 2005, fires have been detected in North Africa, Spain, Portugal and the south of France.  The detected fire pixels are highlighted by yellow rectangles in the image below.

MSG Fire Tracker screen-shot
Sample screenshot from the MSG Fire Tracker program

 
Here's another screenshot from 2009 Jul 23 showing fires in Spain.  My thanks to Harry at http://www.meteomalaga.es.  You can see Harry's current data here.

 

MSG timestamp program

Some data sources such the EUMETSAT Archive (MARF) provide MSG HRIT segment files which can be processed with the MSG Data Manager, but the timestamp on these files is either incorrect or becomes lost when transferring between the source and your PC.  The MSG timestamp program is a very simple, free utility which will timestamp MSG HRIT according to their file names, so that they will process in their natural order.  This is useful to allow the MSG Data Manager to process these files, and may help other manufacturer's software as well.  This is a free utility.

 

Sample images from Meteosat-8 (MSG-1)

Guidelines on publishing EUMETSAT data

UK Snow - January 2004

This image was from January 29th 2004 after extensive snow-falls in the UK.  The high-resolution visible channel (HRV) has been combined in GeoSatSignal with thermal information from the 10.8Ám thermal channel to make a false-colour image, which has then been rectified to a polar-stereographic projection.

Image copyright EUMETSAT 2004.

UK Snow January 2004

 

Tropical cyclone Manou - May 2003

Here is an image taken during the trials of EUMETSAT's MSG-1.  It show tropical cyclone Manou off Madagascar.  It was processed by the MSG Data Manager and Paint Shop Pro.  Image taken 2003 May 08 at 0744 UTC.  Click for a full resolution version (130KB).  Note that the image is near the edge of the field of view, and there appears slightly squashed.  The HRV channel provides images in the visible waveband, 11136 pixels by 5568 pixels.

Image copyright EUMETSAT 2003.


Click the image for a full-resolution version


Other data from your EUMETCast service

In addition to the Meteosat-8/9 and other geostationary weather satellite data carried on EUMETCast, you will also have access to Sea-Ice concentration, sea-surface temperature data, and ATOVS atmospheric sounding data.  I can provide programs which decode and display this data if you are interested in these aspects of the weather.  Here are some sample images to whet your appetite....

Sea-surface temperature sample data

Sea-surface temperature sample screen-shot

Sea-surface temperature sample data

Sea-Ice sample data

Sea-ice concentration sample screen-shot

Sea-Ice sample data


ATOVS sample data

ATOVS sample data

 

 
Copyright © David Taylor, Edinburgh   Last modified: 2013 Aug 16 at 07:52