NorCPM post-processing tools for Climate Futures

This documentation is designed to allow the user to run and post-process the seasonal NorCPM1 forecast, which produces output for the Climate Futures project.

1. About the seasonal forecast

This page contains some brief notes about the seasonal forecast which is run every month for the Climate Futures project. The seasonal forecast consists of 60 ensemble members and is run every month. Some basic anomaly, quantile and index plots are made available here.

Raw hindcasts for the seasonal forecast are located in NIRD in /projects/NS9873K/norcpm-cf/raw/norcpm-cf-system1/norcpm-cf-system1_hindcast1/. Hindcasts and forecasts are also regridded, and their metadata modified in order to match other C3S seasonal forecasts for provision to the Climate Futures project. Select post-processed variables are available through the NS9039K web interface at http://ns9039k.web.sigma2.no/seasonal_forecast/climate_futures/.

In order to perform accurate forecasts each month, up-to-date SST, salinity and temperature data is assimilated into the system. The SST data assimilated is the OISSTv2 data from NOAA, which from December 2022 is being replaced by the OISSTv2.1 dataset. Temperature and salinity profiles are downloaded from the UK Met Office EN4.2.2. dataset here.

2. Running the seasonal forecast

This page provides documentation on how to run the NorCPM seasonal forecast. In general there are four steps to this process:

  1. Update the observational data.

  2. Run forward the analysis to present day, using new observational data.

  3. Run the prediction/forecast.

  4. Clear up, and post-processing (see next page).

The forecast should always be provided on the evening of the 14th of the month, as our collaborators need it on the 15th.

The Met Office provides monthly updates to EN4.2.2. observational temperature and salinity profiles, and these updates happen on the 9th-17th of each month. If this update can be included into the assimilation then this is good, as it gives us an additional month of temperature and salinity data to assimilate. You can check when the last update was on the Met Office site (see bottom of page, or look at file links). I would recommend running the analysis on the 12th/13th of the month. It takes ~3 hours to run. I would then recommend running the prediction on the 12/13th, this takes ~2 hours. The post processing takes another ~2 hours. Therefore, if there are no issues the whole workflow takes around 1 working day, but doing this on the 12/13th allows for potential issues (e.g. machine instability) to be addressed. It is important to keep an eye out for Betzy downtime announcements in the week preceding the 15th to ensure there is no significant disruption.

2.1 Update the data for assimilation

The first step in running the seasonal forecast is to update the observational data that is assimilated into the analysis.

Update the temperature and salinity profile data.

cd /cluster/shared/noresm/norcpm/Obs/TEM/EN422/
./Update_profile.sh

Now the SST, temperature and salinity profiles are updated.

When it comes to running the analysis, it is important to note that some of the temperature and salinity profiles are initially uploaded by the Met Office as ‘preliminary’, and then are replaced later by ‘final’ files. We always want to run the analysis with the best possible data, which means that each month, we will re-run the part of the analysis that used ‘preliminary’ data last month. This means that it is important to keep track of what the latest observational files are each month.

A log file of each SST, temperature and salinity retrieval has automatically be generated by the download scripts. This includes a file listing, so that the most recently downloaded final and preliminary files can be seen.

For example the file listing might include the files:

EN.4.2.2.f.profiles.g10.202207.nc
EN.4.2.2.p.profiles.g10.202208.nc

This means that we have preliminary profiles for 2022/08 and final profiles for 2022/07. Next month when we run the analysis, we will run it from 2022/07.

2.2 Run the analysis

Move into the analysis directory. The settings file for this seasonal forecast is then in settings/default_settings_VCF.sh

cd /cluster/projects/nn9039k/people/tbi045/NorCPM/analysis/
ls settings/default_settings_VCF.sh

Check which date should be the start date for the analysis run this month. This is the date of which the last ‘final’ EN.4.2.2. file was for last month’s run.

Check if the correct restart files for your start date exist in your run directory (usually these will be automatically deleted).

ls /cluster/work/users/tbi045/noresm/norcpm-cf-system1_assim_19811115/norcpm-cf-system1_assim_19811115_mem01/run/

If they do not exist, then you need to copy them over by providing the start date as an argument in the form YYYY-MM-DD.

cd script_pot/
./mv_rst_from_archive_to_run.sh YYYY-MM-DD

To run the analysis, make sure that you load the correct environments, and make sure you provide a start date argument to sbatch in the form YYYY-MM-DD

source /cluster/projects/nn9039k/people/tbi045/NorCPM/analysis/load_envs.sh
sbatch submit_reanalysis_VCF.sh YYYY-MM-DD

Check that the analysis ran to the expected month (it will crash when there is no more data to assimilate for a month).

2.3 Run the prediction

Now that the analysis is updated, we can run the prediction.

Move into prediction directory.

cd /cluster/projects/nn9039k/people/tbi045/NorCPM/prediction/use_cases/

Create a new use_case file. The date in the filename should be changed to the date of last data assimilated. The date inside the file needs to be changed in two places

cp norcpm-cf-system1_hindcast1_20221015.in norcpm-cf-system1_hindcast1_<date>.in
vi norcpm-cf-system1_hindcast1_<date>.in

Obtain restart files using your new use_case file

cd /cluster/projects/nn9039k/people/tbi045/NorCPM/prediction/
./obtain_restarts.sh use_cases/norcpm-cf-system1_hindcast1_<date>.in

Create a template, create an ensemble, and submit the prediction.

./create_template.sh use_cases/norcpm-cf-system1_hindcast1_<date>.in
./create_ensemble.sh use_cases/norcpm-cf-system1_hindcast1_<date>.in
./submit_ensemble.sh use_cases/norcpm-cf-system1_hindcast1_<date>.in

Check that the prediction ran to the expected month, and was archived correctly e.g. check files have been moved to archive.

ls /cluster/work/users/${USER}/archive/norcpm-cf-system1_hindcast1_20221115/norcpm-cf-system1_hindcast1_20221115_mem01/atm/hist/

2.4 Clearing up

Run mergediag.sh (modify the directory name in the file).

cd /cluster/projects/nn9039k/people/tbi045/NorCPM/tools
vi mergediag.sh
sbatch mergediag.sh

This produces a merged directory in the archive (takes some time) e.g.

/cluster/work/users/${USER}/archive/norcpm-cf-system1_hindcast1_20220915/norcpm-cf-system1_hindcast1_20220915_mem01-60

Copy forecast files to NIRD, using scp or rsync from above directory. e.g.

cd /cluster/work/users/${USER}/archive/norcpm-cf-system1_hindcast1_20220915/
scp -r norcpm-cf-system1_hindcast1_20220915_mem01-60/ $USER@login1-trd.nird.sigma2.no:/projects/NS9039K/shared/ClimateFutures/new/

Delete restart files that will not be needed anymore. You need to determine which restart files you will need next month. The analysis next month should start from the last ‘final’ assimilation files that were used, so in this example, if temperature and salinity profiles were noted down as:

EN.4.2.2.f.profiles.g10.202207.nc
EN.4.2.2.p.profiles.g10.202208.nc

Then the analysis will continue from 202207, so keep these restart files, with care, remove any others e.g.

cd /cluster/work/users/${USER}/archive/norcpm-cf-system1_assim_19811115/
rm -rf norcpm-cf-system1_assim_19811115_mem??/rest/2022-04-15-00000/

Compress files.

cd /cluster/projects/nn9039k/people/tbi045/NorCPM/tools
sbatch fanf_noresm2netcdf4.pbs

Move analysis files to NIRD, on NIRD:

cd /projects/NS9039K/shared/tbi045/scripts/
./Transfer_reana.sh

Check the files on NIRD, and this concludes generation of the forecast. For steps on processing the output, please see the separate documentation page.

3. Processing seasonal forecast

This page provides documentation on how to process the seasonal forecast every month. This begins with running scripts that modify metadata so that the forecast can be provided in a C3S-like format to the Climate Futures project. After this, some standard plots and indices are produced, and the website. is updated.

3.1 Metadata processing

In order to perform the metadata processing, you will need to load the shared conda environment on NIRD.

source /projects/NS9039K/shared/python3env/etc/p3env.sh

Then you need a copy of the GitHub repository with the processing scripts. If you don’t already have this, then take a clone of it.

git clone git@github.com:BjerknesCPU/C3S-NorCPM-CF.git

Now run h0.atm and h2.atm to process monthly and six-hourly data respectively. You need to modify the INPUT_GLOB and OUTPUT_DIR variables in the Input section of both files, then run.

python h0.atm
python h2.atm

These should produce files in your selected output directory which have modified metadata. These should be placed somewhere that can be accessed by Alex Lenkoski, the current location is:

/projects/NS9039K/www/seasonal_forecast/climate_futures/forecast/

Please follow the existing directory structure to make Alex’s processing easier. Email Alex (lenkoski@nr.no) and Ole (owul@norceresearch.no) to let them know that the forecast is complete.

3.2 Generating standard plots

Now the delivery to the Climate Futures team is complete, we will generate some plots and place them on the NS9039K website.

In order to produce ENSO and IOD index plots, we need up-to-date ERA5 data, so update this first:

cd /projects/NS9039K/data/external/reanalysis/ECMWF/ERA5/scripts
python download_era5_recent.py

This should download ERA5 data for 202101-Present and run a conversion to NetCDF using CDO. Check this downloaded correctly and converted to NetCDF:

ls /projects/NS9039K/data/external/reanalysis/ECMWF/ERA5/original/monthly_single_level/monthly_averaged_reanalysis/sst/

Then modify the two index plot scripts. You need to modify two input arguments at the top (marked clearly within the scripts). The scripts should output the ENSO and IOD plots to the correct location.

cd C3S-NorCPM-CF
vi enso_plot.py
python enso_plot.py
vi iod_plot.py
python iod_plot.py

Run the mean anomaly and quantile plotting scripts. Make sure to update the input and output locations and dates. Run twice, once for ‘global’ and once for ‘europe’:

python mean_anomaly_plots.py
python quantile_plots.py

This produces output plots in your specified directory.

/projects/NS9039K/www/seasonal_forecast/plots/

Quantile and mean anomaly plots are generated by comparing the forecast to pre-calculated mean and quantile values, these values exist in files in:

/projects/NS9039K/www/seasonal_forecast/thresholds/

In the normal seasonal forecast production, these do not have to be modified, but if there are changes to the data request (e.g. modification of the grid or addition of variables) then these will have to be regenerated using the scripts in the C3S-NorCPM-CF repository:

calculate_mean_thresholds.py
calculate_quantile_thresholds.py

3.3 Updating the website

You need to update the index file to display the new plots on the website.

vi /projects/NS9039K/www/seasonal_forecast/index.html

In this file, you must:

  1. Add the current date to the issueListQuantile

  2. Input the current date as the issueStringQuantile

  3. Add the current date to the issueListAnomaly

  4. Input the current date as the issueStringAnomaly

  5. Add the current date as the top option in the issueSelectQuantile form

<form>
<b> Issue Date </b>
<select id="issueSelectQuantile" onchange="updateIssueQuantile()" autocomplete="off">
<option selected> 2022-10 </option> <!-- Replace with most recent date -->
<option> 2022-09 </option> <!-- Add extra line for second most recent date -->
<option> 2022-08 </option>
<option> 2022-07 </option>
<option> 2022-06 </option>
<option> 2022-05 </option>
</select>
</form>

  1. Add the current date as the top option in the issueSelectAnomaly form

<form>
<b> Issue Date </b>
<select id="issueSelectAnomaly" onchange="updateIssueAnomaly()" autocomplete="off">
<option selected> 2022-10 </option> !-- Replace with most recent date -->
<option> 2022-09 </option> !-- Add extra line for second most recent date -->
<option> 2022-08 </option>
<option> 2022-07 </option>
</select>
</form>

  1. Add the date to issueSelectIOD and issueSelect ENSO as above.

Now the forecast production and processing is complete! Have a look at the forecasts here.