The preprocessor in SMS does all the reading of the sms-files and processes them to form the job-file, manual-page or the file for XCdp/CDP for editing. It also does the interpretion of the SMSCMD and SMSKILL.
In version 4.4.2 the preprocessor was extended to handle new symbols. This is to aid use of perl, but can be useful for other things too.
Table of contents:
SMS-preprocessor was developed to be part of the SMS and is available in standard SMS distribution since SMS V4.2. It allows users to do some of the C-preprocessor like tasks, namely including files. It works on the same principle by looking at the first character on each line in sms-file (or sms-script. If that is found to be SMS-micro character (by default '%', see more in SMSMICRO) the line is for the preprocessor. If, however, the line starts with two of these characters, a single `%'-character is passed on to the next phase (to be used as a SMS-variable introducer.)
At the same time the handling of the manual pages was changed. From SMS V4.2 onwards manual pages are part of the sms-files. This is to ensure that the manual page will be updated when the script is being updated. Having them in different files was felt to be too dangerous; one might change the script but not the manual page. SMS-preprocessor can be used to extract the manual page from the script file to be viewed by XCdp/CDP.
Suites and families can also have manual pages. These are separate files placed in the suite/family directories.
Part of the code can be commented out, so that it is not part of the script anymore.
Currently there is no %if-statement in the preprocessor, this might be implemented in some future version. If-statements, however, can be handled by the shell running the script.
Another task that the preprocessor does is to substitute variables. When SMS is about to execute a task it reads the sms-file or sms-script and produces the job-file in which it replaces all the variables. There is a separate page on variables.
Here is a table of preprocessor symbols that SMS understands. Notice that some of them work in pairs.
| %include <filename> | Include the contents of file %SMSINCLUDE%/filename into the output |
|---|---|
| %include "filename" | Include the contents of file %SMSHOME%/%SUITE%/%FAMILY%/filename into the output |
| %include filename | Include the contents of file filename into the output. Notice that since the $CWD of SMS can be anywere, the only form that can be safely used must start with slash '/'. |
| %includenopp filename | Same as %include,
but the file is not interpreted at all. This allows to test the filename separately with ease. (Same three formats for filename as for plain %include.) |
| %comment | Remove all the lines from the output until a line with %end is found. |
| %manual | If creating a job-file remove all the lines from the output until a line with %end is found. If creating a manual page include all the lines until a line with %end is found. |
| %nopp | Stop the preprocessing until a line starting with %end is found. No interpretion of the text will be done (eg no variable substitutions) |
| Line is retained, if preprocessing is requested by XCdp/CDP. | |
| %end | End processing of %comment or %manual or |
| %smsmicro CHAR | Change the smsmicro character to the character given.
If set in an include file the effect is retained for the rest
of the job (or until set again.)
This does not change how SMSFETCH or SMSCMD work, they still use SMSMICRO |
| %export fmt NAMES | Export variables into the sms-job file. (Needs a bit more thinking/work) |
Notice that for %include if the filename starts with slash, '/'-character, no interpretion will be made. The full path name of the file will be used.
SMS does also look for the sms-files based on variable SMSFILES. This variable points to a directory where the original sms-files are. This is best explained by an example; when SMS is trying to send a task and tries to locate the sms-file for a task:
The first one is of course overkilling, but one might put all the files of all suites on a separate filesystem etc.
If the original SMSSCRIPT did not exist, SMS will check the directories for the job file in SMSHOME (SMSSCRIPT is derived from SMSHOME.). If a directory does not exist, SMS will create it. This helps to clean the old job-files and the output and makes the maintenance of the scripts easier. This will also guarantee that the output can be redirected into the file without the job creating the directory. (eg NQS option QSUB -ro, or when using redirection.)
The advantage of using SMSFILES is clear; you do not have to create and maintain link-jungle. Eg the model.sms above exists in 5 different families in ECMWS operational suite. The file is placed in a directory .../smsfiles/fc/ and used by nodes /o/00/fc/model, /o/06/fc/model etc. This trick works nicely as long as there are no other task named model in the same family.
Manual pages are used as comments to users. The text on manual pages is not copied into the job-file when SMS sends a task into execution.
Suites, families, tasks can have manual pages. (Actually events, meters etc can also have manual pages, but they can not be viewed using XCdp and their use s strongly discouraged since they might be discontinued in future releases of SMS.)
Manual pages for tasks are placed in SMSSCRIPT inside a pair of preprocessor lines like in the following extract:
%manual OPERATORS: If this task fails, set it complete and report and next working day METAPPS: Check something or do something clever! end ls -l pwd hostname %manual Rest of the manual page is placed here, closer to the code end
As above there can be multiple manual section in the same file. When viewed they are simply concatenated. This helps in maintaining the manual pages. When the script is changed, the manual should be changed at the same time.
Viewing manual page from the above sms-script would look something like
OPERATORS: If this task fails, set it complete and report
and next working day
METAPPS: Check something or do something clever!
Rest of the manual page is placed here, closer to the code
After %manual all preprocessor symbols are ignored until %end is found. Thus you can not use %comment - %end to un-comment manual pages. As from SMS V4.3 the manual pages may have include statements like in the following extract:
%manual
OPERATORS: If this task fails, set it complete and report
and next working day
METAPPS: Check something or do something clever!
%include <manual/foo.bar>
%end
ls -l
pwd
hostname
%manual
Rest of the manual page is placed here
%end
For example standard instructions for operators could be placed in a single file and then included in every task (like contact phone numbers etc.) How the include file is found is explained in Include files
For all other type of nodes the manual pages are in a separate files. For suites and families a separate man-page must be placed in the directories like the following example show (SMSHOME == /usr/local/home/emos_sms/def/)
For suites the manual page is in file:
%SMSHOME%/%SMSNAME%/%SUITE%.man eg for o-suite it is /usr/local/home/emos_sms/def/o/o.man
For families the manual page is in file:
Include files are used where same pieces of code would be inserted into multiple of files. This allows all files using that include file to be easily changed. A group of files may have their own include file; eg all the tasks in a archieving family, would include one common file for, let's say, the variable definitions needed. This makes the maintenance of the tasks much easier. Alternative would be to use some source code maintenance package, but then you need to create the files by running a job etc.
Like in C-preprocessor, SMS include files do nest. There is no limit of how many times they nest, except the number of files that can be open simultaneously (enough for any practical application.)
In the simplest case a SMS-file would have two include statement. One in the beginning and one in the end of the file. An example is given below. There is two extra lines apart from the lines needed for the task itself. This helps to understand the script since only lines needed for this task are visible. The SMS stuff is not visible.
When SMS needs to read an include-file it tries to locate them from the directory pointed to by variable SMSINCLUDE (unless full path name was given.) Typically this variable is set in the suite definition file at the same time as SMSFILES
Typical use of SMSFILES and SMSINCLUDE
set suite x
suite $SUITE
edit SMSFILES /home/ma/map/def/$SUITE/smsfiles
edit SMSINCLUDE /home/ma/map/def/$SUITE/include
edit SMSHOME /tmp/map/sms
Example of using include statements in sms-file %include <sms.h> do the steps for the task %include <end.h>
The start of all the tasks in your suite is something like that in the next extract. You declare the SMS-variables needed, make sure that any command failing will trigger smsabort(sms) and tell the SMS that the task is active by using smsinit(sms).
Just imagine if you would have a big suite (a few hundreds of tasks) and you needed to execute same command in each of them. Editing just a single file is somewhat easier than editing them all.
File sms.h
#!/bin/ksh
SMSNAME=%SMSNAME%
SMSNODE=%SMSNODE%
SMSPASS=%SMSPASS%
SMS_PROG=%SMS_PROG%
SMSTRYNO=%SMSTRYNO%
export SMSNAME SMSNODE SMSPASS SMSTRYNO SMS_PROG
ERROR() { echo ERROR ; $SMSBIN/smsabort; exit 1 ; }
trap ERROR 0
trap '{ echo "Killed by a signal"; ERROR ; }' 1 2 3 4 5 6 7 8 10 12 13 15
set -e
smsinit $$
The same applies to the end of the task. You want to tell the SMS that the task is complete by using smscomplete(sms) and un-trap the shell.
File end.h
smscomplete
trap 0
exit
Let's say that you now want to add job accounting information into all your tasks. Just add a line into sms.h
ja
and a line into end.h
ja -s
In more realistic suites you would need more that just a single include file in the beginning of the sms-file. Eg one to have common options for NQS, then a few lines for the NQS options that are unique for the job. Then maybe an include file if this is an experimental suite, and so on. There is less than ten different include files like that in ECMWF's operational suite.
There is three different formats for the include statement:
| %include <filename> | %SMSINCLUDE%/filename directory is searched for the filename. If that variable is not defined SMSHOME is used instead. This is the recommeded format for include. |
|---|---|
| %include "filename" | Included form the directory where the task itself is, like: /SMSHOME/SUITE/FAMILY/filename |
| %include filename | Included from the directory where the SMS is. Normally this is the same as SMSHOME for the super node. |
If the filename starts with slash, '/'-character, all three formats are equal; the full path name is used.
Notice that the following is ILLEGAL:
%include <%SUITE%/file.h> # is ILLEGAL!!!
Comments are enclosed between preprocessor lines %comment and %end. Those lines are ignored when the file is being processed, either for running the job or for the manual-page. You can not, however, comment-out a part of the manual page.
Since comments are processed before a jobs is created they can be placed anywhere in the script. Just remember that comments do not nest and that you can not use comments inside manual pages. (You can of course change the %manual to be %comment!)
Following extract is an example of comments:
mars << EOF RETRIEVE, PARAM=10U/10V,DATE=..., %comment temp mod by OP / 1.1.1999 BC TARGET="/xx/yy/zz", %end TARGET="zz", END EOF
And the following is he corresponding output when the comment is removed.
mars << EOF RETRIEVE, PARAM=10U/10V,DATE=..., TARGET="zz", END EOF
In version 4.4.2 the SMS preprocessor was extended to allow parts of the sms-script to be included as is of without being preprocessed. This was done mainly to make it easy to use languages like perl which use %-sign.
Preprocessing can be stopped in to ways
%nopp
echo "char like % can be safely used here"
date +%Y.%m.%d
%end
echo "otherwise we must write"
date +%%Y.%%m.%%d
In version 4.4 of SMS the preprocessor was extended to optionally excute a user command to read a file. This enables you to use databases or version control systems like SCCS, ClearCase or similar systems.
The fetch is activated by defining a variable SMSFETCH in the suite definition file. The contents of the variable define a command which should get the desired file.
Instead of opening the file SMS will execute a command and read the output of that command instead (UNIX popen(2) is used.)
Like SMSCMD, the variable SMSFETCH is scanned and a command is formed, except that an option and filename is to be added. The Option to be added is
Since your command is called for each file that would otherwise be opened it is expensive. For every sms-file there are a few include-files to be opened and this command is executed for each of them. Even worse: the way SMS reads the files is that it first scans them for variables, and then it reads the files again!
Example:
suite x
edit SMSHOME /tmp/map/sms
edit SMSINCLUDE /home/ma/map/sms/example/x
edit SMSFILES /home/ma/map/sms/example/x
family fe
edit SMSFETCH "/home/ma/map/sms/example/smsfetch -F %SMSFILES% -I %SMSINCLUDE%"
task d1
repeat date YMD 19980101 19980105
label info "empty"
endfamily
An example script can be used. Notice that the output from the script is still scanned by SMS. That means that if the output contains any sms-preprocessor characters thay are interpreted.
The variable SMSMICRO can be set to change the SMS micro-character. This effects both the commands and the interpretion of the sms-files. Default value (as set in config.h) is "%".
The following suite definition and the sms-file go together:
suite x
edit SMSMICRO "&"
family f
task t
File t.sms &include <sms.h> echo job here &include <end.h>
Another way of changing the micro-character is to set it up in the sms-script. It only effects the script interpretion not the commands SMSCMD or SMSKILL.
Let's imagine that we have a sms-file like
task.sms
%manual
OPERATORS: Set the task complete and report next day
%end
%include <sms.h>
echo do some work
sleep %SLEEPTIME%
echo end of job
%include <end.h>
and that we have the above files sms.h, end.h and that SLEEPTIME had value 60. After editing the job-file would look something like:
task.job1
#!/bin/ksh
SMSNAME=/suite/family/task
SMSNODE=localhost
SMSPASS=xYz12AbC
SMS_PROG=314159
SMSTRYNO=1
export SMSNAME SMSNODE SMSPASS SMSTRYNO SMS_PROG
ERROR() { echo ERROR ; $SMSBIN/smsabort; exit 1 ; }
trap ERROR 0
trap '{ echo "Killed by a signal"; ERROR ; }' 1 2 3 4 5 6 7 8 10 12 13 15
set -e
smsinit $$
echo do some work
sleep 60
echo end of job
smscomplete
trap 0
exit