FDTD
Contents
Category
Physics, OpticsDescription
FDTD is a high performance 3D FDTD-method Maxwell solver for the design, analysis and optimization of nanophotonic devices, processes and materials.Available versions (Euler, old software stack)
| Legacy versions | Supported versions | New versions |
|---|---|---|
| 8.5.3 | 2019b, 8.11, 8.15 |
Please note that this page refers to installations from the old software stack. There are two software stacks on Euler. Newer versions of software are found in the new software stack.
Environment modules (Euler, old software stack)
| Version | Module load command | Additional modules loaded automatically |
|---|---|---|
| 8.5.3 | module load legacy open_mpi/1.4.5 fdtd/8.5.3 | |
| 2019b | module load open_mpi/1.6.5 fdtd/2019b matlab/R2017b | |
| 8.11 | module load open_mpi/1.6.5 fdtd/8.11 matlab/8.2 | |
| 8.15 | module load open_mpi/1.6.5 fdtd/8.15 matlab/8.2 |
Please note that this page refers to installations from the old software stack. There are two software stacks on Euler. Newer versions of software are found in the new software stack.
Initial setup
Please note that for using FDTD on the Euler cluster, there are some initialization steps that you need to do first.Prerequisites
FDTD is a commercial software, requires a license. In order to use it on Euler, you need to purchase a license through the IT shop of ETH to get the licensing information that is required to configure your account on Euler for FDTD. If you are using a Windows computer you also need to setup an X11 server. FDTD solutions has been tested with Cygwin/x (x11 and ssh packages required) whereas some problems occured with Xming. Therefore we recommend to use Cygwin/x as X11 server when using a Windows computer.
Preparing the environment
As you need to load 4 different modules to prepare the environment for running FDTD Solutions, we recommend to create an alias in your $HOME/.bash_profile file. For this, open the .bash_profile in a text editor and add the two following lines:
# fdtd modules alias fdtdm="module load new open_mpi/1.6.5 fdtd/8.11 matlab/8.2"
The changes will become active if you open a shell next time. If you want to use the alias already in the current shell session, then please type:
source $HOME/.bash_profile
Before you start an interactive FDTD session or submit a batch job, you have to type fdtdm to load all the required modules.
Setting up the license configuration
For setting up the license configuration, you need to login to the Euler cluster with X11 forwarding enabled. Then you need to make sure that the necessary module are loaded as described above. When starting the graphical user interface with the command fdtd-solutions, you will be asked to provide information about the license server. Please specify then the license information (hostname, port) that you were provided by IDES, when purchasing your FDTD Licenses.
This license information will be stored in a hidden file ($HOME/.flexlmrc) in your home directory on Euler. If you ever need to change the license settings, then open this file in a text editor and delete the line that starts with LUMERICL_LICENSE_FILE= and repeat the steps described above.Interactive session
In order to start an interactive FDTD session, you need to login with X11 forwarding enabled and load all the required modules. Afterwards you can start the interactive session with the commandfdtd-solutionsPlease note that this will check out a license and block it as long as the GUI is open. Therefore if you are no longer working with the interactive GUI, then please close it immediately.
How to submit a job
For submitting a FDTD job on Euler, you can use the lsfsubmit.sh script. You can Download the script from the link given above. Please save the script in your $HOME/bin directory and make sure that you set execution permission for your user account. There are 4 parameters that you need to specify in order to run a FDTD job on Euler. Short explanation of the parameters:[sfux@euler01 home]$ lsfsubmit.sh --help Usage: lsfsubmit.sh -n X -w XX:XX -mem X -inp inputfile Options: --help -h Display information of wrapper script -n Specify number of cores -w Specify maximal run time [hours:minutes] -mem Specify memory in MB per core -inp [inp_file] Use inp_file as input file -dryrun Display command, but don't execute
Regarding the nanowire.fsp example from FDTD solutions, the job could be submitted in the following way:
lsfsubmit.sh -n 4 -w 1:00 -mem 250 -inp nanowire.fsp
This would reserve 4 CPU cores for this job, specify a run time limit of 1 hour and reserve 4*250=1000 Megabyte of memory. Alternatively you can also directly submit jobs with the following command:
bsub [LSF options] mpirun fdtd-engine-ompi-lcl input_file.fspHere you need to replace input_file.fsp with the name of your input file and [LSF options] with LSF parameters for the resource requirements of the job. Please find a documentation about the parameters of bsub on the wiki page about the batch system.
Example
For the example simulation, we are using the nanowire.fsp input file, which is provided by lumerical solutions.[leonhard@euler01 ~]$ ls -ltr nanowire.fsp
-rw-r----- 1 leonhard T0000 202664975 Jun 10 2015 nanowire.fsp
[leonhard@euler01 ~]$ module load open_mpi/1.6.5 fdtd/8.11 matlab/8.2
[leonhard@euler01 ~]$ lsfsubmit.sh -n 4 -w 1:00 -mem 250 -inp nanowire.fsp
Job, requesting 4 CPU's submitted at Tue Oct 11 12:56:22 CEST 2016
FDTD job.
Job <30077428> is submitted to queue <normal.4h>.
[leonhard@euler01 ~]$ bjobs
JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME
30077428 leonhard PEND normal.4h euler01 *owire.fsp Oct 11 12:56
[leonhard@euler01 ~]$ bjobs
JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME
30077428 leonhard RUN normal.4h euler01 4*e1041 *owire.fsp Oct 11 12:56
[leonhard@euler01 ~]$ bjobs
No unfinished job found
[leonhard@euler01 ~]$ grep -A5 Overall nanowire_p0.log
Overall wall time measurements in seconds:
time to mesh and initialize: 0
time to run FDTD simulation: 2
time to finalize data and save to files: 2
FDTD solver speed: 51.1248 Mnodes/s
total FDTD solver speed on 4 processes: 204.499 Mnodes/s
[leonhard@euler01 ~]$
The lsf.o* log file contains information about the resource usage and the overall progress of the job, whereas nanowire_p0.log contains more detailed information about the simulation.