FDTD

From ScientificComputing
Jump to: navigation, search

Please note that this application page is referring to the old CentOS software stack which is obsolete and does not work any more with the new Ubuntu setup. You can find an overview on the Ubuntu software stack on this wiki page.

Category

Physics, Optics

Description

FDTD is a high performance 3D FDTD-method Maxwell solver for the design, analysis and optimization of nanophotonic devices, processes and materials.

Available versions (Euler, old software stack)

Legacy versions Supported versions New versions
8.5.3 2019b, 8.11, 8.15

Please note that this page refers to installations from the old software stack. There are two software stacks on Euler. Newer versions of software are found in the new software stack.

Environment modules (Euler, old software stack)

Version Module load command Additional modules loaded automatically
8.5.3 module load legacy open_mpi/1.4.5 fdtd/8.5.3
2019b module load open_mpi/1.6.5 fdtd/2019b matlab/R2017b
8.11 module load open_mpi/1.6.5 fdtd/8.11 matlab/8.2
8.15 module load open_mpi/1.6.5 fdtd/8.15 matlab/8.2

Please note that this page refers to installations from the old software stack. There are two software stacks on Euler. Newer versions of software are found in the new software stack.

Initial setup

Please note that for using FDTD on the Euler cluster, there are some initialization steps that you need to do first.

Prerequisites

FDTD is a commercial software, requires a license. In order to use it on Euler, you need to purchase a license through the IT shop of ETH to get the licensing information that is required to configure your account on Euler for FDTD. If you are using a Windows computer you also need to setup an X11 server. FDTD solutions has been tested with Cygwin/x (x11 and ssh packages required) whereas some problems occured with Xming. Therefore we recommend to use Cygwin/x as X11 server when using a Windows computer.

Preparing the environment

As you need to load 4 different modules to prepare the environment for running FDTD Solutions, we recommend to create an alias in your $HOME/.bash_profile file. For this, open the .bash_profile in a text editor and add the two following lines:

# fdtd modules
alias fdtdm="module load new open_mpi/1.6.5 fdtd/8.11 matlab/8.2"

The changes will become active if you open a shell next time. If you want to use the alias already in the current shell session, then please type:

source $HOME/.bash_profile

Before you start an interactive FDTD session or submit a batch job, you have to type fdtdm to load all the required modules.

Setting up the license configuration

For setting up the license configuration, you need to login to the Euler cluster with X11 forwarding enabled. Then you need to make sure that the necessary module are loaded as described above. When starting the graphical user interface with the command fdtd-solutions, you will be asked to provide information about the license server. Please specify then the license information (hostname, port) that you were provided by IDES, when purchasing your FDTD Licenses.

This license information will be stored in a hidden file ($HOME/.flexlmrc) in your home directory on Euler. If you ever need to change the license settings, then open this file in a text editor and delete the line that starts with LUMERICL_LICENSE_FILE= and repeat the steps described above.

Interactive session

In order to start an interactive FDTD session, you need to login with X11 forwarding enabled and load all the required modules. Afterwards you can start the interactive session with the command
fdtd-solutions
Please note that this will check out a license and block it as long as the GUI is open. Therefore if you are no longer working with the interactive GUI, then please close it immediately.

How to submit a job

For submitting a FDTD job on Euler, you can use the lsfsubmit.sh script. You can Download the script from the link given above. Please save the script in your $HOME/bin directory and make sure that you set execution permission for your user account. There are 4 parameters that you need to specify in order to run a FDTD job on Euler. Short explanation of the parameters:
[sfux@euler01 home]$ lsfsubmit.sh --help
Usage:
lsfsubmit.sh -n X -w XX:XX -mem X -inp inputfile

Options:
--help -h               Display information of wrapper script
-n                      Specify number of cores
-w                      Specify maximal run time [hours:minutes]
-mem                    Specify memory in MB per core
-inp [inp_file]         Use inp_file as input file
-dryrun                 Display command, but don't execute

Regarding the nanowire.fsp example from FDTD solutions, the job could be submitted in the following way:

lsfsubmit.sh -n 4 -w 1:00 -mem 250 -inp nanowire.fsp

This would reserve 4 CPU cores for this job, specify a run time limit of 1 hour and reserve 4*250=1000 Megabyte of memory. Alternatively you can also directly submit jobs with the following command:

bsub [LSF options] mpirun fdtd-engine-ompi-lcl input_file.fsp
Here you need to replace input_file.fsp with the name of your input file and [LSF options] with LSF parameters for the resource requirements of the job. Please find a documentation about the parameters of bsub on the wiki page about the batch system.

Example

For the example simulation, we are using the nanowire.fsp input file, which is provided by lumerical solutions.
[leonhard@euler01 ~]$ ls -ltr nanowire.fsp
-rw-r----- 1 leonhard T0000 202664975 Jun 10  2015 nanowire.fsp
[leonhard@euler01 ~]$ module load open_mpi/1.6.5 fdtd/8.11 matlab/8.2
[leonhard@euler01 ~]$ lsfsubmit.sh -n 4 -w 1:00 -mem 250 -inp nanowire.fsp
Job, requesting 4 CPU's submitted at Tue Oct 11 12:56:22 CEST 2016
FDTD job.
Job <30077428> is submitted to queue <normal.4h>.
[leonhard@euler01 ~]$ bjobs
JOBID      USER     STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
30077428   leonhard PEND  normal.4h  euler01                 *owire.fsp Oct 11 12:56
[leonhard@euler01 ~]$ bjobs
JOBID      USER     STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
30077428   leonhard RUN   normal.4h  euler01     4*e1041     *owire.fsp Oct 11 12:56
[leonhard@euler01 ~]$ bjobs
No unfinished job found
[leonhard@euler01 ~]$ grep -A5 Overall nanowire_p0.log 
Overall wall time measurements in seconds:
    time to mesh and initialize: 0
    time to run FDTD simulation: 2
    time to finalize data and save to files: 2
    FDTD solver speed: 51.1248 Mnodes/s
    total FDTD solver speed on 4 processes: 204.499 Mnodes/s
[leonhard@euler01 ~]$ 
The lsf.o* log file contains information about the resource usage and the overall progress of the job, whereas nanowire_p0.log contains more detailed information about the simulation.

License information

Lumerical FDTD is a commercial software that requires a license. You can purchase the FDTD license from the IT shop.

Links

https://www.lumerical.com/tcad-products/fdtd/