MENUAs the years go by, more and more engineers and scientists are working with files containing hundreds of millions of cells, with billions on the horizon. Tecplot 360 introduced a new data file format called SZL or subzone loadable (filename extension .szplt) to address the needs of customers who are visualizing large data files, improving interactive performance, and simultaneously reducing memory requirements.
When faced with the task of making large data files load faster, two possible solutions come to mind:
- Significantly increase the performance of the computer system, particularly its storage
- Figure out a way to load a lot less data for most operations
The first approach relies on the giants of the computer industry to continue advancing the state of the art. Unfortunately, this approach only gets us so far; the ever-increasing size of simulation data is clearly swamping even the fastest of today’s processors, memory, and storage when it comes time to visualize the results. Next year’s hardware will certainly be faster, but next year’s data will just as surely be even bigger. And continually chasing the absolutely fastest hardware is very expensive.
The subzone loadable format employs the second approach. In the past, Tecplot 360 has supported load-on-demand for several popular CFD file formats, which reduces perceived loading time by loading a given variable in a particular zone only when it is needed for a plot. If you never needed a particular variable, Tecplot 360 wouldn’t waste time or memory on it. If you did need all the data in the file displayed at once, you wouldn’t notice any benefit, but more common use cases saw a great improvement in performance.
Subzone loading is the next evolution of this concept. By dividing each zone into subzones and arranging the data for easy access by spatial location, we can substantially reduce the amount of data that needs to be read for any given operation and easily locate the data we do need. For example, rather than needing to read an entire variable into memory to slice into a volume zone, at a potential cost of several minutes, we can now read the only the cells the slice cuts through in a few seconds. As an added benefit, subzone loading requires a fraction of the memory required by traditional load-on-demand.
Unfortunately, it isn’t possible to use subzone loading with existing data files in industry-standard formats. Virtually without exception, these formats lay out zones and variables sequentially: first all the variables from zone 1 in order, then the variables from zone 2 in order, and so on. The data’s location in the file bears no real relation to where it is located spatially, and there is no way provided to easily locate data near given coordinates. Solving this problem necessitated the invention of the .szplt file format, which is the key to the performance and memory usage improvements of Tecplot 360.
Creating Subzone Loadable Data Files
The simplest way to create subzone loadable data files is to upgrade your solver to a version that can write them natively. For developers, this is simple if their solver is already writing .plt files using Tecplot’s TecIO library. A version of TecIO that writes .szplt files is now available, and developers merely need to use that version instead of the one they’re using now and make a few minor changes.
If your solver does not output data in Tecplot’s formats, or if you already have large data files in other formats that Tecplot 360 can read, you can convert them to .szplt files to enjoy the benefits of subzone loading.
To do this:
- Load the file in Tecplot 360 in the usual fashion.
Choose Load Data File(s) from the File menu. Then choose the existing file format of the data from the Files of Type menu in the Load Data Files dialog. Finally, select the file to be loaded and click Open.
If you wish to use options other than the defaults when loading the data file, mark the Advanced Options checkbox before clicking Open, then set the options in the dialog that appears next.
- Write the data back out of Tecplot 360 in .szplt format.
Choose Write Data File from the File Menu. Choose Tecplot Subzone Data Writer from the Files of Type menu in the Write Data File dialog. Finally, enter a name for the file and click Save.
Automatic Conversion of Data Files
Converting a large number of files manually is cumbersome and time-consuming. You may want to add conversion to .szplt format to your post-processing workflow, so that after your solver finishes processing a case, it is automatically converted and ready to view interactively in Tecplot 360.
Automating conversion of data files to .szplt format is best done by using Tecplot 360 in batch mode. In most cases, you will want to do the conversion on the cluster where the simulation is run, so you have enough memory to load the file (see “Memory Requirements” below).
There are three pieces to the solution:
- A layout that specifies the options for loading the data. The data file specified in the layout will be ignored when we later pass the name of the file on the command line, so there is no need to actually load one of your typical data files. You can use a smaller file as long as it is your usual format and uses the same options.
To create the layout, simply load a data file that is similar to the ones you will be converting later, then save a layout file. Choose Load Data File(s) from the File menu. Next, choose the data file format from the Files of Type menu in the Load Data Files dialog and mark the Advanced Options checkbox. Select the file to be opened, then click Open. In the next dialog, set the loading options. Finally, choose Save Layout As from the File menu and save the layout as convert2szplt.lay.
- A macro that contains the instruction to save the data in .szplt format. Paste the instructions below into a text editor and save the resulting file as convert2szplt.mcr:
#!MC 1410 $!EXTENDEDCOMMAND COMMANDPROCESSORID = 'Tecplot Subzone Data Tools' COMMAND = 'WRITEDATASET FILENAME="|DATASETFNAME|.szplt"'
- A short batch file or shell script that ties it all together. On Windows, this is:
tec360 -b -p convert2szplt.mcr convert2szplt.lay %1
On Mac OS X or Linux, this should instead be:
#!/bin/sh tec360 -b -p convert2szplt.mcr convert2szplt.lay $1
Save the file as convert2szplt.bat (Windows) or just convert2szplt (Mac/Linux) in a directory that is in your system path. On Mac OS X or Linux, also execute the following command to make the script executable:
chmod +x convert2szplt
To convert a single data file to .szplt format, you can now issue the following from a command prompt (or include it in the script that runs your solver, or configure it as a post-processing script in your solver):
convert2szplt /path/to/data/file.dat
The new file is named the same as the original file with .szplt added. You can create multiple sets of layout/macro/script files (under different names or in different directories) to handle multiple categories of files that you deal with on a regular basis.
Converting Multiple Files
If the data you are working with is in multiple files that need to be loaded separately, or uses a format that requires that multiple filenames be specified when loading the file, a different approach is required. An entire data set can be saved to a single .szplt file regardless of the number of files it was originally stored in. The only trick is getting Tecplot 360 to load the necessary files. The procedure we’ll use here is:
- In Tecplot 360, record a macro that loads the desired data and saves it as a .szplt file. To do this, begin recording (choose Scripting > Record Macro), name and save the macro file, record the loading of the data, and then click Stop Recording in the Macro Recorder window.
- Convert this macro to a “template” by replacing the filenames with placeholders, which can later be replaced with the names of the files you want to convert and the name under which to save them. Open the macro in a text editor, find all the filenames resulting from loading and saving your file, and replace the different filenames in the script with {1}, {2}, {3}, and so on.
- Create a Python script to accept filenames from the command line and plug them into a template, then tell Tecplot 360 to run the resulting script. Here’s the script:
#!/usr/bin/python
import sys, os
sys.argv.pop(0)
template = sys.argv[0]
macro = template.replace(".mcr", "_.mcr")
open(macro, "w").write(open(template).read().format(*sys.argv))
os.system("tec360 –b " + macro)
Save this script as macrotemplate.py and (on Linux and Mac OS X) perform chmod +x macrotemplate.py.
So if you have created a macro file template named convert3files.mcr, which loads three files and saves them as a single .szplt file, it should have four placeholders in it: one each for the names of the files to load, and one for the name of the file to save. You can thus invoke it using macrotemplate.py as follows:
macrotemplate.py convert3files.mcr file1.dat file2.dat file3.dat outputfile.szplt
The first argument to the script is the name of the macro template file to be used. The script substitutes each of the listed files into the macro replacing the tokens {1}, {2}, and so on (in this case up to {4}). The resulting macro is then written to the file convert3files_.mcr (note the underscore in the filename, which is added to prevent the original file from being overwritten). Finally, Tecplot 360 is invoked in batch mode and executes the macro, loading the files and saving the data in the .szplt file.
Memory Requirements
It is difficult to predict exactly how much memory (RAM) will be required to convert data to the subzone loadable format, as it varies depending on the characteristics of the data. Generally, however, the process requires substantially less memory than would be required to load the entire file and work with it interactively. Only one data variable needs to be loaded at a time, and the memory used by each variable can be reused by the next. For finite-element data, the X, Y, and Z variables and the connectivity data, which may be substantial, must remain loaded throughout the conversion. There is also some scratch memory overhead.
Once the data is converted to .szplt format, for most operations only small a fraction of this memory is needed on the workstation being used to work with the data interactively.