« Back

Postprocessing on AWS – Part 2: Saving Cost and Time


Welcome back for the second post in a series about running Tecplot 360 on Amazon Web Services (AWS) cloud compute resources. In our first post we discussed how to set up your license server, spin up a compute node, and install Tecplot 360 on AWS. The goal of this second article is to highlight different ways to work in the cloud, each satisfying a different objective. At times you may be interested in cost efficiency, and at others in speed. We’ll also talk about how to access AWS in a way that imitates a typical on-premise HPC environment. Lastly, we’ll share a few best practices for working in the cloud to save you time, effort, and anxiety.

Astrid Walle

Astrid Walle, the author of this article, is a mechanical engineer with a PhD in CFD and more than a decade of experience in applied fluid mechanics. She has held several positions in gas turbine aeromechanics, R&D and AI development at Siemens Energy, Vattenfall SE and Rolls Royce. As a recognized industry expert she has recently taken on the challenge of starting her own business, CFD Solutions. As a freelancer, Astrid is following her professional determination to combine numerical simulation and data analytics. 

Prerequisites

There are some prerequisites, which are assumed to be in place and won’t be discussed in detail in this article but are covered in the previous blog. These are:

  • You have an AWS Account.
  • You set up a ssh key pair to connect to virtual machines from your local one.
  • You have installed the AWS CLI on your local machine.
  • Your files for postprocessing are located in a S3 bucket.
  • You have a Tecplot 360 network license, and the license server is set-up and running.

This article focuses on the setup with Linux OS, the relevant links for Windows users are provided where possible.

Creating a Visualization Node on AWS from the Command Line

If you have the need for more resources in terms of memory or CPU power for your postprocessing from time to time, then this section is for you. This text guides you through creating an AWS EC2 instance and accessing this remote machine via remote desktop virtualization without the need to log onto the AWS console. In combination with the following section, which describes how to speed the set-up procedure, this enables you to get the resources you need to get your postprocessing done fast and with a great user experience. So, all we need is a terminal with AWS CLI installed.

Considerations When Configuring a Visualization Node

Before creating your visualization node you’ll need to make some decisions, such as:

  • What Amazon Machine Image (AMI, aka OS) to use? Linux systems generally have better file system performance than Windows systems.
  • What Amazon Instance Type to use? This defines hardware resources such as CPU, GPU, and available RAM.
  • How much storage is needed to host the data for post-processing?

Amazon Machine Image (AMI)

In the last blog we used the AWS parallel cluster service, as this comes along with Nice DCV remote desktop virtualization preinstalled. But now there is also the possibility to start a single EC2 instance, which fits your resource needs exactly and uses an Amazon Machine Image (AMI) with Nice DCV instead. These AMIs can be found in the AWS marketplace. Depending on the operating system you prefer or need for your applications, there are various images from different vendors. You can search for your requirements like OS and software and screen the resulting images.

For Tecplot 360, Amazon Linux works well. So, for this tutorial we can search for “Nice DCV” and select the first result with Amazon Linux and no additional costs. In order to get the ID of this AMI, we have to navigate to the subscription (Figure 1) and the configuration (Figure 2). For region eu-west-1 we get this AMI ID ami-0547e6987ff6a09e6. Please note, that the image ID is region specific, so you might need to adapt it.

Figure 1. AMI Subscription

Figure 1. AMI Subscription

 

Figure 2. AMI Configuration

 

Instance Type

Besides the image, you also need to decide which instance type you want to use. For that you must consider the requirements in terms of CPU/GPU/memory defined by the application and the amount of data. Here you get an overview of all provided instance types and the recommended applications. For Nice DCV it is recommended to use the g4dn.xlarge instance which has a NVIDIA T4 GPU onboard.

Tecplot 360 also benefits from this graphics acceleration. Now you just need to decide whether you are fine with the smallest instance of the g4dn family, or if need more cores or memory. Here we will select the g4dn.xlarge instance with 4 virtual CPUs and 16 GB of memory.

And just in case you find out that you should have selected another instance type, you can always stop the instance, change the instance type and restart. It might happen that you need to request an increase of your AWS quota to enable the g4dn instances for your account. This usually takes one day. Nevertheless, you can continue with this tutorial for now and use a c4 instance instead. The remote desktop experience won’t be as good as with the g4dn instance, but for testing the procedure it is absolutely sufficient.

Storage

The default g4dn.xlarge instance comes with 8 GB root base storage and additional 125 GB NVMe SSD. As we want to use the instance as a remote desktop, with the possibility of turning it on and off whenever needed, we will ignore the NVMe SSD and adapt the root base storage to our needs. Hence you have to estimate how much storage you will need. But don’t worry, in case you miscalculated, you always can increase the root base storage and extend the filesystem afterwards or even easier add an additional block storage to your instance.

Security and Management

When creating an instance on AWS you’ll also need to understand how to ensure proper access and security to the machine, since it will contain potentially proprietary information. This is done through IAM (Identity Access Management) and security groups.

The tagging system is also important to help organize your AWS resources.

IAM Role

Our EC2 instance needs access to S3 storage. For speeding up the setup procedure, we will store some installation files on S3. We also need to fetch the files, which we want to process with Tecplot 360 from S3. Furthermore, we need to grab a Nice DCV license from S3 for our remote desktop visualization. So, you need to create a so-called Identity Access Management role for enabling instances to access S3. You can find instructions on how to do that here. For the ec2 run-instances command we will provide the name of this IAM role with the option –iam-instance-profile.

Security Groups

With security groups you control inbound and outbound traffic for your EC2 instance. Here we will create two security groups. The security group #1 is dedicated for communication with the Tecplot 360 license server, if this server is also running on an AWS EC2 instance. This security group allows all inbound and outbound traffic within this security group. For details about the set-up procedure, please check out the previous blog. The second security group has the following rules:

  • Inbound traffic on Port 22 from your local IP to enable ssh access.
  • Inbound traffic on Port 8443 from your local IP to access the Nice DCV remote desktop session.
  • If your license server is running on an on-premise server, you need to allow traffic from that IP.
  • All outbound traffic to enable web browsing.

When launching the instance later, we need to add these two security groups to our instance. This will allow all traffic listed above and the communication between our instance and the Tecplot license server.

Key Pair

For establishing a ssh connection to your EC2 instance you need to have a key pair for the desired region set up in your account.

Tags

To make our instance searchable and for easier accounting of the used resources, we will also add tags.

Using Remote Desktop Visualization Without the AWS Console

In this section we will follow the procedure for launching an EC2 instance with AWS CLI from our local terminal as described here, enriched with some additional options. All the following terminal commands can be replaced by interactions with the AWS console web interface. But the goal of this guide is to minimize the effort by accelerating and automating the setup procedure, so we will just use our local terminal. Please notice that all of the commands below were run in a macOS Linux shell with AWS CLI version 2.1.11. For other OS and other AWS CLI versions please refer to the documentation to check the correct syntax.

Start the Instance

The command we have to type into our local terminal for launching the instance is as follows (you CAN replace and alter all options, but you HAVE to replace the red strings with the values applicable to your account. And please make sure that image_id and region are compatible as described above). Using the tag-specifications and providing a “Name” key has the advantage that this key value will also be displayed in the AWS console, so we can easily identify our machine in the web interface.

aws ec2 run-instances \
        --image-id ami-0547e6987ff6a09e6 \
        --count 1 \
        --instance-type g4dn.xlarge \ 
        --key-name MyKeyPair \
        --security-group-ids SG1 SG2\
        --iam-instance-profile  Name=S3_access_for_EC2 
        --block-device-mappings  'DeviceName=ROOT_DEVICE, Ebs={ VolumeSize=50,VolumeType=gp2,DeleteOnTermination=false}' \ 
        --tag-specifications  'ResourceType=instance,Tags=[{Key=Name,Value=tecplot-dcv}]' \ 
        --region eu-west-1

Now your AWS instance is running. Next, we will log onto that machine via ssh to launch Nice DCV. Afterwards we can access our remote desktop session and do all necessary installations.

Connect to Your New Instance Using ssh

At first you need the public IP address of your instance, with which you can establish a ssh connection. Therefore, we can use the AWS CLI describe-instances command and write the output into a variable.

instance=$(
aws ec2 describe-instances \
        --filters Name=tag:Name, Values=tecplot-dcv \ 
        --region eu-west-1 \ 
        --query 'Reservations[*].Instances[*].[PublicDnsName]' \ 
        --output text
)

Now you can connect to the machine via ssh with username ec2-user:

ssh ec2-user@"${instance}"

Start NiceDCV and Access Your Instance Via a Web Browser

Now you are logged on your remote machine and for starting the remote desktop session, you just need to execute the following two commands. The first one is for setting a password for our user ec2-user and the second command is for starting the remote desktop session. Afterwards you can check if the session creation was successful and logout.

sudo passwd ec2-user 
dcv create-session session1
dcv list-sessions
logout

You can either use the browser or the native dcv client to connect to your remote session. Type the following command into your terminal, then you can copy and paste the URL into the browser.

echo https://$instance:8443

Use ec2-user as username and the password you just created for login (Figure 3) and now you can access the AWS instance via your remote desktop in the browser (Figure 4). For more information about the DCV functionalities, please check out this guide.

Figure 3: Nice DCV Login Screen

Figure 4: Remote Desktop of our AWS instance


Speeding Up the Setup Procedure for Your Virtual Machine

Now that you can access your instance via ssh and remote desktop, you can run all installations you need. In this tutorial we will do the first-time installations for

  • Firefox browser
  • Gedit editor (or your favorite text editor)
  • Tecplot 360

and while we are doing that, we prepare a script, with which we can automate the installation procedure for our next instance launches.

Installing Firefox

At first, we start with the Firefox browser and using this description, we enter the following commands into the terminal in our remote desktop session:

cd /usr/local 
sudo wget http://ftp.mozilla.org/pub/firefox/releases/83.0/linux-x86_64/en-US/firefox-83.0.tar.bz2
tar xvjf firefox-83.0.tar.bz2
sudo ln -s /usr/local/firefox/firefox /usr/bin/firefox

Installing Tecplot 360

Now we will install Tecplot 360 and we can do this via the terminal as described in the last blog or we can download the software from MyTecplot and install it manually. After the successful installation we will store Tecplot 360 in a S3 bucket to retrieve it superfast for our next instance launches. Thanks to the IAM role, which we assigned to our instance, we don’t have to do anything else to enable the S3 access. So, for packing the Tecplot 360 files and copying it to your S3 bucket, type:

tar czvf tecplot360ex2020r2.tgz tecplot360ex2020r2 
aws s3 cp tecplot360ex2020r2.tgz s3://yourbucket/tecplot360ex2020r2.tgz

Automating the Installation of Our Tools

To our script for automating the procedure we have to add the lines for copying the zipped archive from S3 to our instance and unpacking it. Also, we add one line for adding the Tecplot 360 executable to the $PATH system variable. By combining these commands for Firefox, Gedit, Tecplot 360 and additional tools you need into one script, saving it on our local machine and calling it when we launch a new instance with the following option, we have automated and accelerated our set up procedure.

aws ec2 run-instances \ 
        --…
        --user-data file://post_install_script_tecplot.txt

The complete post install script with all commands described in this article is available on GitHub and the combined command for starting the instance is as follows:

aws ec2 run-instances \ 
        --image-id ami-0547e6987ff6a09e6 \ 
        --count  1 \ 
        --instance-type  g4dn.xlarge \ 
        --key-name  MyKeyPair \ 
        --security-group-ids SG1 SG2 SG3\ 
        --iam-instance-profile Name=S3_access_for_EC2 \ 
        --block-device-mappings  'DeviceName=/dev/xvda,Ebs={VolumeSize=50,VolumeType=gp2,DeleteOnTermination=false}' \ 
        --tag-specifications  'ResourceType=instance,Tags=[{Key=Name,Value=Tecplot-dcv}]' \ 
        --user-data file:///PATH_TO_FILE/post_install_script_tecplot.txt \
        --region eu-west-1

Saving Costs: Stop and Terminate Instances When Not Needed

To avoid unnecessary expenses, you should terminate or at least stop your instances when you don’t need them. Stopping the instance has the advantage that you can restart it anytime and it is in the same state as before, meaning all your installations, data and settings are still there. But you will be charged for the storage, which is needed to keep the snapshot of your instance. So, stopping and starting your instance is recommended when you have to leave your desk, but haven’t finished your project, so you haven’t moved your postprocessed data from the instance to S3 yet. But when you have to pause your work for a longer period of time, then you should consider terminating it and launching a new one at a later point in time. Of course, only after you saved all your data to S3. And of course, these considerations also depend on your budget and the chosen instance type.

Sometimes you have several instances launched and have lost track of which ones are still running. With the following command you can check for running and stopped instances:

aws ec2 describe-instances \
        --filters  Name=instance-state-name,Values=running \ 
        --query  'Reservations[*].Instances[*].[InstanceID, Tags[?Key==`Name`]|[0].Value]' \ 
        --region  eu-west-1 \ 
        --output  text

Now you can call stop and start or terminate commands with the resulting instance ID’s.

aws ec2 stop-instances \ 
        --instance-ids Your-Instance-ID
aws ec2 start-instances \
        --instance-ids Your-Instance-ID
aws ec2 terminate-instances \
        --instance-ids Your-Instance-ID

The start-instances command can be used for a previously stopped instance. When you terminated your instance and you want to have a new one, then you need to launch a new instance with the previously described run-instances command.

Tagging your instances is also a good idea, if you want to make sure that you don’t terminate an instance by mistake. So, assuming that you have started a postprocessing script, which will run for a long period of time, you can give your instance an extra tag. You can do this also after you’ve launched the instance and started your work. First get the instance ID with the command described above and then run the following command.

aws ec2 create-tags \ 
        --resources Your-Instance-ID \
        --tags Key=longrun,Value=yes \
        --region  eu-west-1

Now you can modify the command for listing all running instances and get all instances, which are supposed to stay running by adding this option:

 aws ec2 describe-instances \
         …
         --filters  Name=tag:longrun,Values=yes \

In the next article we will take a closer look at these topics, which can further improve your cloud experience:

  • Data Management
  • Batch Processing
  • Parallel Postprocessing

Try Tecplot 360 for Free

Free Trial