On the triangulation dialog there is a Use Boundary Zone(s) checkbox which allows you to pick a zone to use as a boundary. This will prevent the triangulation from crossing that boundary. How can you make a zone that will work as a boundary?
1) Start by loading your source data.
2) Go to Data > Create Zone > Rectangular and create a zone that encompasses the source data points (should be set to do this by default)
3) Go to Data > Extract > Points from Polyline, left click on the nodes which will make up the desired boundary and hit the ESC key when you are done creating it.
4) Choose to extract the data to a zone
5) Go to Data > Triangulate, click “Use Boundary Zones” and choose your new zone as the boundary zone.
6) Choose the source zone as usual and click Compute.
Alternatively, you could create a more precise boundary outside of Tecplot and load it. Let me know if you have any questions.