Step 1) Click Utilities and then Media Types
Step 2) Create a new media type for the new format you plan to offer. Use existing media types as reference.
Step 3) Click Plug-Ins and then "Edit" for the Video Transcoder.
Step 4) Add a new entry and specify the settings for your new encoding output format. You may specify the encoder source, ourput media type, audio and video bitrates and codecs and number of clips or clip length. Use existing transcoding formats as reference.
Once this has been done, the new encoding type will show as a checkbox on the Add/Edit Set and batch utilities pages.