Workload Automation – Part 4: Pull Files

This blog in the series will look at how to use the TWS built-in File Transfer advanced job type to pull files from a 3rd party site. One of the problems with this type of “pull” transfer is that you can’t be certain that the expected file is actually sitting there on the remote system waiting to be “pulled”. For this reason, there is usually a high chance of failure due to the file not being present.

Our objective is to automate as much as it practical the pull” of the remote file and the recovery/retry processing that TWS can perform. If the recovery/retry still fails after several attempts, a TWS event rule can be used to automatically alert the support teams of the failure and/or raise a ticket with your problem management system.

The built-in File Transfer advanced job type supports different types of file transfer protocol. These being FTP, FTPS, FTPES, SSH and Windows. Alternatively, you can specify Auto and TWS will determine the best protocol to use for the transfer of the file. Whichever protocol is selected (or defaulted) to the remote system must support the same protocol.

For a download transfer, the remote system hosts the file that is to be transferred to the local system (where the dynamic agent is running). For an upload transfer, the file is transferred from the local system (where the dynamic agent is running) to the remote system.


TWS supports specifying credentials using a username and password or using certificate and passphrase. Note that a passphrase is required when using certificates. A password is optional for the remote user when using certificates. Test the connection to the remote server by clicking the Test Connection button.

Pull a sample file

To set up a job to pull a file from a remote server, configure the appropriate protocol and credentials as mentioned earlier. Complete the section defining the transfer type (Download or Upload) and the server, remote file and local file. For the example below, we will pull a file from a test IWS 9.4 system using SSH.

Define the credentials used to access the remote system -the example uses a username “orb” with the password. Define the credentials to access the local system where the dynamic agent is installed and has write access to the local file path on that system – the example uses the TWS instance owner and password.

Ensure the connection to the remote server is successful by clicking the Test Connection button.

Recovery Job

Set up a recovery job for the File Transfer job that we just created. The recovery job doesn’t do a great deal other than sleep for a specific period of time before exiting so that the file transfer job can then automatically rerun for another attempt at pulling the remote file.

As the file transfer job executes an advanced job type on a dynamic agent or pool, the recovery job will also execute on the same workstation or pool and be set up as an advanced job executable type as shown below. This uses an inline script to set a random wait time between 250-300 seconds between retries of the file transfer. You could of course just execute the sleep command with a fixed wait time instead.

Add the retry job as a recovery job to original file transfer job as shown below

Note the above screenshot is from TWS 9.4 Fix Pack 1 which has additional options to automatically perform a rerun after a specified delay. This can be used as an alternative to the recovery job which is the only option available for earlier TWS versions.

Test the solution

Before testing the solution, check that the remote file doesn’t exist so that we can force a “typical” error. On the remote system the file doesn’t exist as shown below

When we run the file transfer job it understandably goes to error and the recovery job starts execution as shown below

The job will continue to go to error and retry the file transfer as shown above until either the remote file becomes available or the job is killed/cancelled. To do the latter, an event rule can be setup up to alert the support staff that the file transfer did not work and/or open a ticket in the problem management system. A later blog will look more closely at these options.

Now if we create the missing remote file the next retry should complete successfully as shown below

Additional logic could be added to the above retry process to limit the number of retries for example. To make the process more flexible TWS variables can be used to substitute values for the remote server, the file name to be transferred, etc.

The next blog in the series will look at TWS variables and their sometimes misunderstood usage within TWS.

Views: 78