Retrying Failed File Transfer Attempts

When file transfer send attempts fail, CIC retries them automatically and on an extended basis should the retries also fail. You can also trigger retries manually.

Automatic and Extended Retries

Automatic Retry occurs immediately when a file transfer send attempt fails. 

Extended Retry is a continuation of Automatic Retry that occurs when the initial retry attempt fails. Subsequent retry attempts are made after 1 minute, 2 minutes, 4 minutes, 8, 16, 32, and 60 minutes, and every 60 minutes thereafter.

The Extended Retry cycle ends when all payload retries have either succeeded or been manually canceled and marked as “failed.”  See Canceling a payload.

How Automatic Retries Work

The following outlines how automatic retries fit into CIC processing.

1. A Data Flow runs, for example, started by a customer, automatically on a schedule, or triggered by an incoming request with AS2. 
2. A Job fails to send files to a target endpoint.
   • The failure results in a job ticket being created.
   • The failed Job is retried automatically.
     Automatic/extended retries do not create new jobs, but rather continue the job that failed.
3. If the initial retry succeeds: 
   • The file goes to its destination.
   • Job is updated in CIC Cockpit and Studio. See Understanding Retry Information on the Jobs List and Understanding Retry Information on the Job Details screen.
   • The Job ticket is updated as "resolved."
4. If the initial retry fails:
   • Extended retry attempts another retry after 1 min.
   • Then, another after 2, 4, 8, 16, 32, 60 minutes, and every 60 minutes thereafter.
   • If an extended retry succeeds, the original failed ticket is set to "resolved".

The extended retry cycle ends when all extended file transfer send attempts succeed or the payloads are manually marked as "failed".

Manual Retries

Manual Retry occurs when a user selects a failed job and clicks the Retry button.

Manual retry is also available at any point during the Extended Retry cycle. If an extended retry attempt is already active at the time a manual retry is requested, the manual retry request is, in essence, ignored. However, this does reset the retry cycle to the beginning.

Retrying Failed Jobs Manually

Sometimes when sending a job to its destination, a failure may occur due to permission or storage issues, etc. Once the issue gets corrected, you can quickly resend the job from exactly where it failed using the Retry Job button, which displays at the top right corner of the Job Detail page when manual retry is available.

Not all jobs can be retried with this option. However, the Retry Job button only appears when a job can be.

How Manual Retries Work

1. Select failed inbound or outbound job to retry.
2. Click the Retry Job button.
3. Refresh your browser.
4. View the results.

In the simple example below we see that a job has failed and an Issue has been created.

You can check the Payloads tab to see details of the issue, and also the Log tab to see the events, which in this example, shows a SEND_FAILURE.

After troubleshooting and correcting the problem, you would click the Retry Job button. This results in the job being queued for retry.  Important: At this time, users must refresh their browser in order to see the outcome of the retry.

Once complete (and issues addressed), the Job Detail page displays the job as successful. 

You can also confirm this in the Payloads and Log tabs (shown here). Note that the SEND_FAILURE, as well as MANUAL_RETRY, have been logged.

Resending failed Jobs that contain multiple payloads and job steps

Failed jobs that contain multiple payloads and multiple job steps can also be retried using the Retry Job button. In this example, two of the five job steps have failures. Once the issues have been resolved, clicking the button would retry the failed job steps. Note that not all jobs can be retried with this option. However, the Retry Job button only appears when a job can be.

A successful run of these job steps would then display.

 

Understanding Retry Information on the Jobs List

Jobs that fail and are subsequently retried are displayed on the Jobs List screen. They have a new icon and new status - Retrying. If a job runs and exhausts its initial command retries, the job enters into an extended retry mode and it is displayed with the Retrying icon. 

Like other jobs on the list, you can drill into a job with a status of Retrying to see it on the Jobs Details screen. 

In addition, the Retrying status is available for you to use as criteria for filtering the Jobs list.

Understanding Retry Information on the Job Details screen

When you click a Job on the Job List to “drill into’ it, the Job Details screen displays. 

When you click into a “retrying job” (like the one above), the Job Progress bar shows the Job as “retrying” and the Issues tab displays.

Payloads Tab

The Payloads tab shows the failed payload as retrying. Retrying status is displayed at the payload level to show which payload is currently being retried.

If you are using AS2 and have receipts configured, you can click the Receipt icon to display the Receipt dialog box.

Log Tab

Click the Log tab to display log entries related to this job.
For a failed payload being automatically retried, you see a SEND_FAILURE event followed immediately by a RETRYING event. See Job Events.

If that retry attempt fails, we again see a SEND_FAILURE event, but followed immediately by a DELAYING_BEFORE_RETRY event in accordance with the extended retry schedule (see above).

Number of retries is displayed at the top of the Job Details page.

When you click the Retries icon, you see a short history of your retries and when the next retry is scheduled.

Also, when you hover over the Retrying icon on the Payloads page, information about the next retry is displayed.

Canceling a payload

1. Select a payload (1) and then click the Cancel button (2).
   
2. Refresh the page to see that the Status has changed to failed and that processing continues on next payload.
   

If all payloads are manually canceled and marked as failed, the retry cycle will stop, and the job will enter a failed Status. The manual Retry button will no longer be available on the job itself because the failure will be considered intentional, but you can still retry individual canceled payloads to start a new retry cycle.

Retrying a canceled payload

Select a payload previously marked as “failed”(1) and click the Retry button (2).