Working with Data Flows in CIC Express

A Data Flow connects two Endpoints so that data can be transferred between them in a single direction. For more details on Data Flows, see What is a Data Flow?

This article contains information about how you can work with Data Flows in CIC. 

• Understanding the Data Flow Page
• Creating a Data Flow
• Editing a Data Flow
• Removing a Data Flow
• Disabling a Data Flow
• Filtering Source folder
• Advanced Commands

Understanding the Data Flow Page

The Data Flow page is displayed when you click Integrations > Data Flows.

This page displays information about each of the Data Flows in your CIC system, including the following:

Creating a Data Flow

1. Go to the Integrations > Data Flows page and click the Add Data Flow button.
   The New Data Flow page displays.
   
2. Give your data flow a name to identify it throughout the system.
3. Select a From Endpoint. This will be your source Endpoint.
4. For File System, FTP or SFTP Endpoints, specify a folder on the Endpoint to be the source of the transfer. You can use the Endpoint's default folder or specify a subfolder.
   
   • To use the default folder, simply click Select on the top right of the From Endpoint panel.
     
   • To use a subfolder :
     • For a File System Endpoint, you can either browse the Endpoint’s file system to select the subfolder manually or click the Edit icon and type in the path to the subfolder. You do not need to precede the path with a forward slash(/). For example, you can enter /bin/documents or bin/documents as your path.

       On Windows, you can type a UNC path to specify network file shares, but you cannot browse to them. 

       Once you have selected a folder or subfolder, click Go to confirm your selection.

       

       Then, click Select to select the Endpoint.

   Note: With a File System Endpoint, by default the transfer will include all folders and files below the Access Point root directory. By selecting a subfolder you restrict the scope of the transfer to the subfolder and its children. You can further restrict the scope by specifying a filter pattern. See Filtering Source Folder below.

5. For FTP and SFTP Endpoints, click Select and then type in the path to the subfolder. There is no option to browse for a folder.
   
6. Specify any advanced options if available for your Endpoint. Not all Endpoint types offer advanced options. See Advanced Options below.
7. Select an Endpoint from the To Endpoints list. This will be your destination Endpoint. The process for selecting the Endpoint and folder is the same as described in steps 2 and 3 above.
8. (Optional) For Data Flows where the source Endpoint is a File System, SFTP, or FTP(S) Endpoint, you can run the Data Flow manually or schedule the Data Flow to run on a recurring basis. For more details, see Scheduled in Running a Data Flow in CIC Express.
9. Click Save to save the Data Flow.

Editing a Data Flow

1. Go to Integrations > Data Flows. The Data Flows page displays.
2. In the list, click the Data Flow you want to edit.
3. For either Endpoint in the Data Flow, click Unselect and then edit the path or commands.

Removing a Data Flow

1. Go to Integrations > Data Flows.
2. Select the check box for the Data Flow you want to remove.
   You can select more than one Data Flow from the list.
3. Click the Trash Can icon in the upper right part of the page and then click Delete to confirm that you want to remove the Data Flow.

Disabling a Data Flow

When a Data Flow is disabled, it is greyed out on the Data Flows page.

You cannot disable a Data Flow directly. Instead, to disable a Data Flow, you disable one of its Endpoints.

Note: When you disable an Endpoint, all the Data Flows that use that Endpoint are also disabled.

See Enabling and Disabling Endpoints in the Managing Endpoints in CIC Express article.

Advanced Options

Filtering Source Folder

For File System Endpoints, once you have selected your source folder and clicked the Select button, you have the option to filter the source folder for a subset of files. This results in the Data Flow only sending the files that match the filter.

To specify a filter, click the filter button.

This displays a field where you can enter one or more filters. Enter one or more filters to apply in this field. Only files matching the pattern you enter are transferred for a given run of the Data Flow.

Filter Matching Rules and Examples

The following are the pattern matching rules for filtering path and file names.

• The * character matches zero or more characters of a name without crossing directory boundaries.

• The ** characters matches zero or more characters crossing directory boundaries.

• The ? character matches exactly one character of a name.

• The backslash character (\) is used to escape characters that would otherwise be interpreted as special characters. The expression \\ matches a single backslash and "\{" matches a left brace for example.

• The [ ] characters are a bracket expression that match a single character of a name out of a set of characters. For example, [abc] matches "a", "b", or "c". You can use the hyphen (-) to specify a range. For example, [a-z] specifies a range that matches from "a" to "z" (inclusive). You can mix these forms, so that [abce-g] matches "a", "b", "c", "e", "f" or "g".

• If the character after the [ is a !, it is used for negation. For example, [!a-c] matches any character except "a", "b", or "c".

  Within a bracket expression the *, ? and \ characters match themselves. The (-) character matches itself if it is the first character within the brackets, or the first character after the ! if negating.

• The { } characters are a group of subpatterns, where the group matches if any subpattern in the group matches. The "," character is used to separate the subpatterns. Groups cannot be nested.

• Leading period/dot characters in file name are treated as regular characters. For example, the "*" pattern matches file name ".login". 

• All other characters match themselves.

Here are some examples of common filters:

Advanced Commands

If your source Endpoint is FTP or SFTP, once you have selected your source Endpoint, you can enter advanced commands in the From Endpoint panel as shown below.

See Command Reference for a list of supported commands. When you specify advanced commands, you can use macro variables, wildcards, and regular expressions as described in Using Macro Variables in CIC and Using Wildcards and Regular Expressions in CIC, respectively.

IMPORTANT: Any Advanced Commands specified for the Data Flow override the Move action of the Data Flow. This means if you specify any advanced commands and you want to move data from the source Endpoint to the destination Endpoint, you must include commands to do that.