Working with Data Flows

A Data Flow connects two Endpoints so that data can be transferred between them in a single direction. For more details on Data Flows, see What is a Data Flow?

You can use File System Endpoints as both source and destination in a single Data Flow regardless of whether the Endpoints reside on the same or different Access Points. This means you can now support multiple units or independent brands with common customers by transferring files between endpoints all within your CIC environment, allowing you to offer common services for joint customers between individual brands and subsidiaries. See Setting up Agent-to-Agent Transfers.

This article contains information about how you can work with Data Flows in CIC.

Understanding the Data Flow Page
Creating a Data Flow
Working with folders in Source Endpoints
Editing a Data Flow
Removing a Data Flow
Disabling a Data Flow
Filtering Source folder
Advanced Commands

Understanding the Data Flow Page

The Data Flow page is displayed when you click Integrations > Data Flows.

This page displays information about each of the Data Flows in your CIC system, including the following:

Column	Description
Name	The name of the Data Flow. Used to identify the Data Flow throughout your CIC system.
From	The source Endpoint for the Data Flow.
To	The destination Endpoint for the Data Flow.
Schedule	The schedule the Data Flow runs on, if any exists.
Created by	The person who created the Data Flow.
Modified	The date and time the Data Flow was last modified.

Filtering the Data Flow list

You can control what is displayed on the list of Data Flows through filtering available on the left side of the Data Flow List page. When you specify a value for one or more of the filter attributes, the Data Flow list is updated accordingly.

Note: Filters you apply are retained when you leave the Data Flow List page and subsequently return to it.

As you type in any of the filter fields (except either of the Endpoint Type fields, where you must choose a value), CIC displays any values that match the string you type and you can select the value you want at that point.

Filters currently applied are displayed at the top of the list. You can click the X on any of these to remove the filter from the list or you click Clear All to remove all filters.

You can click the Hide/Unhide button to toggle the Filter by panel.

Creating a Data Flow

Go to the Integrations > Data Flows page and click the Add Data Flow button.
The New Data Flow page displays.
Give your data flow a name to identify it throughout the system.
Select a From Endpoint. This will be your source Endpoint.
For File System, FTP, SFTP, S3, GCS, AzureBlob, and Partner Mailbox Endpoints, you can specify a folder on the Endpoint to be the source of the transfer. See Working with folders in Endpoints.

For Transformation Endpoints, select the Event (object created in CIC Studio). Any deployed Event in your workspace will be available for selection.
Specify any advanced options if available for your Endpoint. Not all Endpoint types offer advanced options. See Advanced Options below.
Select an Endpoint from the To Endpoints list. This will be your destination Endpoint. The process for selecting the Endpoint and folder is the same as described in step 3 above.
(Optional) For Data Flows where the source Endpoint type is File System, SFTP, FTP(S), S3, AzureBlob, Email, or OFTP(S), you can run the Data Flow manually or schedule the Data Flow to run on a recurring basis. For more details, see Scheduled in Running a Data Flow.
Click Save to save the Data Flow.

Working with folders in Source Endpoints

For File System, FTP, SFTP, S3, GCS, AzureBlob, and Partner Mailbox Endpoints, you can specify a folder on the Endpoint to be the source of the transfer. You can use the Endpoint's default folder or specify a sub-folder.

Note: By default, all Data Flows use non-recursive transfers. However, recursion is supported by File System, FTP, SFTP, GCS, S3, AzureBlob, and Partner Mailbox Endpoint types when specified as the Source Endpoint. See Controlling Recursion for a File System Endpoint. You can further restrict the scope by specifying a filter pattern. See Filtering Source Folder.

Selecting the default folder

Click Select on the top right of the From Endpoint panel.

Selecting a subfolder

Either browse the Endpoint’s file system to select the subfolder manually or click the Edit icon and type in the path to the subfolder. You do not need to precede the path with a forward slash(/). For example, you can enter /bin/documents or bin/documents as your path.

Once you have selected a folder or subfolder, click Go to confirm your selection.

Then, click Select to select the subfolder.

Selecting subfolders for Partner Mailbox Endpoints

For Partner Mailbox Endpoints, you select folders and subfolders using checkboxes.

Note: Subfolders must exist. They cannot be created from the Data Flow edit page. See Adding Folders and Subfolders in Partner Mailbox Endpoint.

When a Partner Mailbox is the source Endpoint and a Transformation Endpoint is the destination, information about the files' location is passed along as metadata to the Transformation Endpoint and thus to the Event selected for the Transformation Endpoint. The location metadata is then consumed by the Integration Engine to allow processing specific to the files being processed based on their locations.

Note: Using subfolders in this manner requires some configuration in CIC Studio. See Draft- Configuring Subfolders for Partner Mailbox Endpoints in Data Flows with Transformation Endpoints.

Editing a Data Flow

Go to Integrations > Data Flows. The Data Flows page displays.
In the list, click the Data Flow you want to edit.
For either Endpoint in the Data Flow, click Unselect and then edit the path or commands.

Removing a Data Flow

Go to Integrations > Data Flows. The Data Flows page displays.
Select the check box for the Data Flow you want to remove. You can select more than one Data Flow from the list.
Click the Trash Can icon in the upper right part of the page and then click Delete to confirm that you want to remove the Data Flow.

Disabling a Data Flow

When a Data Flow is disabled, it is greyed out on the Data Flows page.

You cannot disable a Data Flow directly. Instead, to disable a Data Flow, you disable one of its Endpoints.

Note: When you disable an Endpoint, all the Data Flows that use that Endpoint are also disabled.

See Enabling and Disabling Endpoints in the Managing Endpoints article.

Advanced Commands

CIC offers two types of advanced commands you can invoke during and after file transfers:

Override Default Receive/Send Method
Post-Transfer Commands

Select the Advanced Commands toggle to enable the Advanced Commands.

Override Default Receive/Send Method

Override Default Methods are used at transfer time and can apply to Source or Target Endpoints. The commands you specify here supercede the default commands typically applied at transfer time.

Override Default Receive Method is defined for Source Endpoints and applies the commands you specify when files are received from this Endpoint.

Override Default Send Method is defined for a Target Endpoint and applies the commands you specify when files are sent to this Endpoint.

See Command Reference for a list of supported commands. When you specify advanced commands, you can use macro variables, wildcards, and regular expressions as described in Using Macro Variables in CIC and Using Wildcards and Regular Expressions in CIC, respectively.

IMPORTANT: Any Advanced Commands specified for the Data Flow override the Move action of the Data Flow. This means if you specify any advanced commands and you want to move data from the source Endpoint to the destination Endpoint, you must include commands to do that.

Reverting Overrides to Default Values

Use the Revert button to return to the default send or receive method.

Post-Transfer Commands

Note: Post-Transfer commands are only available for Local File System Endpoints used as a Destination in a Data Flow.

You can configure a Data Flow to execute a SYSTEM command synchronously whenever a file is transferred to a File System endpoint.

On the Data Flow Edit page, select Advanced Commands and then select Post-Transfer command. The command entry field displays.

Enter the command you want to run after the transfer. See Command Syntax and Examples.

Click the Settings button to display a dialog box where you can set a time limit after which CIC stops attempting to run the command. A timeout value of 0 means there is no timeout.

Command Syntax

The post-transfer command supports a replacement token syntax, ${token}, that allows you to easily use the name of the file that was just transferred, formatted timestamps, and JavaScript expressions. The following tokens are supported:

Token	Description
`file`	the filename including path and full filename
`filename`	the full filename, excluding the path
`path`	the filename path (filename removed, trailing / or \ included)
`base`	the base portion of the filename (path and .extension removed)
`noext`	the base portion of the filename (.extension removed)
`ext`	the filename extension (including the . prefix)
`date('format')`	the current date/time formatted with 'format'
`expression`	a JavaScript expression, possibly including the above

The following diagram describes the relationship between the various filename components:

/path/to/file/basefilename.extension
|<------------- file ------------->|
|<-- path -->||<---- filename ---->|
              |<- base ->||<- ext >|
|<-------- noext ------->||<- ext >|

Examples

These expressions all evaluate to the same result:
${path}${base}${ext}
${noext}${ext}
${file}
${path+base+ext}

The following example of a Linux command renames the file by inserting the date and time before the extension:
/bin/mv ${file} ${noext}-${date('yyyy-MM-dd-HH-mm-ss')}${ext}

Command Output Logging

Successful post-transfer commands return a code of 0 while failed commands return a non-zero code. The command processor captures any output and copies it as detail records to the CIC log, with one log message per line. Standard output records are prefixed with [stdout] and are logged at log level LOW, while standard error records are prefixed with [stderr] and are logged at log level ESSENTIAL.

Filtering Source Folder

For File System, FTP, SFTP, GCS, S3, and AzureBlob Endpoints, once you have selected your source folder and clicked the Select button, you have the option to filter the source folder for a subset of files. This results in the Data Flow only sending the files that match the filter.

To specify a filter, click the filter button.

This displays a field where you can enter one or more filters. Enter one or more filters to apply in this field. Only files matching the pattern you enter are transferred for a given run of the Data Flow.

Filter Matching Rules and Examples

The following are the pattern-matching rules for filtering path and file names.

The * character matches zero or more characters of a name without crossing directory boundaries.
The ** characters match zero or more characters crossing directory boundaries.
The ? character matches exactly one character of a name.
The backslash character (\) is used to escape characters that would otherwise be interpreted as special characters. The expression \\ matches a single backslash and "\{" matches a left brace for example.
The [ ] characters are a bracket expression that match a single character of a name out of a set of characters. For example, [abc] matches "a", "b", or "c". You can use the hyphen (-) to specify a range. For example, [a-z] specifies a range that matches from "a" to "z" (inclusive). You can mix these forms, so that [abce-g] matches "a", "b", "c", "e", "f" or "g".
If the character after the [ is a !, it is used for negation. For example, [!a-c] matches any character except "a", "b", or "c".
Within a bracket expression the *, ? and \ characters match themselves. The (-) character matches itself if it is the first character within the brackets, or the first character after the ! if negating.
The { } characters are a group of subpatterns, where the group matches if any subpattern in the group matches. The "," character is used to separate the subpatterns. Groups cannot be nested.
Leading period/dot characters in file name are treated as regular characters. For example, the "*" pattern matches file name ".login".
All other characters match themselves.

Here are some examples of common filters:

Other Advanced Options

Controlling Recursion for a File System Endpoint

By default, recursion is disabled for all Souce Endpoints. This means transfers do not include folders and files below the selected directory.

However, if the Source is File System, FTP, SFTP, GCS, S3, or AzureBlob Endpoint, the recursion is supported, and the Recursion Toggle button is displayed on the Data Flow screen.

Data Flow Folder FS Endpoint Recursion Control

You can click the button to toggle recursion on or off. The value of the Recursion Toggle (on or off) is saved as part of the Data Flow.

For File System Endpoints, if the Access Point is not configured to allow recursion to be disabled, the Recursion Toggle button is not displayed and you cannot turn recursion off.

Defining the Event in Transformation Endpoints

Configure the Transformation Endpoint to include an Event (the object created in CIC Studio), which provides the means by which data can be brought into the Integration Server for further transformation using your Data Flow.

To do this, set an Event (and related Business Process) from within the CIC Cockpit as part of Data Flow creation (with a Transformation Endpoint), shown below. Any deployed Event in your workspace will be available for selection as you build the Data Flow.

Controlling whether files are deleted as a Data Flow runs

By default when applicable, Data Flows are configured to delete files from the Source directory when they run. However, for File System Endpoints, you can control whether the source files are deleted as the Data Flow runs.

Warning: Unchecking this option without a time-based filter in place will result in sending the same files repeatedly for each run of the Data Flow. (See Filtering Source Folder. ) Unchecking this option without a mechanism to clean up the Source folder in place could result in the folder becoming very large.

To prevent the Data Flow from deleting files from the Source folder:

Click on the vertical dots icon on the Source folder panel.

The Advanced Settings dialog displays.
Deselect the checkbox to not remove files after successful transfer.
Click OK to confirm your choice.

Understanding the Data Flow Page

Filtering the Data Flow list

Creating a Data Flow