< Back to Top

Working with Data Flows in Express

A Data Flow connects two Endpoints so that data can be transferred between them in a single direction. For more details on Data Flows, see What is a Data Flow?

This article contains information about how you can work with Data Flows in Express. 

Understanding the Data Flow Page

The Data Flow page is displayed when you click Integrations > Data Flows.

This page displays information about each of the Data Flows in your CIC system, including the following:

Name The name of the Data Flow. Used to identify the Data Flow throughout your CIC system.
From

The source Endpoint for the Data Flow.
To The destination Endpoint for the Data Flow.
Created by The person who created the Data Flow.
Modified

The date and time the Data Flow was last modified.
Modified By

The person who last modified the Data Flow.
Last Job

Status, date, and time when the Data Flow was last run.
Total Jobs

The number of times the Data Flow has been run.

Creating a Data Flow

  1. Go to the Integrations > Data Flows page and click the Add Data Flow button.
    The New Data Flow page displays.
  2. Give your data flow a name to identify it throughout the system.
  3. Select a From Endpoint. This will be your source Endpoint.
  4. For File System, FTP or SFTP Endpoints, specify a folder on the Endpoint to be the source of the transfer. You can use the Endpoint's default folder or specify a subfolder.
    • To use the default folder, simply click Select on the top right of the From Endpoint panel.
      mceclip0.png
    • To use a subfolder :
      • For a File System Endpoint, you can either browse the Endpoint’s file system to select the subfolder manually or click the Edit icon mceclip1.pngand type in the path to the subfolder. You do not need to precede the path with a forward slash(/). For example, you can enter /bin/documents or bin/documents as your path.

        On Windows, you can type a UNC path to specify network file shares, but you cannot browse to them. 

        Once you have selected a folder or subfolder, click Go to confirm your selection.

        image8.png

        Then, click Select to select the Endpoint.

    Note: With a File System Endpoint, by default the transfer will include all folders and files below the Access Point root directory. By selecting a subfolder you restrict the scope of the transfer to the subfolder and its children. You can further restrict the scope by specifying a filter pattern. See Filtering Source Folder below.

  5. For FTP and SFTP Endpoints, click Select and then type in the path to the subfolder. There is no option to browse for a folder.
    mceclip1.png
  6. Specify any advanced options if available for your Endpoint. Not all Endpoint types offer advanced options. See Advanced Options below.
  7. Select an Endpoint from the To Endpoints list. This will be your destination Endpoint. The process for selecting the Endpoint and folder is the same as described in steps 2 and 3 above.
  8. (Optional) For Data Flows where the source Endpoint is a File System, SFTP, or FTP(S) Endpoint, you can run the Data Flow manually or schedule the Data Flow to run on a recurring basis. For more details, see Scheduled in Running a Data Flow in Express.
  9. Click Save to save the Data Flow.

Editing a Data Flow

  1. Go to Integrations > Data Flows. The Data Flows page displays.
  2. In the list, click the Data Flow you want to edit.
  3. For either Endpoint in the Data Flow, click Unselect and then edit the path or commands.

Removing a Data Flow

  1. Go to Integrations > Data Flows.
  2. Select the check box for the Data Flow you want to remove.
    You can select more than one Data Flow from the list.
  3. Click the Trash Can icon in the upper right part of the page and then click Delete to confirm that you want to remove the Data Flow.

Disabling a Data Flow

When a Data Flow is disabled, it is greyed out on the Data Flows page.

You cannot disable a Data Flow directly. Instead, to disable a Data Flow, you disable one of its Endpoints.

Note: When you disable an Endpoint, all the Data Flows that use that Endpoint are also disabled.

See Enabling and Disabling Endpoints in the Managing Endpoints in Express article.

Advanced Options

Running Post-Transfer Commands

You can configure a Data Flow to execute a SYSTEM command synchronously whenever a file is transferred to a File System endpoint.

On the Data Flow Edit page, select the Run-post transfer command check box under the To Endpoint to display the command entry field.

Enter the command you want to run after the transfer. See Command Syntax and Examples.

Click the Settings button to display a dialog box where you can set a time limit after which CIC stops attempting to run the command. A timeout value of 0 means there is no timeout.

Command Syntax

The post-transfer command supports a replacement token syntax, ${token}, that allows you to easily use the name of the file that was just transferred, formatted timestamps, and JavaScript expressions.  The following tokens are supported:

Token Description
file the filename including path and full filename
filename the full filename, excluding the path
path the filename path (filename removed, trailing / or \ included)
base the base portion of the filename (path and .extension removed)
noext the base portion of the filename (.extension removed)
ext the filename extension (including the . prefix)
date('format') the current date/time formatted with 'format'
expression a JavaScript expression, possibly including the above

The following diagram describes the relationship between the various filename components:

/path/to/file/basefilename.extension
|<------------- file ------------->|
|<-- path -->||<---- filename ---->|
              |<- base ->||<- ext >|
|<-------- noext ------->||<- ext >|

Examples

These expressions all evaluate to the same result:
${path}${base}${ext}
${noext}${ext}
${file}
${path+base+ext}

The following example of a Linux command renames the file by inserting the date and time before the extension:
/bin/mv ${file} ${noext}-${date('yyyy-MM-dd-HH-mm-ss')}${ext}

Command Output Logging

Successful post-transfer commands return a code of 0 while failed commands return a non-zero code. The command processor captures any output and copies it as detail records to the CIC log, with one log message per line. Standard output records are prefixed with [stdout] and are logged at log level LOW, while standard error records are prefixed with [stderr] and are logged at log level ESSENTIAL.

Filtering Source Folder

For File System Endpoints, once you have selected your source folder and clicked the Select button, you have the option to filter the source folder for a subset of files. This results in the Data Flow only sending the files that match the filter.

To specify a filter, click the filter button.image5.png

This displays a field where you can enter one or more filters. image1.pngEnter one or more filters to apply in this field. Only files matching the pattern you enter are transferred for a given run of the Data Flow.

Filter Matching Rules and Examples

The following are the pattern matching rules for filtering path and file names.

  • The * character matches zero or more characters of a name without crossing directory boundaries.

  • The ** characters matches zero or more characters crossing directory boundaries.

  • The ? character matches exactly one character of a name.

  • The backslash character (\) is used to escape characters that would otherwise be interpreted as special characters. The expression \\ matches a single backslash and "\{" matches a left brace for example.

  • The [ ] characters are a bracket expression that match a single character of a name out of a set of characters. For example, [abc] matches "a", "b", or "c". You can use the hyphen (-) to specify a range. For example, [a-z] specifies a range that matches from "a" to "z" (inclusive). You can mix these forms, so that [abce-g] matches "a", "b", "c", "e", "f" or "g".

  • If the character after the [ is a !, it is used for negation. For example, [!a-c] matches any character except "a", "b", or "c".

    Within a bracket expression the *, ? and \ characters match themselves. The (-) character matches itself if it is the first character within the brackets, or the first character after the ! if negating.

  • The { } characters are a group of subpatterns, where the group matches if any subpattern in the group matches. The "," character is used to separate the subpatterns. Groups cannot be nested.

  • Leading period/dot characters in file name are treated as regular characters. For example, the "*" pattern matches file name ".login". 

  • All other characters match themselves.

Here are some examples of common filters:

image6.png

Advanced Commands

If your source Endpoint is FTP or SFTP, once you have selected your source Endpoint, you can enter advanced commands in the From Endpoint panel as shown below.image7.jpg

See Command Reference for a list of supported commands. When you specify advanced commands, you can use macro variables, wildcards, and regular expressions as described in Using Macro Variables in CIC and Using Wildcards and Regular Expressions in CIC, respectively.

IMPORTANT: Any Advanced Commands specified for the Data Flow override the Move action of the Data Flow. This means if you specify any advanced commands and you want to move data from the source Endpoint to the destination Endpoint, you must include commands to do that.

Related articles