< Back to Top

S3 Endpoint

The S3 Endpoint enables integration with Amazon S3 cloud storage, allowing you to configure Data Flows that interact directly with S3 buckets. You can also link an S3 Endpoint to Partner Mailbox folders so partners can upload and download data from a shared location.

The S3 Endpoint attributes are organized into the following sections: General and Advanced.

General

Use the following attributes to identify the Endpoint in CIC, specify how CIC connects to Amazon S3, provide the credentials CIC uses to authenticate, and select the region and bucket. You can also configure cross-account access.

View General attributes

Attribute Description
Connect

Specify how to connect this Endpoint to other entities.

Directly Connect to external Trading Partners via CIC Cloud.
via Access Point Use an Access Point to connect to an S3 server when access is restricted for security purposes, for example, when only connections from your internal network are allowed. If you select this option, you must also select an Access Point. See Managing Access Points for more information.
Access Key Part of the AWS credentials. Used in conjunction with the Secret Access Key to access an S3 bucket.
Secret Access Key Part of the AWS credentials. Used in conjunction with the Access Key to access an S3 bucket.
Region The region to use for this S3 connection. Defaults to DEFAULT_REGION.
Bucket

The S3 bucket to use for this Endpoint.

Example: test/Data
Protocol The protocol (i.e. HTTP or HTTPS).
Enable Path Style Access

Force path-style access to the bucket.  It is recommended to use the default setting instead of enabling this option. This setting should be configured only for non-standard S3 connections.

Note: This field is always displayed when you specify Connect-Direct. It is displayed when you specify Connect-via Access Point only for Access Points with CIC Agent version 1.2.13 or later.
Cross-Account Access

Cross-account access allows you to share resources in one AWS account with users in another account without having to create additional users in the first account.

In AWS, cross-account access requires an IAM role with permissions for anything you want the role to be able to perform and a trust relationship that specifies which entities are allowed to assume the IAM role.

In CIC, the user who wants access to the S3 endpoint uses their own credentials for the S3 account, and then fills out the cross-account fields. See the sub-table below.

See your AWS documentation for more detailed information about cross-account access.

In the Cross-Account Access field, choose from the following:

None Don't use cross-account access.
AssumeRole

Use cross-account access. Choosing this option displays the following fields.

Amazon Resource Name (ARN) The Amazon Resource Name (ARN) of the role to assume. 
Session Name Use this property to identify a session when the same role is assumed by different principals or for different reasons. The role session name is visible to, and can be logged by, the account that owns the role. 
External ID A cross-account role is usually set up to trust everyone in an account. Therefore, the administrator of the trusting account might send an external ID to the administrator of the trusted account. That way, only someone with that ID can assume the role, rather than everyone in the account. 
Source Identity The source identity specified by the principal that is calling the AssumeRole operation.
You can require users to specify a source identity when they assume a role. You do this by using the sts:SourceIdentity condition key in a role trust policy. You can use source identity information in AWS CloudTrail logs to determine who took actions with a role. You can use the aws:SourceIdentity condition key to further control access to AWS resources based on the value of source identity. 

Advanced

The Advanced section contains optional settings that control security (Security), how CIC connects to S3 and transfers files (Connection & Transfer), who is notified about transfers (Events), upload settings (Cache), whether debug logging is on (Logging), and where the Endpoint is used as a Partner Mailbox linked folder (Linked Folders).

Security

Use these settings to specify how data is encrypted in the S3 bucket and whether CIC authenticates preemptively against proxy servers.

Attribute Description
Server Side Encryption

The options for server-side encryption.

Choose from the following:
KMS Key Id Used when encrypting with SSE-KMS. Leave blank to use default KMS key for the bucket.
Preemptive Basic Proxy Auth Select the check box to attempt to authenticate preemptively against proxy servers using basic authentication.

Connection & Transfer

Use these settings to control how CIC connects to S3 and transfers files. You can configure connection limits and timeouts, parallel downloads and compression, the order in which files are processed, how many directory entries are returned, retry behavior for failed transfers, and the number of simultaneous outbound connections.

Attribute Description
Parallel Download Threads

The maximum number of download threads allowed concurrently. 

Default value is 0 (no parallel downloads).

Maximum value is 10.
Use Gzip Select to use gzip compression.
Max Connections The maximum number of allowed open HTTP connections.
Client Execution Timeout (milliseconds) The amount of time to allow the client to complete the execution of an API call.
Connection Max Idle Time (milliseconds) The maximum amount of time that an idle connection can sit in the connection pool and still be eligible for reuse.
Connection Timeout (milliseconds) The amount of time to wait when initially establishing a connection before timing out.
Connection TTL (milliseconds) The expiration time for a connection in the connection pool.
Socket Timeout (milliseconds) The amount of time to wait for data to be transferred over an established, open connection before the connection times out and is closed.
Use Expect Continue Select to enable use expect continue.
Use Reaper Select to start the IdleConnectionReaper as a daemon thread.
Use TCP Keep Alive Select to enable TCP KeepAlive support at the socket level.
Use Throttled Retries Select to use throttled retries.
Use Transfer Acceleration Select to use S3 transfer acceleration.
Signer Override The name of the signature algorithm to use for signing requests made by this client.
Request Timeout (milliseconds) The amount of time to wait for the request to complete before timing out.
Retrieve Directory Sort

Specify the sorting order for files being transferred and processed. Choose from the following:

  • None  (Default value)
  • Alphabetical (ascending)
  • Alphabetical (descending)
  • Date/Time Modified (ascending)
  • Date/Time Modified (descending)
  • Size (ascending)
  • Size (descending)
Maximum Directory Entries Returned

Specify the maximum number of directory entries to return and display. 

Note: Use this property with caution when this Endpoint is being used as a linked folder for a Partner Mailbox. 

  • Specifying a value that is too small might not show all the files actually in the directory when browsing the Endpoint or when browsing the Endpoint in a Data Flow. In addition, jobs might not include all of the available files.
  • Specifying a value that is too large might cause timeouts because of the large number of directory entries.

Minimum value: 0 (no limit)
Maximum value: 2147483647
Default value: 1500
Automatic Retries

The number of retries that should be made during an attempt. An attempt consists of the number of transfer retries you specify. For example, if you specify 5 retries and 30 seconds, an attempt would consist of 5 retries occurring in the span of 30 seconds.

This setting applies to both sends and receives. 

The minimum number of retries is 0 and the maximum is 5.

The time between retries can range from 0 to 120 seconds.

See Retrying Failed File Transfer Attempts.
Extended Outbound Retries

The period of time during which outbound retries are attempted after a failed send attempt and subsequent failed automatic retry. Retries will not be attempted after the end of the period specified.

This period starts when the first retry is attempted and ends after the value you specify elapses.

Minimum value is 15 minutes.

Maximum value is 3 days.

See Retrying Failed File Transfer Attempts.
Concurrent Outbound Transfers Specify the maximum number of connections that can be transferring to this Endpoint at the same time. Minimum value is 1 and maximum is 10.
Endpoint The AWS endpoint to use for the S3 connection. Note: We recommended using the Region setting instead of this setting. This setting should be configured only for non-standard S3 connections.

Events

CIC allows you to send notification email to one or more recipients upon successful Send to or Receive from this Endpoint.  Use the Upon successful Send to this Endpoint... and Upon successful Receive from this Endpoint... fields to specify the recipients.

Recipients can be CIC users or external recipients. CIC users receive more detailed emails and benefit from a type-ahead dropdown when entering names, while external recipients get a simpler email and require full email addresses to be entered manually.

For all recipients, the notification email includes a link where the recipient can opt out of notifications from this Endpoint or all Endpoints. If a recipient opts out of these notifications, their email address no longer appears in the list of recipients you specified.

Cache

Use this setting to specify the part size CIC uses for multipart uploads to S3.

Attribute Description
Multipart Buffer Size (mbytes) The part size used for a multipart upload.

Logging

Use this setting to turn on debug logging for this Endpoint. Debug logging records additional detail about Endpoint activity, which can help when troubleshooting transfer problems.

Attribute Description
Enable Debug Select to enable debug-level logging for this endpoint.

Linked Folders

Linked folders let partners upload and download files directly from the bucket without copying files into each mailbox. This section lists where this Endpoint is used as a linked folder. To add or remove a link, edit the folder in the Partner Mailbox Endpoint. See Linking Partner Mailbox Folders.

S3 Endpoint Commands

The following commands are available for S3 Endpoints.

CLEAR

Clear a property string value. The cleared value only affects the commands that follow the CLEAR.

CLEAR property
property Property name with no embedded spaces.

DIR

Get a directory listing of available files from the host

DIR "source"
source Remote source directory path.

GET

Receive one or more files from the host

GET -REC -DEL "source" "destination"
-REC Recursively retrieve nested subdirectories.
If used in conjunction with -DEL, the retrieved files, but not subdirectories, are deleted on the server.
-DEL If the GET is successful, delete the remote file.
source

Remote source path. You can specify a path (folder/) or a path+filter (folder/*.txt)

If the path contains a macro variable, space, dash (-), comma (,), or equal sign (=), it must be enclosed with double quotes ("...").
destination

CIC destination path.

  • Path can be to a filename or to a directory.
  • You can use macro variables. See Using Macro Variables in CIC (Source context) for a list of the applicable macros.
  • If the path contains a macro variable, space, dash (-), comma (,), or equal sign (=), it must be enclosed with double quotes ("...").

PUT

Send one or more files to the host.

PUT "source" "destination"
source
  • Source is a filename.
  • You can use * and ?, or a regular expression when you specify a filename. See Using Wildcards and Regular Expressions in CIC for additional information.
  • You can use macro variables. See Using Macro Variables in CIC (Source context) for a list of the applicable macros.
  • If the source contains a macro variable, space, dash (-), comma (,), or equal sign (=), it must be enclosed with double quotes ("...").
destination Remote destination filename. If the destination contains a macro variable, space, dash (-), comma (,), or equal sign (=), it must be enclosed with double quotes ("..."). The use of macro variables is supported. See Using Macro Variables in CIC (Destination context) for a list of the applicable macros. 

SET

Change a property value. The new value only affects the commands that follow the SET.

SET property=value
property = value 

Property and new value.

  • The property name must have no embedded spaces.
  • The value specified remains in effect until it is set again or until the end of the Data Flow.
  • To reset the property back to default value, specify
    SET property
    or
    SET property=

Valid properties for SET command are as follows:

RetrieveDirectorySort

Use this property to control the order in which files are downloaded from the server. Possible values include:

  • Alphabetical (ascending)
  • Alphabetical (descending)
  • Date/Time Modified (ascending)
  • Date/Time Modified (descending)
  • Size (ascending)
  • Size (descending)
TerminateOnFail

Valid for all Endpoints configured as source in a Data Flow.
Use this property to control command processing when errors occur.
Possible values:

  • True - Command processing stops when an error occurs.
  • False - Command processing continues even when an error occurs.

Default value is True.

WAIT

Pause execution.

WAIT seconds
seconds Number of seconds to pause.

Related articles

Comments

0 comments

Please sign in to leave a comment.