Sign in
< Back to Top

S3 Endpoint

The S3 Endpoint has the following attributes:

Attribute Description
Connect Specify how to connect this Endpoint to other entities.
Directly Connect to external Trading Partners via CIC Cloud.
via Access Point Use an Access Point to connect to an S3 server when access is restricted for security purposes, for example, when only connections from your internal network are allowed. If you select this option, you must also select an Access Point. See Managing Access Points for more information.
Access Key Part of the AWS credentials. Used in conjunction with the Secret Access Key to access an S3 bucket.
Secret Access Key Part of the AWS credentials. Used in conjunction with the Access Key to access an S3 bucket.
Region The region to use for this S3 connection. Defaults to DEFAULT_REGION.
Bucket The S3 bucket to use for this Endpoint.
Protocol The protocol (i.e. HTTP or HTTPS).
Enable Path Style Access

Force path-style access to the bucket.  It is recommended to use the default setting instead of enabling this option. This setting should be configured only for non-standard S3 connections.

Note: This field is always displayed when you specify Connect-Direct. It is displayed when you specify Connect-via Access Point only for Access Points with CIC Agent version 1.2.13 or later.
Cross-Account Access

Cross-account access allows you to share resources in one AWS account with users in another account without having to create additional users in the first account.

In AWS, cross-account access requires an IAM role with permissions for anything you want the role to be able to perform and a trust relationship that specifies which entities are allowed to assume the IAM role.

In CIC, the user who wants access to the S3 endpoint uses their own credentials for the S3 account, and then fills out the cross-account fields. See the sub-table below.

See your AWS documentation for more detailed information about cross-account access.

In the Cross-Account Access field, choose from the following:

None Don't use cross-account access.
AssumeRole Use cross-account access. Choosing this option displays the following fields.
Amazon Resource Name (ARN)

The Amazon Resource Name (ARN) of the role to assume. 
Session Name

Use this property to identify a session when the same role is assumed by different principals or for different reasons. The role session name is visible to, and can be logged by, the account that owns the role. 
External ID A cross-account role is usually set up to trust everyone in an account. Therefore, the administrator of the trusting account might send an external ID to the administrator of the trusted account. That way, only someone with that ID can assume the role, rather than everyone in the account. 
Source Identity

The source identity specified by the principal that is calling the AssumeRole operation.
You can require users to specify a source identity when they assume a role. You do this by using the sts:SourceIdentity condition key in a role trust policy. You can use source identity information in AWS CloudTrail logs to determine who took actions with a role. You can use the aws:SourceIdentity condition key to further control access to AWS resources based on the value of source identity. 
Advanced - Security
Server Side Encryption

The options for server-side encryption.

Choose from the following:
KMS Key Id Used when encrypting with SSE-KMS. Leave blank to use default KMS key for the bucket.
Preemptive Basic Proxy Auth Select the check box to attempt to authenticate preemptively against proxy servers using basic authentication.
Advanced - Connection
Parallel Download Threads

The maximum number of download threads allowed concurrently. 

Default value is 0 (no parallel downloads).

Maximum value is 10.
Use Gzip Select to use gzip compression.
Max Connections The maximum number of allowed open HTTP connections.
Client Execution Timeout (milliseconds) The amount of time to allow the client to complete the execution of an API call.
Connection Max Idle Time (milliseconds) The maximum amount of time that an idle connection can sit in the connection pool and still be eligible for reuse.
Connection Timeout (milliseconds) The amount of time to wait when initially establishing a connection before timing out.
Connection TTL (milliseconds) The expiration time for a connection in the connection pool.
Socket Timeout (milliseconds) The amount of time to wait for data to be transferred over an established, open connection before the connection times out and is closed.
Use Expect Continue Select to enable use expect continue.
Use Reaper Select to start the IdleConnectionReaper as a daemon thread.
Use TCP Keep Alive Select to enable TCP KeepAlive support at the socket level.
Use Throttled Retries Select to use throttled retries.
Use Transfer Acceleration Select to use S3 transfer acceleration.
Signer Override The name of the signature algorithm to use for signing requests made by this client.
Request Timeout (milliseconds) The amount of time to wait for the request to complete before timing out.
Retrieve Directory Sort

Specify the sorting order for files being transferred and processed. Choose from the following:

  • None  (Default value)
  • Alphabetical (ascending)
  • Alphabetical (descending)
  • Date/Time Modified (ascending)
  • Date/Time Modified (descending)
  • Size (ascending)
  • Size (descending)
Automatic Retries

The number of retries that should be made during an attempt. An attempt consists of the number of transfer retries you specify. For example, if you specify 5 retries and 30 seconds, an attempt would consist of 5 retries occurring in the span of 30 seconds.

This setting applies to both sends and receives. 

The minimum number of retries is 0 and the maximum is 5.

The time between retries can range from 0 to 120 seconds.

See Retrying Failed File Transfer Attempts.
Extended Outbound Retries

The period of time during which outbound retries are attempted after a failed send attempt and subsequent failed automatic retry. Retries will not be attempted after the end of the period specified.

This period starts when the first retry is attempted and ends after the value you specify elapses.

Minimum value is 15 minutes.

Maximum value is 3 days.

See Retrying Failed File Transfer Attempts.
Concurrent Outbound Transfers Specify the maximum number of connections that can be transferring to this Endpoint at the same time.

Minimum value is 1 and maximum is 10.
Endpoint The AWS endpoint to use for the S3 connection. Note: We recommended using the Region setting instead of this setting. This setting should be configured only for non-standard S3 connections.
Advanced - Cache
Multipart Buffer Size (mbytes) The part size used for a multipart upload.
Response Metadata Cache Size The response metadata cache size.
Socket Send Buffer Size Hint (bytes) The optional size hint for the low level TCP send buffer
Socket Receive Buffer Size Hint (bytes) The optional size hint for the low level TCP receive buffer.
Advanced - Logging
Enable Debug Select to enable debug-level logging for this endpoint.

S3 Endpoint Commands

The following commands are available for S3 Endpoints.

CLEAR

Clear a property string value. The cleared value only affects the commands that follow the CLEAR.

CLEAR property
property Property name with no embedded spaces.

DIR

Get a directory listing of available files from the host

DIR "source"
source Remote source directory path.

GET

Receive one or more files from the host

GET -REC –DEL "source" "destination"
-REC Recursively retrieve nested subdirectories.
If used in conjunction with –DEL, the retrieved files, but not subdirectories, are deleted on the server.
-DEL If the GET is successful, delete the remote file.
source

Remote source path. You can specify a path (folder/) or a path+filter (folder/*.txt)

If the path contains a macro variable, space, dash (-), comma (,), or equal sign (=), it must be enclosed with double quotes ("...").
destination

CIC destination path.

  • Path can be to a filename or to a directory.
  • You can use macro variables. See
  • See Using Macro Variables in CIC (Source context) for a list of the applicable macros.
  • If the path contains a macro variable, space, dash (-), comma (,), or equal sign (=), it must be enclosed with double quotes ("...").

PUT

Send one or more files to the host.

PUT "source" "destination"
source
  • Source is a filename.
  • You can use * and ?, or a regular expression when you specify a filename. See Using Wildcards and Regular Expressions in CIC for additional information.
  • You can use macro variables. See Using Macro Variables in CIC (Source context) for a list of the applicable macros.
  • If the source contains a macro variable, space, dash (-), comma (,), or equal sign (=), it must be enclosed with double quotes ("...").
destination Remote destination filename. If the destination contains a macro variable, space, dash (-), comma (,), or equal sign (=), it must be enclosed with double quotes ("..."). The use of macro variables is supported. See Using Macro Variables in CIC (Destination context) for a list of the applicable macros. 

SET

Change a property value. The new value only affects the commands that follow the SET.

SET property=value
property = value  Property and new value.
  • The property name must have no embedded spaces.
  • The value specified remains in effect until it is set again or until the end of the dataflow. 
  • To reset the property back to default value, specify
    SET property
    or
    SET property=

Valid properties for SET command are as follows:

RetrieveDirectorySort Use this property to control the order in which files are downloaded from the server. Possible values include:
  • Alphabetical (ascending)
  • Alphabetical (descending)
  • Date/Time Modified (ascending)
  • Date/Time Modified (descending)
  • Size (ascending)
  • Size (descending)
TerminateOnFail

Valid for all Endpoints configured as source in a Data Flow.
Use this property to control command processing when errors occur.
Possible values:

  • True - Command processing stops when an error occurs.
  • False - Command processing continues even when an error occurs.

Default value is True.

WAIT

Pause execution.

WAIT seconds
seconds Number of seconds to pause.

Related articles

Comments

0 comments

Please sign in to leave a comment.