The S3 connector is used to communicate with the Amazon Simple Storage Service (Amazon S3). In Amazon S3, buckets and objects are the primary resources, where objects are stored in buckets. Amazon S3 has a flat structure without the hierarchy you might see in a typical file system. However, for organizational simplicity, the Cleo S3 connector supports the folder concept as a means of grouping objects. Therefore, you can treat your folder and file objects as you would in a standard hierarchical file system. For example:
PUT example.txt S3TestFolder/example.txt
This command transfers the local file, example.txt, to the configured S3 bucket within a "folder" object called S3TestFolder. Note that S3TestFolder must exist as an object in S3.
Notes:
- This connector supports server-side encryption. The methods supported at this time are Server-Side Encryption with Amazon S3-Managed Keys (SSE-S3) and with Amazon KMS-Managed Keys (SSE-KMS).
- The ATTR command currently supports a subset of all Amazon S3 attributes.
- The Pseudo Folders property can be used to control whether actual folder objects are created and needed by the connector.
User Metadata
The User Metadata property is a table of Key/Value pairs where the Key is a simple string, but the Value is an expression that can expand metadata available in a PUT operation. The Value expression supports a replacement token syntax, which supports filename tokens, additional tokens, and expressions to allow the Value to include formatted timestamps, source metadata, and JavaScript expressions.
Tokens and expressions are embedded in the Value using the ${expression}
syntax, where the simplest expression is a direct token reference. The following tokens and expressions are supported:
Token | Description |
---|---|
file |
the filename including path and full filename |
filename |
the full filename, excluding the path |
path |
the filename path (filename removed, trailing / or \ included) |
base |
the base portion of the filename (path and .extension removed) |
noext |
the base portion of the filename (.extension removed) |
ext |
the filename extension (including the . prefix) |
date('format') |
the current date/time formatted with 'format' (see http://docs.oracle.com/javase/8/docs/api/java/text/SimpleDateFormat.html) |
source['metadata'] |
metadata propagated from the source (see below) |
expression |
a JavaScript expression, possibly including the above |
The following diagram describes the relationship between the various filename components:
/path/to/file/basefilename.extension |<------------- file ------------->| |<-- path -->||<---- filename ---->| |<- base ->||<- ext >| |<-------- noext ------->||<- ext >|
For example, the following expressions all evaluate to the same result:
- ${path}${base}${ext}
- ${noext}${ext}
- ${file}
- ${path+base+ext}
As a performance optimization, any JavaScript expression of the form ${token}
for one of the tokens above is expanded without invoking the JavaScript engine. Also, the primitives ${date('string')}
or ${date("string")}
are similarly expanded without invoking JavaScript. This means that although the four examples above produce the same result, the first three execute much more quickly. But the JavaScript engine provides more capability for some use cases. This example illustrates computing a Value extracted from the last portion of the DocDBTransferID (a GUID):
${source.DocDBTransferID.replace(/.*-/,'')}
Source Metadata
If an S3 connection is used directly through a URI reference as the inbox of another Harmony connection (or host or mailbox), additional protocol-specific metadata may be available in addition to the file content being stored. The metadata is represented as a JavaScript array, so use square brackets to access the values. For example, values available for AS2 include the following:
Key | Description |
---|---|
${source['AS2-From']} |
The AS2-From (partner) name |
${source['AS2-To']}
|
The AS2-To (local) name |
${source['ClientServer.Role']}
|
"Server" in a typical use case as described |
${source['Content-Disposition']}
|
HTTP.header.Content-Disposition |
${source['Content-Type']}
|
HTTP.header.Content-Type |
${source['DocDBTransferID']}
|
The Harmony REST API transfer ID |
${source['HTTP.header.'+header]}
|
The HTTP header "header" |
${source['Protocol']}
|
"AS2" |
${source['Subject']}
|
The AS2 Subject |
${source['TransferID']}
|
The Harmony transfer ID |
${source['VersaLex.Serial.Number.Short']}
|
The Harmony serial number, e.g. "HY1234" |
To inspect the available metadata for a particular configuration, use a Value like the following:
${JSON.stringify(source)}
S3 Connector Properties
Each instance of the S3 Connector can be configured using the following settings:
Property | Description | Required |
---|---|---|
Access Key | Part of the AWS credentials; the access key ID. | Yes |
Secret Access Key | Part of the AWS credentials; the secret access key. | Yes |
Assume Role Arn | Role ARN to use when assuming an AWS role for special permissions. Leave blank if not assuming a role. | |
Assume Role Duration (seconds) | Assume Role credentials duration. Must be in the range 900-43200 to conform to AWS specs. The resource owner may set a lower maximum than 43200. Valid range: [900-43200]. |
. |
Assumed Session Name | Session name used when assuming an AWS role for special permissions. | |
Assume Role External ID | External ID used to assume an AWS role if required by the policy owner. Optional when assuming a role. | |
Assume Role Source Identity | Source identity of principal assuming the AWS role. The policy owner may require it, but it is optional when assuming a role. | |
Canned Access Control List | The Canned Access Control List (ACL) to use when new S3 objects are created. Blank indicates no canned ACL deferring, to existing bucket trust poles. | |
Region | The regional endpoint to use for this S3 connection. Defaults to DEFAULT_REGION. | |
Bucket | The S3 bucket to use. | Yes |
User Metadata | The list of user metadata key/value pairs to add to new S3 objects when they are created. | |
Put Object Key |
The object key (or key ${expression}) to use when creating a new S3 object, overriding the destination filename. |
|
Force Unique | Force new S3 Object keys to be unique, even if -UNI is not specified. | |
Endpoint | The endpoint to use for the S3 connection. Note: It is recommended to use the Region setting instead of this setting. This setting should be configured only for non-standard S3 connections. |
|
Enable Path Style Access | Force path-style access to the bucket. Note: It is recommended to use the default setting instead of enabling this option. This setting should be configured only for non-standard S3 connections. |
|
Protocol | The protocol (for example, HTTP or HTTPS). | Yes |
Use Gap | Whether gzip compression should be used. | |
MuRipart Buffer Size (mbytes) | The part size used for a multipart upload. Valid range: [5-5120]. |
|
Multipart Attempts | Number of times to try each part of a multipart transfer. Valid range: [1-10]. |
|
Server side Encryption | The options for server-side encryption. Blank indicates no server-side encryption. | |
Kms Key Id | The KMS Key ID to use when encrypting with KMS. Leave blank to use the default KMS key for the bucket. | |
Pseudo Folders | Whether MKDIR/RMDIR should create/remove pseudo folder objects and where pseudo folder objects are required for DIR. | |
Preemptive Basic Proxy Auth | Whether to attempt to authenticate preemptively against proxy servers using basic authentication. | |
Client Execution Timeout (milliseconds) | The amount of time to allow the client to complete the execution of an API call. Valid range: [0-2147483647] |
|
Connection Max Idle Time (milliseconds) | The maximum amount of time that an idle connection may sit in the connection pool and still be eligible for reuse. Valid range: [0-9223372036854775807] |
|
Connection Timeout (milliseconds) | The amount of time to wait when initially establishing a connection before giving up and timing out. Valid range: Í0-2147483647] |
|
Connection TTL (milliseconds) | The expiration time for a connection in the connection pool. Valid range: [-1-9223372036854775807] |
|
Local Address | The optional local address the client will bind to. | |
Max Connections | The maximum number of open HTTP connections allowed. Valid range: [2-2147483647] |
|
Request Timeout (milliseconds) | The amount of time to wait for the request to complete before giving up and timing out. Valid range: [0-2147483647] |
|
Response Metadata Cache Size | The response metadata cache size. Valid range: [0-2147483647] |
|
Signer Override | The name of the signature algorithm to use for signing requests made by this client. | |
Socket Send Buffer Size Hint (bytes) | The optional size hint for the low-level TCP send buffer. Valid range: [0-2147483647] |
|
Socket Receive Buffer Size Hint (bytes) | The optional size hint for the low-level TCP receive buffer. Valid range: [0-2147483647] |
|
Socket Timeout (milliseconds) | The amount of time to wait for data to be transferred over an established, open connection before the connection times out and is closed. Valid range: [0-2147483647] |
|
Use Expect Continue | Whether "use expect continue" should be enabled. | |
Use Reaper | Whether the IdleConnectonReaper is to be started as a daemon thread. | |
Use Tcp Keep Alive | Whether or not to enable TCP KeepAlve support at the socket level. | |
Use Throttled Retries | Whether throttled retries should be used. | |
Use Transfer Acceleration | Whether S3 transfer acceleration should be used. | |
Multipart Upload Threads | Number of threads to use for a multipart transfer. Valid range: [1-32]. |
|
Command Retries | The number of times the command should be retried when an error or exception occurs. Valid range: [0-5]. |
Yes |
Command Retry Delay (seconds) | The number of seconds to wait between retries. Valid range: [0-120]. |
Yes |
Do Not Send Zero Length Fies | For PUT, a switch that indicates to not send a fife if it is zero-length. | |
Delete Received Zero Length Fies | For GET, a switch that indicates to remove a received file that is zero-length. | |
Retrieve Directory Sort | For PUT, the sorting options for the list of outbound files. | |
Enable Debug | A switch that indicates whether to perform debug logging. | |
System Scheme Name | The URI scheme name used as a shortcut to this host. Valid pattern: [[a-zA-Z]{l}[a-zA-Z0-9+\-\.]{l,23}] . |
|
System Public | A switch that indicates whether the connector is public. |
Comments
0 comments
Please sign in to leave a comment.