In the context of Cleo Harmony, containerization refers to packaging Harmony services into lightweight, portable containers using Docker and orchestrated via Kubernetes or another platform, resulting in faster deployments and scalable infrastructure.
This article outlines requirements, setup, platforms, and known issues and limitations.
Requirements
Before you deploy Harmony in a containerized environment, certain requirements must be met to ensure that the system is properly licensed, has access to necessary configuration and runtime resources, and can operate properly in your environment.
Enterprise license
A Harmony Enterprise license is required when running the product within containers. Please contact your Cleo sales representative for more information.
Shared repositories
Two shared, persisted, centralized repositories are required -- one for configuration and one for runtime.
The configuration for these shared repositories are passed as secrets to the Harmony container. See Secrets for more information.
Configuration repository
Harmony creates a subdirectory under the base path using the system name. This directory stores configuration files shared across all nodes in the cluster, including hosts, options, certificates, and more.
Supported repository types: smb, s3, azureblob, gcpbucket.
Runtime repository
Each Harmony node creates a parent subdirectory using the system name (if not already present), with a child subdirectory named after the host. This directory stores node-specific runtime files, primarily protocol message IDs and receipts.
Repository types supported currently include smb.
Access to cleo.com
Each Harmony container verifies the enterprise license at startup. To do this, https://license.cleo.com must be accessible from the container. Note that during the beta timeframe, the URL will be https://license-beta.cleo.com.
If license.cleo.com is not accessible at startup, Harmony will still start up and periodically retry license verification. However, a 4-day grace period is initiated. If the license is not verified during this period, the container will exit. In case the problem reaching license.cleo.com is on Cleo’s side, an internal IT ticket is created for the enterprise license’s serial number by posting to https://it-ticket.cleo.com.
Harmony will email the system administrator if the grace period is initiated. Therefore, Cleo strongly recommends setting the system administrator email address and the necessary SMTP proxy. See YAML system settings below.
Resource requirements
Refer to the latest Harmony release system requirements: Cleo Harmony 5.8.1 System Requirements. Additional disk space for logging is still required during the beta phase; eventually, logging will be applied to a customer-provided enterprise log management system.
VLProxy
If you're using Cleo VLProxy within your network, note that VLProxy itself is not containerized and its setup remains the same. The only difference when configuring it for a Harmony container cluster is in the VLProxy Serial Numbers property. Instead of specifying a static serial number, use a regular expression. For example, if your Enterprise license serial number is HC1234-ZZ5678, set the VLProxy Serial Numbers property to HC1234-ZZ5678-.* to match relevant container instances. During the beta period, Cleo will provide a version of the VLProxy installer that supports this regex-based configuration.
Setting Up a Harmony Container
Setting up a Harmony container involves several key steps to ensure the environment is properly initialized, securely configured, and ready for operation. This section outlines the required inputs, secret management practices, and environment variable configurations necessary for a successful deployment.
The following diagram illustrates the Harmony containerization environment.
Initialization
Before Harmony can run in a containerized environment, it must be initialized with system-specific settings and credentials. Initialization is performed by launching a single Harmony container with the appropriate inputs. Both the YAML system settings and Administrator user password are passed as Secrets (see Secrets below) to the container. When initialization is complete, the Harmony container exits.
The inputs described below define the system’s configuration and establish administrative access.
YAML system settings
- Available YAML system settings are documented here: https://developer.cleo.com/api/api-reference/resource-settings. Note: During the beta timeframe, this URL will be https://developer-beta.cleo.com/api/api-reference/resource-settings.
- Include settings for local listener, system options, proxies, nodes, and so on.
-
At a minimum, nodes must be included in the YAML settings to specify the number of containers and each host name. For example:
--- nodes: - alias: harmony-1 url: https://harmony-1.harmony-service.harmony.svc.cluster.local:6443 enabled: true - alias: harmony-2 url: https://harmony-2.harmony-service.harmony.svc.cluster.local:6443 enabled: true - alias: harmony-3 url: https://harmony-3.harmony-service.harmony.svc.cluster.local:6443 enabled: trueFor more information about other sections of the YAML file, see https://developer-beta.cleo.com/api/api-reference/get-settings.
- It is recommended, however, to review all of the default system settings and change as many as necessary in advance using YAML.
- The expected maximum number of containers should be configured.
- It is recommended to retrieve and back up the system settings from Harmony whenever they have been changed at runtime. Use the
GET includeProtected=truerequest parameter to ensure all passwords are included in the backup. Note that this requires an administrator group user.
Administrator user password
- Set a strong and secure password for the default administrator account.
Secrets
Secrets are used to securely pass sensitive configuration data to the Harmony container, including credentials, license files, and repository connection settings.
The technique of passing a secret is dependent on the platform being used (see Platforms section below).
| Secret name | Purpose | When required |
|---|---|---|
| cleo-system-settings | See Initialization section above | Initialization only |
| cleo-default-admin-password | See Initialization section above | Initialization only |
| cleo-license | Enterprise license file provided by Cleo | Always required |
| cleo-license-verification-code | Enterprise license verification code provided by Cleo | Always required |
| cleo-config-repo | YAML connection settings for the configuration repository. See examples below. | Always required |
| cleo-runtime-repo | YAML connection settings for the runtime repository. See examples below. | Runtime only |
If the same repository is to be used for both configuration and runtime files, the YAML connection settings must still be provided in both the cleo-config-repo and cleo-runtime-repo secrets.
Environment variables
Environment variables must be set correctly to ensure Harmony operates as expected in a containerized environment.
| Environment variable name | Purpose | When required |
|---|---|---|
| CLEO_SYSTEM_NAME | The name used to identify the system. It can be up to 15 characters with syntax in the style of a hostname. The enterprise license can be used for any number of systems across the enterprise. It is the customer's responsibility to make sure system names are unique across Harmony container deployments. Failing to ensure uniqueness can cause corruption. | Always required |
| CLEO_SECRETS_MOUNT_POINT | The location of the secrets (see Secrets section above and Platforms section below) | Always required |
Platforms
Cleo Harmony containerization currently supports Kubernetes. Other platforms, like Docker Swarm, are planned for future support.
Kubernetes
This section outlines how Harmony uses Kubernetes manifests to define and control deployment, including initialization and runtime operations.
Kubernetes manifests
There are two manifests used for Harmony containerization: harmony-init.yaml and harmony-run.yaml.
harmony-init.yaml
This manifest defines a Kubernetes Job that initializes Harmony with system settings, environment variables, and secrets. It runs once, sets up the environment, and exits after completing the setup. The following example is valid for both AWS and Azure.
View example
apiVersion: batch/v1
kind: Job
metadata:
name: harmony-init
namespace: harmony
spec:
# The number of times a Job will be retried before it is marked as failed
backoffLimit: 2
template:
spec:
containers:
- name: harmony-init
image: cleodev/harmony:PR-11140
imagePullPolicy: Always
args: ["-s", "service"]
# Environment variables
env:
# CLEO_SYSTEM_NAME: the name of this system (group of Harmony instances)
- name: CLEO_SYSTEM_NAME
value: System1
# CLEO_SECRETS_MOUNT_POINT: the location of the secrets
- name: CLEO_SECRETS_MOUNT_POINT
value: /var/secrets
# Volume mounts
volumeMounts:
# Specify where initsecrets volume should be mounted in the container. Should match CLEO_SECRETS_MOUNT_POINT
- name: initsecrets
mountPath: /var/secrets
readOnly: true
restartPolicy: Never
# Volumes mounted in volumeMounts
volumes:
# Initialization secrets to be used
- name: initsecrets
projected:
sources:
- secret:
name: cleo-license
- secret:
name: cleo-license-verification-code
- secret:
name: cleo-config-repo
- secret:
name: cleo-runtime-repo
- secret:
name: cleo-default-admin-password
- secret:
name: cleo-system-settings
harmony-run.yaml
This manifest defines how Harmony runs in a Kubernetes environment after initialization. It creates a StatefulSet to manage Harmony pods, configure environment variables and secrets, and provision services for network access and load balancing. This manifest is different for AWS and Azure. See examples of each below.
View AWS EKS example
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: harmony
namespace: harmony
spec:
serviceName: harmony-service
selector:
matchLabels:
app: harmony
# Number of Harmony instances to run
replicas: 2
template:
metadata:
labels:
app: harmony
spec:
containers:
- name: harmony
image: cleodev/harmony:PR-11140
imagePullPolicy: Always
args: ["-s", "service"]
# Environment variables
env:
# CLEO_SYSTEM_NAME: the name of this system (group of Harmony instances)
- name: CLEO_SYSTEM_NAME
value: System1
# CLEO_SECRETS_MOUNT_POINT: the location of the secrets
- name: CLEO_SECRETS_MOUNT_POINT
value: /var/secrets
# Volume mounts
volumeMounts:
# Specify where runtimesecrets volume should be mounted in the container.
# Should match CLEO_SECRETS_MOUNT_POINT.
- name: runtimesecrets
mountPath: /var/secrets
readOnly: true
# Resource settings for the Harmony containers
resources:
requests:
memory: "4096Mi"
limits:
memory: "8192Mi"
# Volumes mounted in volumeMounts
volumes:
# Runtime secrets to be used
- name: runtimesecrets
projected:
sources:
- secret:
name: cleo-license
- secret:
name: cleo-license-verification-code
- secret:
name: cleo-config-repo
- secret:
name: cleo-runtime-repo
ordinals:
# Start naming at harmony-1 instead of harmony-0
start: 1
---
apiVersion: v1
kind: Service
metadata:
name: harmony-service
namespace: harmony
spec:
clusterIP: None
selector:
app: harmony
---
apiVersion: v1
kind: Service
metadata:
name: harmony
namespace: harmony
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: external
service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: "preserve_client_ip.enabled=true,stickiness.enabled=true,stickiness.type=source_ip"
spec:
# Load balancer setup for harmony
sessionAffinity: ClientIP
externalTrafficPolicy: Local
selector:
app: harmony
ports:
- name: admin-port
port: 5080
targetPort: 5080
- name: http-port
port: 80
targetPort: 80
- name: https-port
port: 443
targetPort: 443
- name: sftp-port
port: 22
targetPort: 22
- name: oftp-standard-port
port: 3305
targetPort: 3305
- name: oftp-tls-port
port: 6619
targetPort: 6619
# If using VLProxy for reverse proxying FTP, then the section below to configure the
# ftp ports is not required.
- name: ftp-data-port
port: 20
targetPort: 20
- name: ftp-port
port: 21
targetPort: 21
# Notes on FTP passive data ports. In the `cleo-system-settings` that are being used,
# the `lowPort` and `highPort` specified need to match the `ftp-passive-port-#` below
# otherwise all the necessary ports will not be open on the load balancer and connections
# will fail.
#
# passiveDataChannelMode:
# lowPort: 20900
# highPort: 20909
#
# Another additional constraint is that the maximum number of open ports on an
# AWS load balancer is 50.
- name: ftp-passive-port-20900
port: 25900
targetPort: 25900
- name: ftp-passive-port-20901
port: 25901
targetPort: 25901
- name: ftp-passive-port-20902
port: 25902
targetPort: 25902
- name: ftp-passive-port-20903
port: 25903
targetPort: 25903
- name: ftp-passive-port-20904
port: 25904
targetPort: 25904
- name: ftp-passive-port-20905
port: 25905
targetPort: 25905
- name: ftp-passive-port-20906
port: 25906
targetPort: 25906
- name: ftp-passive-port-20907
port: 25907
targetPort: 25907
- name: ftp-passive-port-20908
port: 25908
targetPort: 25908
- name: ftp-passive-port-20909
port: 25909
targetPort: 25909
type: LoadBalancerView Azure AKS example
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: harmony
namespace: harmony
spec:
serviceName: harmony-service
selector:
matchLabels:
app: harmony
# Number of Harmony instances to run
replicas: 2
template:
metadata:
labels:
app: harmony
spec:
containers:
- name: harmony
image: cleodev/harmony:PR-11140
imagePullPolicy: Always
args: ["-s", "service"]
# Environment variables
env:
# CLEO_SYSTEM_NAME: the name of this system (group of Harmony instances)
- name: CLEO_SYSTEM_NAME
value: System1
# CLEO_SECRETS_MOUNT_POINT: the location of the secrets
- name: CLEO_SECRETS_MOUNT_POINT
value: /var/secrets
# Volume mounts
volumeMounts:
# Specify where runtimesecrets volume should be mounted in the container.
# Should match CLEO_SECRETS_MOUNT_POINT.
- name: runtimesecrets
mountPath: /var/secrets
readOnly: true
# Resource settings for the Harmony containers
resources:
requests:
memory: "4096Mi"
limits:
memory: "8192Mi"
# Volumes mounted in volumeMounts
volumes:
# Runtime secrets to be used
- name: runtimesecrets
projected:
sources:
- secret:
name: cleo-license
- secret:
name: cleo-license-verification-code
- secret:
name: cleo-config-repo
- secret:
name: cleo-runtime-repo
ordinals:
# Start naming at harmony-1 instead of harmony-0
start: 1
---
apiVersion: v1
kind: Service
metadata:
name: harmony-service
namespace: harmony
spec:
clusterIP: None
selector:
app: harmony
---
apiVersion: v1
kind: Service
metadata:
name: harmony
namespace: harmony
annotations:
service.beta.kubernetes.io/azure-load-balancer-client-ip: "true"
spec:
# Load balancer setup for harmony
sessionAffinity: ClientIP
externalTrafficPolicy: Local
selector:
app: harmony
ports:
- name: admin-port
port: 5080
targetPort: 5080
- name: http-port
port: 80
targetPort: 80
- name: https-port
port: 443
targetPort: 443
- name: sftp-port
port: 22
targetPort: 22
- name: oftp-standard-port
port: 3305
targetPort: 3305
- name: oftp-tls-port
port: 6619
targetPort: 6619
# If using VLProxy for reverse proxying FTP, then the section below to configure the
# ftp ports is not required.
- name: ftp-data-port
port: 20
targetPort: 20
- name: ftp-port
port: 21
targetPort: 21
# Notes on FTP passive data ports. In the `cleo-system-settings` that are being used,
# the `lowPort` and `highPort` specified need to match the `ftp-passive-port-#` below
# otherwise all the necessary ports will not be open on the load balancer and connections
# will fail.
#
# passiveDataChannelMode:
# lowPort: 20900
# highPort: 20909
#
# Another additional constraint is that the maximum number of open ports on an
# AWS load balancer is 50.
- name: ftp-passive-port-20900
port: 25900
targetPort: 25900
- name: ftp-passive-port-20901
port: 25901
targetPort: 25901
- name: ftp-passive-port-20902
port: 25902
targetPort: 25902
- name: ftp-passive-port-20903
port: 25903
targetPort: 25903
- name: ftp-passive-port-20904
port: 25904
targetPort: 25904
- name: ftp-passive-port-20905
port: 25905
targetPort: 25905
- name: ftp-passive-port-20906
port: 25906
targetPort: 25906
- name: ftp-passive-port-20907
port: 25907
targetPort: 25907
- name: ftp-passive-port-20908
port: 25908
targetPort: 25908
- name: ftp-passive-port-20909
port: 25909
targetPort: 25909
type: LoadBalancerKubernetes secrets
Kubernetes secrets are used to store sensitive information, such as licenses, passwords, repository configuration, and so on. They are designed to keep this information secure and separate from the application code.
Secrets can all be stored in a single YAML file or generated directly from the command line.
Secrets via YAML
Secrets can all be stored in a single file, such as secrets.yaml, and then applied to the Kubernetes cluster using the kubectl apply -f secrets.yaml command. However, the secrets must be base64-encoded before they can be used in the YAML file to meet a Kubernetes security requirement.
View example secrets.yaml file
apiVersion: v1
kind: Secret
metadata:
name: cleo-config-repo
namespace: harmony
type: Opaque
data:
# You can use 'cat secrets/cleo-config-repo | base64' to convert the file to base64
secret: |
LS0tCnR5cGU6IHMzCnJvb3RQYXRoOiBicmlhbmluaXQKY29ubmVjdG9yUHJvcGVydGllczoKICBi
dWNrZXQ6IGNsZW9kYWVtb25zCiAgcmVnaW9uOiB1cy1lYXN0LTEKICBwcm90b2NvbDogSFRUUFMK
ICBhY2Nlc3NLZXk6IDxBV1MgYWNjZXNzIGtleT4KICBzZWNyZXRBY2Nlc3NLZXk6IDxBV1Mgc2Vj
cmV0IGFjZXNzIGtleT4KYWR2YW5jZWRQcm9wZXJ0aWVzOgogIG91dGJveFNvcnQ6IERhdGUvVGlt
ZSBNb2RpZmllZAogIHRlcm1pbmF0ZU9uRmFpbDogdHJ1ZQ==
---
apiVersion: v1
kind: Secret
metadata:
name: cleo-runtime-repo
namespace: harmony
type: Opaque
data:
# You can use 'cat secrets/cleo-runtime-repo | base64' to convert the file to base64
secret: |
LS0tCnR5cGU6IHMzCnJvb3RQYXRoOiBicmlhbmluaXQKY29ubmVjdG9yUHJvcGVydGllczoKICBi
dWNrZXQ6IGNsZW9kYWVtb25zCiAgcmVnaW9uOiB1cy1lYXN0LTEKICBwcm90b2NvbDogSFRUUFMK
ICBhY2Nlc3NLZXk6IDxBV1MgYWNjZXNzIGtleT4KICBzZWNyZXRBY2Nlc3NLZXk6IDxBV1Mgc2Vj
cmV0IGFjZXNzIGtleT4KYWR2YW5jZWRQcm9wZXJ0aWVzOgogIG91dGJveFNvcnQ6IERhdGUvVGlt
ZSBNb2RpZmllZAogIHRlcm1pbmF0ZU9uRmFpbDogdHJ1ZQ==Secrets via command line
You can create Kubernetes secrets directly from the command line without needing to encode them in a YAML file. This is useful for scripting, quick setups, or testing.
View examples
Command to see all secrets currently in the harmony namespace:
kubectl get secrets -n harmony
The secrets cannot be created if they already exist. If you need to update an existing secret, you can delete it first using a command like the following:
kubectl delete secret cleo-license-verification-code -n harmony
kubectl delete secret cleo-license -n harmony
kubectl delete secret cleo-config-repo -n harmony
kubectl delete secret cleo-runtime-repo -n harmony
kubectl delete secret cleo-default-admin-password -n harmony
kubectl delete secret cleo-system-settings -n harmony
The following are some example secret files that can be used with the cleo-config-repo and cleo-runtime-repo secrets.
These files define the repository configurations for the Harmony application.
The type of repository can be File, S3, SMB, AzureBlob and GCP, and the
configuration
will vary based on the type. See https://developer.cleo.com/api/api-reference/resource-connections
for
the full set of connectorProperties.
Example File secret file:
---
type: file
connectorProperties:
rootPath: /app/hostrepo
advancedProperties:
outboxSort: Date/Time Modified
Example S3 secret file:
---
type: s3
rootPath: <path>
connectorProperties:
bucket: <bucket>
region: <region>
protocol: HTTPS
accessKey: <access-key>
secretAccessKey: <secret-access-key>
advancedProperties:
outboxSort: Date/Time Modified
Example SMB secret file:
---
type: smb
connectorProperties:
sharePath: //1.2.3.4/containershare
userName: <username>
userPassword: <password>
advancedProperties:
outboxSort: Date/Time Modified
Example AzureBlob secret file:
---
type: AzureBlob
rootPath: <path>
connectorProperties:
accessKey: <access-key>
blobType: BLOCK_BLOB
storageAccountName: <storage-account-name>
container: <container>
advancedProperties:
outboxSort: Date/Time Modified
To view the contents of a secret, you can use the following command. This will decode the base64-encoded data and display it in a human-readable format:
kubectl get secret cleo-license-verification-code -n harmony -o jsonpath='{.data.cleo-license-verification-code}' | base64 --decode
kubectl get secret cleo-license -n harmony -o jsonpath='{.data.cleo-license}' | base64 --decode
kubectl get secret cleo-config-repo -n harmony -o jsonpath='{.data.cleo-config-repo}' | base64 --decode
kubectl get secret cleo-runtime-repo -n harmony -o jsonpath='{.data.cleo-runtime-repo}' | base64 --decode
kubectl get secret cleo-default-admin-password -n harmony -o jsonpath='{.data.cleo-default-admin-password}' | base64 --decode
kubectl get secret cleo-system-settings -n harmony -o jsonpath='{.data.cleo-system-settings}' | base64 --decode
Initializing and Running Harmony
Deploying Harmony in Kubernetes involves two steps: initialization and runtime. Initialization sets up system settings and credentials using a one-time job. Once complete, the runtime phase launches Harmony pods that run continuously and scale as needed.
Initializing Harmony
To run the Kubernetes initialization job, you can use the following command
kubectl apply -f harmony-init.yaml
Use the following command to check the status of the job:
kubectl get pods -n harmony -o wide
Use a command like the following to see the logs of the job. Note: The pod name will vary, so you will need to adjust the command accordingly. The pod name is visible in the output of the previous command.
kubectl logs harmony-init-v7r4g -n harmony
The job is successful if you see a status of COMPLETED. In either case, you can delete the job using the following command:
kubectl delete -f harmony-init.yaml
The job will not run again until you delete it and reapply the harmony-init.yaml file again.
Running the Harmony application
After the Harmony system has been initialized, you can start the Harmony pods using the following command:
kubectl apply -f harmony-run.yaml
Use the following command to check the status of the pods:
kubectl get pods -n harmony -o wide
Use a command like the following to see the logs of a pod. Note: The pod name will vary, so you must adjust the command accordingly. The pod name is visible in the output of the previous command.
kubectl logs harmony-1 -n harmony
The Harmony pod(s) will continuously run until you stop them. Even if there is an issue and Harmony stops, it will restart. To stop the Harmony pods, use the following command:
kubectl delete -f harmony-run.yaml
Miscellaneous useful kubectl commands
This section provides a set of practical kubectl commands that help you monitor pod status, access logs, transfer files, and troubleshoot deployments.
View commands
Get a list of all pods in the Harmony namespace
kubectl get pods -n harmony -o wide
Get the logs (screen output) of a specific pod
kubectl logs harmony-1 -n harmony
Get a bash prompt within the Harmony container
kubectl exec -it harmony-1 -n harmony -- bash
Describe a specific pod to see detailed information
kubectl describe pod harmony-1 -n harmony
Copy file from a pod to your computer
Note: The default folder in harmony-1 is /opt/versalex. You can use the full path if needed:
kubectl cp -n harmony harmony-1:logs/Harmony.xml Harmony.xml
Copy file from your computer to a pod.
Note: The default folder in harmony-1 is /opt/versalex. You can use the full path if needed:
kubectl cp -n harmony temp.xml harmony-1:logs/temp.xml
Viewing events in the Harmony namespace:
kubectl get events -n harmony
Putting it all together
This section outlines the deployment flow, providing a reference for end-to-end container setup.
Secrets phase commands
kubectl create secret generic cleo-license-verification-code -n harmony --from-file=secrets/cleo-license-verification-code kubectl create secret generic cleo-license -n harmony --from-file=secrets/cleo-license kubectl create secret generic cleo-config-repo -n harmony --from-file=secrets/cleo-config-repo kubectl create secret generic cleo-runtime-repo -n harmony --from-file=secrets/cleo-runtime-repo kubectl create secret generic cleo-default-admin-password -n harmony --from-file=secrets/cleo-default-admin-password kubectl create secret generic cleo-system-settings -n harmony --from-file=secrets/cleo-system-settings
Initialization phase commands
kubectl apply -f harmony-init.yaml kubectl get pods -n harmony -o wide kubectl logs harmony-init-6kx6v -n harmony kubectl delete -f harmony-init.yaml
Running Harmony phase commands
kubectl apply -f harmony-run.yaml kubectl get pods -n harmony -o wide kubectl logs harmony-1 -n harmony kubectl exec -it harmony-1 -n harmony -- bash kubectl delete -f harmony-run.yaml
Docker Swarm
TBD
System migration
TBD
Known issues and limitations
System event logs
As noted in the Resource requirements section above, during the beta phase, Harmony log events will still be stored in Harmony's internal Elasticsearch server. These log events will be lost when the containers are torn down.
GitHub repository
Support for using GitHub as a configuration repository is planned but not yet available.
Protocol testing
Testing for some protocols is still in progress.
Comments
0 comments
Please sign in to leave a comment.