How Integrations Work in CIC Cloud Edition

As we found in Overview of CIC Cloud Edition, Cleo Integration Cloud supports numerous types of integrations, including B2B integrations (with trading partners over API or standard protocols (AS2, SFTP, FTPS, email)), application integration with cloud applications (such as cloud marketplaces and eCommerce platforms), and back-end integrations with ground and cloud infrastructure. In this article, we will show you an end-to-end integration, starting with an external trading partner, progressing through data processing in Cleo Integration Cloud, and finally connecting to application infrastructure on the ground. Because there are multiple parts to the integration involving data movement and data processing, this article shows you each individual part first and then brings it all together to show you the end-to-end flow.

 Figure 1: How data is transferred in CIC

In this example, we will be looking at an inbound integration involving a trading partner sending a payload, such as a Purchase Order or Load Tender, to a back-end ERP. The inbound payload can be an EDI file (X12, EDIFACT or TRADACOMS), a non-EDI file (flat file, spreadsheet or XML) or a JSON payload (for API integration). CIC Cloud Edition leverages three primary types of objects for performing data movement of files (EDI or Non-EDI).

• Endpoints: An Endpoint represents a source or destination for data you want to move. For instance, you create Endpoints for each of your customers, partners, and back-end systems that you want to integrate. In the example above, we created an Endpoint for a customer, Acme, and for the ERP in the back end. You can create different types of Endpoints, including AS2, SFTP, FTP and email, depending on what protocol you intend for the Endpoint to either send or receive data over.
• Access Points: An Access Point allows a private server to communicate securely with CIC in the cloud, and is therefore required to create an Endpoint in a private data center (for example, in your back end). Access Points enable CIC to connect to those private Endpoints securely without requiring you to open any inbound ports. You create an Access Point by installing a CIC Agent on the private server. In the diagram above, we see that an Access Point is needed for CIC to connect (through firewalls) to the Endpoint representing the back-end ERP.
• Data Flows: A Data Flow connects two Endpoints--a source Endpoint and a destination Endpoint--and allows data to flow in one direction. Data Flows are used to connect external trading partner Endpoints to CIC, as well as to connect CIC to back-end Endpoints. So, connecting from a partner into CIC for processing and then on to your back-end system requires two Data Flows. If neither data inspection nor data transformation is needed, it is also possible to build Data Flows directly from the partner Endpoint to the back-end Endpoint.

Figure 2: How is data processed 

When CIC's Integration Engine receives data, it performs a series of steps to inspect, route, and transform the data. The illustration above shows a typical sequence, although the sequence might vary somewhat depending on the goal of the integration.

• Receive: Data is first received by a Cloud Monitor, which waits for incoming data. The inbound payload is then passed to the next stage, where the structure of the data is inspected.
• Inspect: CIC employs a rich set of options for building Data Schemas that allow you to define the structure of the expected inbound data. The schemas allow CIC to verify that the inbound payload is structured as expected (for example, a valid EDI document) and to parse the data for subsequent processing.
• Transform: Once the inbound data is understood and parsed in the Inspect phase, CIC can transform it as needed to match the format that the destination Endpoint (the ERP in this case) expects. For instance, the file can be transformed from an EDI X12 file into a flat file to be ingested by the ERP. This is commonly known as Mapping.
• Route: After CIC transforms the data, it determines which business processes to run to handle any orchestration required by the integration. Routes are used to send the inbound payload to the proper business processes. 
• Process: During the Process phase, any orchestration required for the integration is performed by Business Processes defined by the integration designer.
• Send: When processing is complete, the payload is sent to a Cloud Adapter that then sends it on to the destination Endpoint.

Figure 3: Putting it all together: End-to-end flow of a CIC Integration

The example above shows the flow of a CIC integration with all the basic parts. The top section of the diagram indicates the desired outcome of the integration, for example, allowing Acme to send Orders to a company's MS Dynamics ERP in their on-premise data center. 

As discussed earlier, the first step is to create an Endpoint for the source and one for the destination. In this case, the source is Acme and we assume Acme wants to connect via SFTP. So, we create an SFTP Endpoint for Acme. The destination is the ERP. CIC can connect to any on-premise application (such as MS Dynamics) using file-based integration or database integration. In this example, we assume file-based integration, which means creating a file-system Endpoint associated with a folder on a server in a private data center where CIC can 1) put files for MS Dynamics to pick up and 2) pick up files placed there by MS Dynamics. In addition, because this Endpoint is connecting to a private server, we create an Access Point to enable communication between CIC and the private server. 

Next, we build the data flows from the partner to CIC and from CIC to the back-end ERP. You will notice a new object called Transformation Endpoint. The Transformation Endpoint is an out-of-the-box Endpoint created to allow you to easily build data flows from and to CIC from various Endpoints. So, now we can create Data Flow 1 from the SFTP Acme Endpoint to the Transform Endpoint and Data Flow 2 from the Transform Endpoint to the ERP Endpoint. 

Everything discussed so far can be built in the Cleo Integration Cloud central, web-based console for managing CIC. This is where you will:

• Connect CIC to your ecosystem: Build Endpoints, Access Points, and Data Flows to connect CIC to your ecosystem.
• Monitor CIC: The CIC Cockpit includes rich dashboards for getting business insights, as well as detailed transaction monitoring tools for end-to-end visibility into your integrations and troubleshooting. 
• Administer CIC: Manage CIC users, configure branding, and perform other administrative tasks.
• Download CIC Studio: The CIC Studio, described below, can be downloaded from the CIC web console

CIC Studio

CIC Studio is a desktop extension of CIC for building the core parts of your integrations, including data schemas, maps, routing, and orchestration. Figure 3 above indicates the parts of the integration you build in CIC Studio, including the Cloud Monitor and Cloud Adapter. The Cloud Monitor needs to reference the source Endpoint (SFTP Acme Endpoint) and the Cloud Adapter needs to reference the destination Endpoint (fIle-system ERP Endpoint). Through this linkage, CIC knows which set of integration components to run when it receives data from the SFTP Acme inbound data flow and, similarly, which data flow to call after processing within CIC is done. Once integrations are built in CIC Studio, you can deploy them to your CIC Production Instance.

Figure 4: Monitoring integrations end-to-end

CIC Cockpit

The CIC Cockpit provides a rich set of tools for monitoring and gaining insights from your integrations. One such area is the Jobs section under the Activity tab. A Job represents an instance of a single run of an integration. In Figure 4 above, the job represents a single run of the integration we designed in Figure 3. You can see the status of each stage of the integration in the step-by-step integration diagram and then can click on any step to see the content and detailed logs that describe what is occurring at each step.