Ingest Streaming Data with Salesforce Data Cloud Ingestion API and MuleSoft

The Salesforce Data Cloud simplifies the complex process of integration and ingestion of streaming data through very easy to use tools.

I am documenting below my work on a mini-project (my github repo is included at the end).

A Step by Step Walkthrough of How I Configured Streaming Ingestion

In this post I have used the terms CDP and Data Cloud interchangeably.

  • Defined Ingestion API from CDP Setup
  • Created & uploaded a YAML file for the source database table (i.e., MySQL database on Heroku). The YAML file is available in the Resources directory of this repository.
  • The DLO in CDP created when you upload the YAML schema – the schema is essentially the schema (table definition) of the external database object.
  • Created a Data Stream which corresponds to the DLO that is generated.
  • I now have a Data Lake Object in which the data incoming from MySQL database table will land.
  • Created a custom Data Model Object
  • Created a data mapping from the Data Lake Object to my Data Model Object
  • Defined the MuleSoft Flow using Mule Anypoint Studio (The JAR file is available in the resources directory of this repository).
  • To authenticate Mule Connector with CDP I defined a Connected App in CDP – this connected app then will have the Consumer Key and Consumer Secret that you can use in Mule.

Illustrations

The following illustrations explain the steps for loading data into CDP with MuleSoft CDP Connector. I have used the streaming feature of the ingestion API. The same Mule Connector can be leveraged for both batch as well as streaming capabilities of the API.
CDP Salesforce Data Cloud Steps

CDP Data Cloud Setup Screens

My GitHub Repo for this Project

This contains my Mule jar file and my example YAML file I used for the Ingestion API: https://github.com/innovationworkshops/datacloudportal

Caution

This is not official Salesforce documentation. This is meant as a quick test/demo only. Not for production environments. Please use the official Salesforce & MuleSoft documentation for your project.

What is Salesforce Data Cloud: https://help.salesforce.com/s/articleView?id=sf.customer360_a.htm&type=5

Customer Data Platform Developer Guide

MuleSoft CDP Connector