Sources
Loading data from internal and external locations.
Last updated
Was this helpful?
Loading data from internal and external locations.
Last updated
Was this helpful?
Sources insert ID profile data to your DCN. The types of data that sources send includes IDs, links between IDs, and traits associated with IDs. There are several types of sources you can configure such as the Optable SDKs & API, Mailchimp and Snowflake via your DCN UI and CLI.
File sources are a structured data source that can be uploaded to your DCN via the File Upload source. These files can be in CSVs or JSON format and must be in a specific way so that all identifiers and their respective traits can be ingested by your DCN. You can learn more about the file upload source here.
Sites and applications are considered SDK sources since data is sent in real-time from your site visitors and application users by way of the Optable SDKs. There are currently three SDK types including iOS, Android and JavaScript SDKs. You can learn more about SDK sources here.
An API source can be used to stream ID profiles from any system, server to server to your Optable DCN. You can create as many API sources as you like, to represent each of the systems or databases from which you are sourcing data. Examples of types of systems that you can bring in data from include DMPs, CDPs, CRMs, bespoke databases, etc. You can learn more about API sources here.
Cloud apps, such as Mailchimp, Shopify or Sendgrid enable you to bring in data from your favorite third-party tools & platforms. These sources directly connect to your Optable DCN and generally require little to no development resources.
You can connect directly to your Snowflake data warehouse to easily load data into your DCN from a Snowflake view or a table using the tabular data schema. You can learn more about Snowflake sources here.
You can connect directly to your BigQuery data warehouse to easily import data into your DCN from a BigQuery dataset, view or a table using the tabular data schema. You can learn more about BigQuery sources here.
When creating a source, you can select the frequency at which your DCN will attempt to retrieve new data from cloud apps such as Mailchimp, Shopify or Sendgrid. The default ingestion frequency is 24 hours, but you can set it to either 6, 12 or 24 hour refresh intervals at source create.
Other external sources, such as BigQuery and Databricks support on-demand (now), hourly, daily & weekly ingestion schedules.
You can archive a source by clicking the context menu and selecting "Archive". To archive a source you must first remove it from any referenced query audiences, or archive those audiences completely.
Once archived, source data ingestion will be paused, and data will be removed from all insights reports. You can restore an archived source by clicking the "Archived" filter and selecting "Restore" from the respective source's context menu.
Archived sources may take up to 1 hour to be completely removed from global insights.
You can view a history of all previous source uploads, ingestions and errors by navigating to your source in Data > Sources and clicking on the "History" tab.
Your DCN can automatically delete ID profile data after some amount of time has elapsed since the last time it was loaded. The maximum elapsed time to deletion is called the expiry. By default, SDK sources have an expiry of 30 days, meaning that data loaded via SDK is automatically deleted after 30 days have elapsed since the last time it was loaded.
Expiry is disabled for API sources by default, but you can optionally configure an expiry duration in the source's settings.
