Azure Blob Storage

Loading data from your Azure Blob Storage container.

To integrate your existing Azure Blob Storage (ABS) container with your DCN, you can add an Azure Blob Storage source. This allows you to map data such as identifiers, associated identifiers, and traits from a container containing any of the supported file types formatted in the tabular data format.

By directly connecting your ABS container with your DCN, you can seamlessly integrate your data and gain valuable insights without the need for complex data migrations or transformations.

Pre-requisites:

• Have an Azure account with permissions to manage storage containers.

• Create an Azure Blob Storage container specific to this source.

• Create credentials that will be used by your DCN to access the previously created ABS container.

• Load file(s) into the bucket that are based on the tabular data schema.

Steps:

1. Open the source create form for Azure Blob Storage and name your source.

2. Enter the Container URL (For example: https://mystorageaccount.blob.core.windows.net/mycontainer/)

3. Set the expiry & ingestion frequency.

Setting up an Azure Blob Storage source.

1. In the next section, enter the previously created credentials that will grant your DCN read access to your container.

The credentials - <tenant-id>, <client-id>, and <client-secret> - are required for Azure. They should correspond to the ID and secret of a service principal in Azure Active Directory (Azure AD) that has at least read permissions to the specified Blob Storage container.

Enter your credentials.

1. Click Create

With these steps completed, your DCN should now have access to the Azure Blob Storage container you entered and will start to ingest files based on the ingestion frequency you set at creation.

Notes:

Your DCN will automatically check for new files in the storage container at the frequency that you set at source creation. If your DCN finds multiple files, it will trigger multiple ingestions simultaneously.

Cloud storage sources have a default rejection threshold of 100%, meaning that even if your file contains 99% errored records, your DCN will ingest the remaining 1%. However, if a file is 100% invalid, it will return an error for that specific file and will continue to attempt to ingest the rest of the files in the container.

If you modify a file in the container, your DCN will see it as modified and attempt to re-ingest it. Files are deemed "New" or "Updated" if the "Last Modified" timestamp has changed.