Azure Data Factory

Limitation in low latency scenarios

Further Azure Data Factory Limits.

• Activities are the individual steps performed
• Pipelines are a logical group of activities
• Triggers define when a pipeline will run
• Integration runtime is the compute infrastructure
• Datasets are the actual representation of the data
• Linked services thell where to find the data

Types of Integration Runtime for Azure Data Factory:

• Azure IR: for data movement between public endpoints
• Self-Hosted IR: for connection to private and on-premises resources
• Azure-SSIS IR: Exclusively for executing SSIS packages

Linked Services are similar to connection strings.

Two types:

• Data Stores
• External Compute Services

Datasets are about the data structure.

Datasets are used in activities as data inputs and outputs.

Examples

• Tables in SQL Database
• Files in Blob Storage

Triggers are the Data Factory components that initiate the execution of a pipeline. They define when a pipeline needs to run.

Types of triggers

• Schedule: focus on ON/AT (on Monday, at midnight, the last day of the month)
• Tumbling Window: focus on EVERY (every 4 hours)
• Event-based: Fire based on an event (file arrival)