Azure Data Engineering interview questions

A cloud service to orchestrate and move data with pipelines, activities, linked services and datasets.

A linked service is the connection (like a connection string). A dataset points to specific data on that connection, like a table or folder.

The compute ADF uses: Azure IR for cloud data, self-hosted IR for on-premises or private networks.

Keep a control table of sources and targets, read it with Lookup, loop with ForEach and pass parameters to a generic Copy activity.

Store a watermark (last modified date or ID), load only newer rows, then update the watermark.

Schedule, tumbling window (with dependencies and backfill), storage event and custom event triggers.

Azure Blob Storage with a hierarchical namespace, built for analytics with folder-level ACLs.

RBAC grants access at account or container level. ACLs give fine-grained access on folders and files.

Zones like raw/bronze, curated/silver and serving/gold, partitioned by date where it helps, using Parquet or Delta.

An identity Azure manages for a service, so it can access other resources without storing passwords.

To store secrets, keys and connection strings securely and reference them from ADF and Databricks.

Serverless queries files in the lake and you pay per data scanned. Dedicated is a provisioned warehouse you pay for while it runs.

A streaming ingestion service for millions of events per second, often read by Stream Analytics or Databricks.

Use the Databricks Notebook or Job activity with a linked service, passing parameters and using job clusters.

Retries on activities, failure paths with alerts, logging run details to a table and Azure Monitor alerts.

Keeping history by adding a new row for each change, with valid_from, valid_to and an is_current flag.

Git integration for development, publish creates ARM templates, and Azure DevOps or GitHub Actions deploy them to test and prod with parameters.

Use parallel copies, partition the source, choose the right integration runtime and avoid many tiny files.

Parquet is columnar, compressed and has a schema, so it is much faster and cheaper for analytics. CSV is simple but slow.

ADF ingests from sources into ADLS bronze, Databricks cleans to silver and builds gold, Synapse or Fabric serves data, Power BI reports, with Key Vault, monitoring and CI/CD around it.