The 10 Pillars of Cloud Data Engineering: A Field Guide for Data and AI Teams

Modern cloud data engineering rests on ten capabilities: storage, ingestion, processing, warehousing, the lakehouse, orchestration, streaming, governance, observability and cost control. Every major cloud (AWS, Azure, GCP) offers a service for each. The names differ, the shapes rhyme. What separates teams that ship AI from teams that ship slideware is not which cloud they picked. It is how honestly they have built each pillar.

We run this stack in production at KeepFlying, where the data describes aircraft and the errors have serial numbers. That colours our view of which pillars deserve your attention first. Here is the map, with opinions attached.

Pillar 1: Cloud storage

Stores raw, processed and curated data at scale. Amazon S3, Azure Data Lake Storage Gen2, Google Cloud Storage.

The commodity pillar, and the one people overthink. Object storage is cheap and durable everywhere. The decision that matters is layout, not vendor: zone your raw, cleaned and curated data separately from day one, because untangling them later is a quarter of somebody's year.

Pillar 2: Data ingestion

Moves data from apps, databases, APIs, files and SaaS tools into the cloud. AWS Glue, Amazon AppFlow, AWS DMS. Azure Data Factory, Synapse Pipelines. Cloud Data Fusion, Datastream, BigQuery Data Transfer.

Ingestion is where data quality is won or lost, because whatever you tolerate here becomes everyone's problem downstream. In our world the hard sources are not APIs. They are scanned airworthiness PDFs from 1998 and utilization reports typed by a lessee. If your ingestion plan only covers structured sources, it covers the easy half.

Pillar 3: Data processing

Cleans, transforms, joins and prepares data for analytics. AWS Glue, Amazon EMR, AWS Lambda. Azure Databricks, Synapse Spark, Azure Functions. Dataflow, Dataproc, Cloud Functions.

Spark in some wrapper, on every cloud. The real question is idempotency: can you re-run any job on any day and get the same answer? If not, your pipeline is a rumour mill. We hold processing to the same standard as the records it touches, since a transformation you cannot reproduce is a number you cannot defend.

Pillar 4: Data warehousing

Stores structured, business-ready data for SQL analytics and reporting. Amazon Redshift, Amazon Athena. Azure Synapse Analytics, Microsoft Fabric Warehouse. BigQuery.

Warehouses are where finance meets engineering, and the trust boundary sits exactly there. A warehouse the CFO does not believe is an expensive cache. Reconcile it against the system of record on a schedule, publish the variance, and belief follows.

Pillar 5: Data lakehouse

Combines data lake flexibility with warehouse-style structure and performance. Amazon S3 with Glue Data Catalog, EMR and Athena. Microsoft Fabric Lakehouse, OneLake, Azure Databricks. BigQuery, Dataproc.

The pillar we lean on hardest. Aviation data is a mix of tabular records, documents, telemetry and images, and forcing that into a pure warehouse loses the evidence. The lakehouse keeps source documents and derived tables in one governed place, which is what lets an answer trace back to the scanned page it came from. That traceability, not query speed, is the reason to adopt one.

Pillar 6: Workflow orchestration

Schedules and manages pipelines, jobs and dependencies. Amazon MWAA, Step Functions, Glue Workflows. Azure Data Factory, Synapse Pipelines, Fabric Data Pipelines. Cloud Composer, Workflows.

Orchestration is unglamorous until 3am, when it is the only thing that matters. Two habits pay for themselves: make every dependency explicit rather than temporal (never "runs at 6 so it finishes by 7"), and treat backfills as a first-class operation you rehearse, not an emergency you improvise.

Pillar 7: Streaming data

Processes real-time events from apps, logs, transactions and IoT systems. Amazon Kinesis, Amazon MSK. Event Hubs, IoT Hub, Stream Analytics. Pub/Sub, Dataflow.

Honest take: most teams need streaming later than they think. Batch that runs hourly and never lies beats a stream that occasionally drops events. Adopt streaming when a decision genuinely changes within the hour, and not before. Flight events qualify. Monthly lease reporting does not.

Pillar 8: Data governance and security

Manages access, classification, lineage, ownership and policy enforcement. Lake Formation, IAM, Glue Data Catalog, Amazon DataZone. Microsoft Purview, Entra ID, Key Vault. Dataplex, IAM, Cloud KMS.

The pillar that decides whether your AI is deployable. A model trained on data nobody can trace is a liability with a login page. In regulated industries this is not a compliance afterthought; lineage is the product. It is also why we run FinTwin inside existing governance boundaries rather than copying data out of them: an insight you cannot audit is an insight legal will not let you use.

Pillar 9: Monitoring and observability

Tracks pipeline failures, latency, freshness and overall system health. CloudWatch, CloudTrail. Azure Monitor, Log Analytics. Cloud Monitoring, Cloud Logging.

Monitor the data, not just the infrastructure. A pipeline can be green while writing garbage. Freshness checks, row-count deltas and schema drift alerts catch the failures that dashboards of CPU graphs never will. If you only track one metric, track time-since-last-good-data per table.

Pillar 10: Cost optimization

Controls storage, compute, query and pipeline costs across cloud data systems. Cost Explorer, AWS Budgets. Cost Management, Azure Advisor. Cloud Billing, Recommender.

Cost is a data quality signal wearing a finance hat. Runaway query bills usually mean somebody is scanning raw data because the curated layer is not trusted or not there. Fix the modelling and the bill follows. Tag by pipeline from day one; untagged spend is unexplainable spend.

The pillar the diagram cannot show: AI readiness

Ten pillars build the machine. None of them guarantee the machine is fed. Our records maturity work in aviation keeps finding the same thing: the architecture is fine and the data is not. Records unscanned or unreconciled, utilization unverified, ownership unassigned. That is why an AI readiness effort has to score the data itself (completeness, traceability, validation, human sign-off) and not just tick off services. The pillars are necessary. Ready data is what makes them worth having.

FAQ

What are the 10 pillars of cloud data engineering?

Cloud storage, data ingestion, data processing, data warehousing, the data lakehouse, workflow orchestration, streaming data, data governance and security, monitoring and observability, and cost optimization. Each has equivalent managed services on AWS, Azure and GCP.

Which pillar should a team invest in first?

Governance and ingestion, ahead of processing power. Compute is easy to buy. Trustworthy, traceable data is not, and every downstream pillar inherits whatever quality ingestion lets in.

What is the difference between a data warehouse and a data lakehouse?

A warehouse stores structured, business-ready tables for SQL analytics. A lakehouse adds warehouse-style structure and performance on top of open data lake storage, so tabular data and source documents live in one governed system. For evidence-heavy domains such as aviation, the lakehouse's ability to keep derived numbers linked to source records is the deciding feature.

How does cloud data engineering relate to AI readiness?

The pillars are the infrastructure. AI readiness is the state of the data inside them: collected, reconciled, validated and traceable, with humans approving where judgement matters. Strong pillars with unready data still produce unreliable AI.

Do the pillars differ across AWS, Azure and GCP?

The capabilities are equivalent; the service names differ. Ingestion is Glue or Data Factory or Data Fusion depending on cloud. Architecture decisions transfer across clouds far better than service-level habits do.

KeepFlying builds FinTwin, the financial twin for aviation, running on this stack in production. ISO 27001 certified since 2022. If you want to know whether your data is as ready as your architecture, ask us about the records maturity assessment.