Cloud building blocks for data

⏱ 30 min

Hook

The disk on your team's reporting server fills up on a Friday afternoon. Adding space means ordering a bigger drive, waiting a week, scheduling downtime, and copying everything across. Meanwhile the dataset keeps growing. On a cloud platform the same problem is a one-line change, because the machine doing the computing and the place holding the data were never the same thing to begin with. Understanding why is the first building block.

Concept

A cloud data platform, no matter how elaborate it looks in a vendor diagram, is assembled from three primitives: compute, object storage, and networking. Learn these three and every managed service later reads as a convenient packaging of them.

Compute is rented processing power — a virtual machine you switch on by the second, or a serverless engine that runs your query and vanishes. The defining trait is that you pay only while it runs. Turn it off and the bill for it stops.

Object storage is the center of gravity for data in the cloud. It is a flat store of objects — files plus metadata — addressed by a key inside a named container (a bucket). It is cheap, effectively unlimited, and extremely durable, but it is not a filesystem and not a database: you get and put whole objects, you do not edit them in place. Almost every cloud dataset lives here first, and other services read it from here.

Networking is the wiring and the boundaries. It decides what can reach what, and — critically for your bill — when data crosses a boundary that costs money to cross.

The single most important structural idea is compute/storage separation. On the old server, the CPU and the disk were bolted together, so scaling one meant scaling both. In the cloud they are independent services. Your data sits in object storage continuously and cheaply; compute attaches to it only when there is work to do, then detaches.

That separation is why the Friday disk problem disappears: storage grows on its own without touching compute, and compute scales up for a heavy job then costs nothing again when it stops.

The boundaries those primitives live inside are regions and availability zones. A region is a geographic location — a city or area — where the provider runs data centers; a dataset physically resides in the region of the bucket you put it in. An availability zone is an isolated data center (or cluster of them) within a region, so a service spread across zones survives one zone failing. The rule that follows: data and the compute that reads it should sit in the same region, because moving data between regions — or out to the public internet — is billed as egress, which you will meet in Lesson 3.

Primitive	What it is	Billed on	Scales
Compute	Rented processing, on/off	Time it runs (or work it does)	Independently of storage
Object storage	Durable bucket of objects by key	GB stored per month, plus egress	Independently of compute
Networking	Boundaries and paths between them	Data crossing paid boundaries	With traffic and topology

🧠 Knowledge check

1. Why does compute/storage separation solve the "disk full on Friday" problem?

a) Object storage compresses data so it never fills upb) Storage grows on its own without changing or resizing the computec) Compute automatically deletes old data to free spaced) The provider adds physical disks to your server overnight

2. Which statement about object storage is correct?

a) It is a filesystem you edit files in place onb) It is a relational database with tables and indexesc) It is a flat store of whole objects addressed by a key in a bucketd) It only holds data while a compute instance is running

Worked example

Let me place a real dataset and count the boundaries it crosses, because that is where cost and latency hide. Suppose an analyst has a 40 GB export of clickstream Parquet files and wants to run a daily aggregation over it.

First decision: where does the data live? I put the bucket in the region closest to the analyst and to the rest of our platform — say a region in Mumbai, because that is where the app's other data already sits. Keeping everything in one region means later reads stay inside that region and avoid cross-region charges.

Second decision: what reads it? I do not provision a permanent server. The job runs once a day for a few minutes, so a permanent machine would bill 24 hours to do a few minutes of work. Instead I attach compute on demand — a serverless query engine or a short-lived job — in the same region as the bucket.

I can express the placement check as plain Python so the rule is unambiguous:

def crosses_paid_boundary(data_region, compute_region):
    """Return True if reading the data will incur cross-region egress."""
    return data_region != compute_region


bucket_region = "ap-south-1"      # where the 40 GB of Parquet lives
job_region = "ap-south-1"         # where the daily aggregation runs

if crosses_paid_boundary(bucket_region, job_region):
    print("Warning: cross-region read — data will be billed as egress")
else:
    print("Same region — no egress; compute attaches to storage locally")

Same region — no egress; compute attaches to storage locally

The pieces map straight onto the three primitives: object storage holds the 40 GB continuously and cheaply, compute attaches only for the daily run, and networking is quiet because nothing crossed a region boundary. Had I put the job in a European region "because that server was free," every daily run would drag 40 GB across regions and bill for it — the same computation, needlessly expensive.

Hands-on

Your turn, with a different scenario. The exercise below hands you a small fleet of datasets and candidate compute regions and asks you to decide, for each, whether a read stays local or crosses a paid boundary — then total the data that would be billed as egress. It runs in Python Arena, so there is nothing to install.

▶ Python Arenacd101-place-the-dataset

Open in Python Arena ↗

Success criteria: your function returns local for same-region pairs and egress for cross-region pairs, and your egress total counts only the datasets whose compute sits in a different region. The Arena validates your output against a hidden set of placements automatically.

Recap

• You can name the three primitives every cloud data platform is built from — compute, object storage, and networking — and say what each is billed on.
• You can explain compute/storage separation and why it lets storage and compute scale and bill independently.
• You can describe regions and availability zones as the boundaries a dataset lives in, and place compute in the same region as its data to avoid egress.

Next up: the managed data services that package these primitives — object stores, warehouses, and lakehouses — and the shared-responsibility line that decides how much of them you still have to run yourself.

🧠 Knowledge check

1. A daily job runs for five minutes over data in object storage. Why prefer on-demand compute over a permanent server?

a) Permanent servers cannot read object storageb) On-demand compute is billed only while it runs, not for the idle 23+ hoursc) On-demand compute stores the data more durablyd) Permanent servers must live in a different region from the data

2. Your bucket is in one region and you run the query from compute in another. What does this most directly cause?

a) The data is automatically replicated for freeb) The read crosses a region boundary and is billed as egressc) The query runs faster because of load balancingd) The bucket is converted into a database

📌 Key takeaways

• Compute, object storage, and networking are the three primitives every cloud data platform is assembled from. - Compute and storage are separate services, billed and scaled independently; storage is the durable center of gravity, compute attaches on demand. - A dataset lives in a region; keep compute in the same region to avoid egress charges when it reads the data.