Architecture Diagrams with Python: Version Control Your System Design For AWS, Azure, GCP, Kubernetes, Alibaba Cloud and Oracle Cloud

2024-11-07

Dragging boxes and drawing lines to explain your architecture is a bit… 1999?

What about designing your cloud architecture directly in Python code?

That would be perfect for quickly prototyping or documenting your system—without any clunky design tools.

Even better, if it could already support major cloud providers like AWS, Azure, GCP, and even works with on-premise setups, SaaS platforms, and popular frameworks.

And since you would be writing your architecture as code, you could keep track of changes in your version control system—no more hunting down outdated architecture docs!

Let’s see how you can do all of that with Python!

Getting Started with Diagrams

You’ll need Python 3.7 or higher, so check your Python version first.

diagrams uses Graphviz to render visuals, so go ahead and install that too.

If you are using macOS, you can install Graphviz with brew:

brew install graphviz

Then, install diagrams:

# using pip
pip install diagrams  

# or using pipenv  
pipenv install diagrams  

# or with poetry  
poetry add diagrams

You are ready to go!

From Code to Diagrams

Let’s get into some examples because code is always more fun than words.

Grouped Workers on AWS

Here’s a quick setup to show a load balancer distributing tasks across multiple EC2 instances:

 from diagrams import Diagram  
 from diagrams.aws.compute import EC2  
 from diagrams.aws.database import RDS  
 from diagrams.aws.network import ELB  
 
 with Diagram("Grouped Workers", show=False, direction="TB"):  
 ELB("lb") >> [EC2("worker1"),  
                EC2("worker2"),  
                EC2("worker3"),  
                EC2("worker4"),  
                EC2("worker5")] >> RDS("events")

Not bad for a start.

Clustered Web Services

This example shows web services distributed across an ELB with a DNS layer and connected databases, all in one go.

 from diagrams import Cluster, Diagram  
 from diagrams.aws.compute import ECS  
 from diagrams.aws.database import ElastiCache, RDS  
 from diagrams.aws.network import ELB  
 from diagrams.aws.network import Route53  
 
 with Diagram("Clustered Web Services", show=False):  
 dns = Route53("dns")  
 lb = ELB("lb")  
 
 with Cluster("Services"):  
      svc_group = [ECS("web1"),  
                     ECS("web2"),  
                     ECS("web3")]  
 
 with Cluster("DB Cluster"):  
      db_primary = RDS("userdb")  
      db_primary - [RDS("userdb ro")]  
 
 memcached = ElastiCache("memcached")  
 
 dns >> lb >> svc_group  
 svc_group >> db_primary  
 svc_group >> memcached

Getting better!

Event Processing on AWS

Here’s how you might handle event processing with an event flow and queues.

 from diagrams import Cluster, Diagram  
 from diagrams.aws.compute import ECS, EKS, Lambda  
 from diagrams.aws.database import Redshift  
 from diagrams.aws.integration import SQS  
 from diagrams.aws.storage import S3  
 
 with Diagram("Event Processing", show=False):  
 source = EKS("k8s source")  
 
 with Cluster("Event Flows"):  
      with Cluster("Event Workers"):  
           workers = [ECS("worker1"),  
                     ECS("worker2"),  
                     ECS("worker3")]  
 
      queue = SQS("event queue")  
 
      with Cluster("Processing"):  
           handlers = [Lambda("proc1"),  
                     Lambda("proc2"),  
                     Lambda("proc3")]  
 
 store = S3("events store")  
 dw = Redshift("analytics")  
 
 source >> workers >> queue >> handlers  
 handlers >> store  
 handlers >> dw

Message Collecting System on GCP

For those of you who use Google Cloud, here’s how to collect messages from IoT sources into various Google services.

 from diagrams import Cluster, Diagram  
 from diagrams.gcp.analytics import BigQuery, Dataflow, PubSub  
 from diagrams.gcp.compute import AppEngine, Functions  
 from diagrams.gcp.database import BigTable  
 from diagrams.gcp.iot import IotCore  
 from diagrams.gcp.storage import GCS  
 
 with Diagram("Message Collecting", show=False):  
 pubsub = PubSub("pubsub")  
 
 with Cluster("Source of Data"):  
      [IotCore("core1"),  
      IotCore("core2"),  
      IotCore("core3")] >> pubsub  
 
 with Cluster("Targets"):  
      with Cluster("Data Flow"):  
           flow = Dataflow("data flow")  
 
      with Cluster("Data Lake"):  
           flow >> [BigQuery("bq"),  
                     GCS("storage")]  
 
      with Cluster("Event Driven"):  
           with Cluster("Processing"):  
                flow >> AppEngine("engine") >> BigTable("bigtable")  
 
           with Cluster("Serverless"):  
                flow >> Functions("func") >> AppEngine("appengine")  
 
 pubsub >> flow

Next cohort will start soon! Reserve your spot for building full-stack GenAI SaaS applications

Exposed Pod with 3 Replicas on Kubernetes

If you’re working with Kubernetes, here’s an example of exposing a pod with replicas.

 from diagrams import Diagram  
 from diagrams.k8s.clusterconfig import HPA  
 from diagrams.k8s.compute import Deployment, Pod, ReplicaSet  
 from diagrams.k8s.network import Ingress, Service  
 
 with Diagram("Exposed Pod with 3 Replicas", show=False):  
 net = Ingress("domain.com") >> Service("svc")  
 net >> [Pod("pod1"),  
           Pod("pod2"),  
           Pod("pod3")] << ReplicaSet("rs") << Deployment("dp") << HPA("hpa")

Core Concepts and Primitives Commonly Used in Diagrams

In these examples, we see some core concepts and primitives commonly used in diagrams.

Let’s break down the key concepts and patterns across these examples.

1. Core Elements: Nodes and Connections

Nodes: Nodes are the individual components representing cloud services, applications, or infrastructure components like EC2, RDS, ELB, etc., from AWS, or BigQuery, Dataflow, etc., from GCP. Each node can have a label that describes its function or identifier, making it easy to recognize in the diagram (e.g., ECS('web1')).
Connections: Connections between nodes often use >> and << symbols. These operators depict the data or workflow direction. For example, ELB('lb') >> [EC2('worker1'), EC2('worker2')] shows data flowing from the load balancer to multiple EC2 instances. This visual flow of connections shows how services interact within the architecture.

2. Diagram: The Overall Structure

Each example starts by defining a Diagram object, which sets up the entire visualization and provides context. For instance, Diagram('Grouped Workers', show=False, direction='TB') initializes a diagram with a title and some display settings (show=False hides the automatic display, and direction='TB' specifies a top-to-bottom layout).
The Diagram class acts as a wrapper that organizes all nodes, connections, and clusters, rendering them into a single cohesive architecture.

3. Clusters: Grouping Related Components

Clusters are used to group related nodes under a shared logical grouping. This is especially useful for visualizing layers or tiers of services within a larger architecture. For example, in Cluster('Services'), a group of ECS instances is enclosed, representing a service tier. Similarly, we see Cluster('DB Cluster'), Cluster('Event Workers'), and so forth.
Clusters often help distinguish different architectural domains, like web services, databases, or processing units, making complex systems easier to follow.

4. Cloud Provider Modules: AWS, GCP, Kubernetes

Different modules (aws, gcp, k8s) allow you to easily represent components specific to each platform. For example:
AWS Module: from diagrams.aws.compute import ECS, from diagrams.aws.network import ELB
GCP Module: from diagrams.gcp.analytics import BigQuery
Kubernetes Module: from diagrams.k8s.compute import Pod, from diagrams.k8s.network import Ingress
These modules provide ready-to-use classes for individual services, making it easy to create detailed, cloud-specific architectures.

5. Directional Workflow

Most examples use directional operators (>> or <<) to clarify data flow. This approach mirrors the natural progression of requests or data movement within a system:
In Event Processing, data flows from EKS to event workers, then to an SQS queue, and finally to Lambda handlers.
In Message Collecting, IoT sources pass data to PubSub, which then flows through Dataflow to various targets like BigQuery or AppEngine.
The directional flow visually communicates the sequence or dependency between components in the system.

6. High Availability and Scaling Concepts

Replication: Some examples show replicated instances to illustrate horizontal scaling or fault tolerance. For instance, [EC2('worker1'), EC2('worker2'), EC2('worker3')] in the Grouped Workers example represents a load-balanced set of EC2 instances.
Pod Replicas in Kubernetes: In the Kubernetes example, multiple pods (e.g., Pod('pod1'), Pod('pod2')) show replication for high availability.
Load Balancers: Load balancers (e.g., ELB in AWS, Ingress in Kubernetes) demonstrate how traffic distribution is managed across multiple instances.

Wrapping Up

The diagrams library gives you an awesome, code-first way to document, share, and version control your architecture diagrams.

You can find all these examples and more on official website.

So next time, instead of whiteboarding (again), try out Diagram as Code.

Happy coding, and let me know how your first Diagram-as-Code adventure goes!

Bonus Content : Building with AI

And don’t forget to have a look at some practitioner resources that we published recently:

Llama 3.2-Vision for High-Precision OCR with Ollama

LitServe: FastAPI on Steroids for Serving AI Models — Tutorial with Llama 3.2 Vision

Run FLUX Models Locally on Your Mac!

GOT-OCR2.0 in Action: Optical Character Recognition Applications and Code Examples

Thank you for stopping by, and being an integral part of our community.

Happy building!

Back to All Posts