How to Reduce SemaphoreCI Network Congestion

Introduction

SemaphoreCI is a cloud-native CI/CD platform that enables efficient and reliable build pipelines. Although the pipelines usually run without issues on SemaphoreCI cloud, network congestion between the EU-Central and US-East regions has been a recurring concern, impacting build performance, artifact transfers, and overall developer experience.

In this blog post, we'll explore how to optimize SemaphoreCI's performance by addressing network congestion, implementing region affinity, optimizing caching strategies, using regional mirrors, and monitoring performance metrics.

Understanding the Problem

Cross-continental data transfers between EU-Central and US-East regions introduce inherent latency challenges due to geographic distance and network topology.

  • Geographic latency: ~80-120ms baseline latency for transatlantic connections
  • Network congestion during peak hours increases transfer times
  • Artifact uploads/downloads become bottlenecks in CI/CD pipelines
  • Docker image pulls from cross-region registries slow down builds
  • Dependency installation from remote package managers adds overhead
# Example: Measuring network latency between regions
$ ping eu-central-semaphore-endpoint
64 bytes from eu-central: time=102ms
64 bytes from eu-central: time=115ms
64 bytes from eu-central: time=98ms

# Average latency: ~105ms (baseline overhead for cross-region operations)

Root Causes

Several factors contribute to network congestion and performance degradation:

1. Region Selection Mismatches

  • Source code repositories hosted in US-East
  • CI/CD workers running in EU-Central
  • Docker registries located in different regions than workers
  • Package registries (npm, PyPI, Maven) served from distant locations

2. Shared Dependency Caching Challenges

  • Cache hits requiring cross-region transfers
  • Large artifact sizes amplifying transfer overhead
  • Inefficient cache warming strategies

3. Network Routing Inefficiencies

  • Suboptimal routing paths between cloud providers
  • Internet backbone congestion during peak usage
  • Lack of direct peering agreements between regions

Impact Analysis

Network congestion manifests in measurable performance degradation:

  • Build Time Increases
    • Cross-region builds can take 2-5x longer than same-region builds
    • Sometimes a build fails due to timeout errors
  • Cost Implications
    • Extended build times increase compute costs and data transfer charges
    • Developers may need to wait longer for builds to complete. Some are even unable to do their work.
  • Developer Experience
    • Slower feedback loops reduce productivity and increase frustration
    • Repeated hours-long outages can cause developers to try different solutions
  • Pipeline Reliability
    • Timeout errors and intermittent failures during network congestion
    • Builds may need to be rerun multiple times

Mitigation Strategies

Although running a self-hosted SemaphoreCI agent in the US-East region can help alleviate network congestion, it's sometimes cost-prohibitive to teams since it incurs additional infrastructure costs and licensing fees.

Strategy 1: Local Caching Optimization

Maximize cache hits within the same region to avoid cross-region transfers:

# Use SemaphoreCI's cache feature with regional awareness
- cache restore
- cache store  # Store dependencies in region-local cache

# Example: Caching node_modules
blocks:
  - name: Dependencies
    task:
      prologue:
        commands:
          - checkout
          - cache restore node-modules-$SEMAPHORE_GIT_BRANCH-$(checksum package-lock.json)
      jobs:
        - name: Install
          commands:
            - npm ci
      epilogue:
        commands:
          - cache store node-modules-$SEMAPHORE_GIT_BRANCH-$(checksum package-lock.json) node_modules

Strategy 2: Dependency Mirroring

Use regional package mirrors to reduce cross-region dependency downloads:

# .npmrc - Use regional npm registry mirror
registry=https://eu.registry.npmjs.org/

# Maven pom.xml - Use regional Maven repository
<repositories>
  <repository>
    <id>eu-central-maven</id>
    <url>https://eu-central.repo.maven.apache.org/maven2</url>
  </repository>
</repositories>

# Python pip.conf - Use regional PyPI mirror
[global]
index-url = https://eu.pypi.org/simple/

Strategy 3: Docker Registry Optimization

Configure Docker to pull images from region-local registries:

# Use regional Docker registry mirrors
blocks:
  - name: Docker Build
    task:
      jobs:
        - name: Build Image
          commands:
            - checkout
            # Pull from regional registry
            - docker pull eu.gcr.io/my-project/base-image:latest
            - docker build -t my-app:$SEMAPHORE_WORKFLOW_ID .
            # Push to regional registry
            - docker push eu.gcr.io/my-project/my-app:$SEMAPHORE_WORKFLOW_ID

Monitoring and Diagnosis

Track network performance to identify and address congestion issues:

  • Monitor build duration trends across different regions
  • Track artifact upload/download times in pipeline logs
  • Use time command to measure individual operation latency
  • Set up alerts for builds exceeding performance thresholds
  • Review SemaphoreCI's workflow performance metrics dashboard
  • Use network diagnostic tools (traceroute, mtr) to identify routing issues

Best Practices Summary

  • Configure region affinity to match your infrastructure location
  • Leverage SemaphoreCI's caching system for region-local dependency storage
  • Use regional mirrors for package registries and Docker images
  • Monitor build performance metrics to detect congestion early
  • Consider multi-region strategies for globally distributed teams
  • Optimize artifact sizes to minimize transfer overhead
  • Use parallel jobs within the same region rather than cross-region dependencies
  • Regularly review and update cache strategies as dependencies evolve

Conclusion

Network congestion between SemaphoreCI's EU-Central and US-East regions can significantly impact CI/CD performance. By implementing region affinity, optimizing caching strategies, using regional mirrors, and monitoring performance metrics, teams can mitigate these challenges and maintain fast, reliable build pipelines. The key is understanding your infrastructure topology and configuring SemaphoreCI to minimize cross-region data transfers wherever possible.

← Back to Guides