Skip to main content

Preparing for high-load events

Overview​

Preparing your system for high-load events, such as NFT mints or token airdrops, involves a strategic approach to testing, scalability, and reliability. By focusing on realistic load simulations, scalable infrastructure, and robust monitoring, you can ensure a seamless experience for users during critical events. This page outlines essential strategies and best practices, emphasizing the importance of advance preparation and continuous performance optimization.

Core principles​

Thorough testing​

  • Calls to partner APIs: focus on connection, authentication, error handling, data validation, and synchronization. Simulate high-load scenarios to assess your system’s capacity and response times.
  • On-chain interaction: use the Saigon testnet to conduct minting simulations and on-chain load tests, leveraging Web3 libraries such as Ethers.js and Web3.js.
  • Bot and attack prevention: implement measures such as social login verification, smart contract (SC) signatures, rate limiting, Cloudflare protection, and CAPTCHAs to distinguish between human users and bots. This involves setting rate limits to ensure that one IP cannot make more than 5-10 requests per second or 60 requests per minute and considering the use of Cloudflare for additional protection.
  • Handling high traffic: stress-test your infrastructure using Locust for API stress testing and the Saigon testnet with Web3 libraries for on-chain interactions. Be mindful of the rate limits of external services, such as RPC and the X API, and ensure backend scalability to manage traffic surges.

Scalable infrastructure​

  • DNS and static caching (CDN): use DNS/CDN tools such as Akamai, Cloudflare, Route53, and Cloudfront for caching static content, optimizing load times for client-side rendered applications built with frameworks like Next.js or React.
  • Backend applications: use autoscaling with VMs or container orchestration platforms such as Kubernetes or Nomad, supported by cloud providers like AWS, Azure, GCP, and DigitalOcean. Use Packer to create identical machine images for multiple platforms from a single source configuration, ensuring consistency and efficiency in your deployment process. For single VM backends, consider using PM2 for spawning multiple processing instances or Docker for containerization. Additionally, integrate Docker with Nginx for load balancing and use Consul and Consul-template for automatic service discovery and configuration, ensuring good scalability.
  • Database management: choose scalable RDBMS solutions like AWS Aurora or GCP Cloud SQL, using caching with Redis for temporary data. Optimize queries for performance by ensuring they read from an index; defaulting to a B-Tree index is usually adequate for most cases. Consider read/write separation to distribute loads efficiently.

Reliability and DDoS mitigation​

In the face of high load or DDoS attacks, CAPTCHA can serve as a simple yet effective tool to regulate user access rates. Both Akamai and Cloudflare offer CAPTCHA functionalities to mitigate excessive server access.

Performance monitoring and insights​

Use short-term monitoring solutions from cloud services for system performance insights, including slow query detection and API endpoint analysis. Integrate tools such as Sentry for real-time monitoring and troubleshooting.

Load testing framework​

  1. Gather requirements: define the critical functionalities and user experiences for testing.
  2. Map user journeys: use monitoring data to simulate realistic user interactions.
  3. Establish a baseline: conduct initial tests to set performance benchmarks. Whenever performance deviates from this benchmark, you'll know a deeper dive into test data is necessary.
  4. Automate testing: integrate load testing into your CI/CD pipeline. Use tools such as Locust or K6 to simulate user behavior.

Best practices for load testing​

  • Create test scenarios that accurately reflect user behavior, targeting the app's critical functions.
  • Predict traffic volumes and patterns using historical data and marketing projections, focusing on APIs expected to experience the highest load.
  • Choose the right testing tools, such as Locust for API testing and Saigon testnet with Ethers.js or web3.js for on-chain interactions.
Did you know?

The Wild Forest Packs sale event experienced nearly 200 requests per second at peak traffic.

Pre-event scalability checklist​

  • Predict event-specific traffic and adjust your infrastructure resources in advance.
  • Pre-scale VMs or Kubernetes clusters before the event and scale down afterwards.
  • Increase the CPU and disk IO size of the database and consider switching to high IOPS disks, such as AWS's NVMe or GCP's local SSDs.
  • Increase the number of connections from the API to the database.