Topic: Nerd Porn: ecommerce cluster

Anonymous A started this discussion 13 years ago #30,116

Since the start of 2013 I’ve been working with my biggest client ever. They sell a lot of products online and have about 500 stores in America.

The cabling is shit (the datacenter techs didn’t follow instructions), but the hardware is interesting.

hardware

* 24 webservers
* 5 database servers
* 3 cache servers
* 2 firewalls, configured as a HA pair
* 2 load balancers, configured as a HA pair
* 3 switches
* 30 kW of power required

website

* 815,000 lines of code (100,000 lines more than core + Zend)
* 96% PHP
* 1.7% Perl
* 1.5% Ruby
* 0.5% Shell
* Products: 101,000
* CMS pages: 1,400

traffic

* 40 million monthly pageviews
* 2.3 million unique visitors
* 46,000 monthly checkouts
* 89% of visitors from the USA
* 34% of visits from iOS
* 14% of visits from Android

(Edited 6 minutes later.)

Anonymous A (OP) double-posted this 13 years ago, 1 minute later[^] [v] #435,402

12 of their 24 webservers

Each costs about $8,000

* Ubuntu 12.04 LTS 64-bit
* 3.5.0-36-generic GNU/Linux kernel
* Dual Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz (4 cores each)
* 32 GB DDR3 1333 MHz RAM
* site-wide average response time: 282 ms
* CPU utilization averages 14%
* network throughput averages 5 mbps transmitted and 17 mbps received

(Edited 13 seconds later.)

Anonymous A (OP) triple-posted this 13 years ago, 3 minutes later, 4 minutes after the original post[^] [v] #435,405

Top to bottom: cache1, util1, data5, data3, data1

The data1 and data2 servers each cost about $40,000:
* Ubuntu 12.04 LTS 64-bit
* 3.5.0-34-generic GNU/Linux
* Quad Intel(R) Xeon(R) CPU E7- 2860 @ 2.27GHz (10 cores each)
* 128 GB buffered DDR3 1333 MHz RAM
* SSD, RAID 1+0, hardware controller
* MySQL 5.6.11 with tcmalloc

Each of these hosts has about 1 TB of storage.

(Edited 15 seconds later.)

Anonymous A (OP) quadruple-posted this 13 years ago, 4 minutes later, 9 minutes after the original post[^] [v] #435,406

One of the load balancers is at the top of this photo. They run OpenBSD.

Anonymous A (OP) quintuple-posted this 13 years ago, 1 minute later, 10 minutes after the original post[^] [v] #435,408

Sadly, this is the best cabling job the datacenter did. At top is one of the HA firewalls. Below it is the third switch, a spare.

Anonymous B joined in and replied with this 13 years ago, 23 seconds later, 10 minutes after the original post[^] [v] #435,409

@435,405 (A)
1TB in SDDs? Neato.

Anonymous A (OP) replied with this 13 years ago, 56 seconds later, 11 minutes after the original post[^] [v] #435,410

;_;

Anonymous A (OP) double-posted this 13 years ago, 1 minute later, 13 minutes after the original post[^] [v] #435,411

@435,409 (B)
The SSDs are set up as RAID 1+0, though. The usable size is 468 GB. The other hosts have mechanical drives that are about 1 TB after RAID allocation.

Anonymous A (OP) triple-posted this 13 years ago, 1 minute later, 14 minutes after the original post[^] [v] #435,412

The colo techs decided to stash our junk RAM on top of a server

Anonymous B replied with this 13 years ago, 20 minutes later, 35 minutes after the original post[^] [v] #435,416

@435,411 (A)
Yeah but that's still a lot of solid.

Anonymous A (OP) replied with this 13 years ago, 6 minutes later, 41 minutes after the original post[^] [v] #435,421

@previous (B)
The array can write at 500 MBps :)

Anonymous C joined in and replied with this 13 years ago, 10 minutes later, 52 minutes after the original post[^] [v] #435,424

I fapped.

Anonymous A (OP) replied with this 13 years ago, 2 minutes later, 55 minutes after the original post[^] [v] #435,427

@435,410 (A)
Also note the power cables crossing and running alongside data cables ;_;

Minichan