If you have been reading my earlier posts a big Thanks 
. If not and if you want to follow from here again Thanks , atleast read SQL Server Data Replication Project - 4 (Summarizing)as it sets context. From this post I am planning to share lot of ideas, code snippets that have been used.
Feel free to comment on these approaches and open to mail communications at gurucb(at)hotmail(dot)com.
In this post it is all about server sizing. For Server sizing, we need to simulate realistic amount of transaction load that is equivalent to load that a server may undergo in production. So, how do we generate such load.
Load generation depends upon data replication technology used. Based on our discussions around network bandwidth, application availability requirements, reporting requirements we have selected
- SQL Server Transactional Replication
- SQL Server Log Shipping (With compression)
SQL Server Transactional Replication:
For Transactional Replication, to size central server build a test lab.
Test lab constitutes of
- Representative Test (Remote) servers act as remote servers. Test remote servers host remote databases. Instead of using 350 remote servers, use 10 servers, with enough capacity to atleast hold 35 remote database(s).
- Ensure in simulated lab there is no constraint on network, if there were to be constraint on network, H/W sizing of Central Server would not be of appropriate. Network becomes bottleneck and load on central server is reduced during parallel load. Larger network pipe better for H/W sizing.
- Ensure enough disk space for Test (Remote) Server
- Database Size and growth of Database due to load
- Transaction log file size and growth due to load
- Distribution Database and growth
- Ensure enough disk space for Test (Central) Server
- Database Size (for 350 Databases) and room for growth
- Transaction Log File Size (for 350 Databases) and room for growth
- Each Test (Remote) server would host a publisher database, publication for each DB and act as its own distributor.
- On publisher DB, apply transaction load (will later detail how to get transaction load) and allow replication to move data from Test (Remote) server to Test (Central) Server.
- If Test (Central) server needs to be tested further, stop distribution agent, apply transactional load on publisher database (on remote servers) and then start distribution agents in parallel to create a spurt in load on central server.
- Monitor Performance Metrics on Test (Central) Server with different configurations (in terms of CPU, RAM) and fine tune in multiple test cases to find out arrive at appropriate server size needed for Central Server for Transactional Replication.
SQL Server Log Shipping:
For Log Shipping, to size central server build a test lab.
Test lab constitutes of
- Representative Test servers that act as remote servers. Test remote servers host remote databases. Instead of using 350 servers, use 10 servers, with enough capacity to at the least hold 35 remote database(s).
- Ensure in simulated lab there is not constraint on network, if there is a constraint on network, H/W sizing of Central Server would not be of appropriate volume. Larger network pipe better for H/W sizing.
- Ensure enough disk space for Test (Remote) Server
- Database Size and growth of Database due to load
- Transaction log file size and growth due to load
- Distribution Database
- Ensure enough disk space for Test (Central) Server
- Database Size (for 350 Databases) and room for growth
- Transaction Log File Size (for 350 Databases) and room for growth
- On each Test (Remote) server for each database configure log shipping. Each database on Test (Remote) Server would act as Primary Database. Each primary server would host transaction log backup job. Enable compression for Transaction log (to reduce network footprint)
- Test (Central) Server acts as secondary server and would host copy and restore jobs. Secondary server could end up with 700 jobs (350 copying and 350 for restoring). So, it may be good idea to move copy job to Primary server (Custom Log Shipping) as that distributes load of copy job to all remote servers.
- As transaction logs are compressed, restore jobs may have additional CPU load to decompress all transaction log file before restoration. Even native sql compression would have impact on secondary server resources.
- To create spurt in load, stop restoration jobs on central server and start all restoration jobs in parallel where for each database there are atleast 10 Transaction log backup files to be restored.
- Monitor Performance Metrics on Test (Central) Server with different configurations (in terms of CPU, RAM) and fine tune in multiple test cases to find out arrive at appropriate server size needed for Central Server for Log Shipping.
Now we have lab setup for Transaction Replication or Log Shipping, important missing piece is how do we apply transaction load. To apply transaction load first step is to capture load and replay it. For capturing and replaying load different options we could think of are
Capture:
- SQL Server Trace for capture
- Change Data Capture feature of SQL Server.
- X Events
- Create Triggers and log changes to different tables.
Replay:
- SQL Server Profiler
- Custom application for reading from Traces and replaying
- Custom application for reading from CDC data and replaying
- Custom application for reading from X events data and replaying
- Custom application for reading from custom tables and replaying
Of all these approaches we could think of, for capture we are using
- SQL Server Trace
- Change Data Capture feature of SQL Server
For Replaying we are using
- SQL Server Profiler
- Custom application for reading from traces and replaying
- Custom application for reading from CDC data and replaying.
Planning to work on x events later..
Next post is about capture and how to use SQL Server Trace and CDC feature to capture changes.
Hope methodologies, approaches and technical tidbits are useful to all readers.
Until next post..
-Guru