TempDB LOGBUFFER and IO_COMPLETION Waits

We observed numerous SQL sessions that were waiting for LOGBUFFER and IO_COMPLETION; all of these sessions were executing an INSERT INTO #Temp table, as illustrated below.

Upon examining the count of Virtual Log Files (VLFs) in the tempdb .ldf file, we discovered that the tempdb transaction log was significantly fragmented.

Subsequently, we executed defragmentation of the VLFs within the TempDB transaction log file, using the SQL commands provided below:

use tempdb

DBCC SHRINKFILE ( 'templog', 0, TRUNCATEONLY)

ALTER DATABASE [tempdb] MODIFY FILE ( NAME = 'templog', SIZE = 150000MB )

The VLFs in the tempdb log file have been decreased to 32, as shown below:

We also observed that the DB Full Backup job was planned to run at that time. Thus, we rescheduled the backup job to operate during off-peak hours.

We found that the maximum memory allocation for SQL Server was set to 75% of the overall server memory, which we think was a result of the SQL Server installer GUI's suggestions.

Next, we increased the SQL max memory limit to 1216GB, representing 95% of the total 1.25TB, on the server, which is chiefly assigned to SQL server services. It should be adequate to reserve 64GB of RAM for the Windows operating system and other various tasks.

Finally, we relocated the tempdb transaction log file to a separate disk that offers improved write speed.

Slow disk in Tempdb leads to unexpected failover of the Availability Group

A problem arose for one of my clients, as their production SQL Server availability group kept failing over back and forth. Presented below is the SQL Server error log from when the failover took place.

I recommended that they relocate the tempdb from the overloaded disk.

In the meantime, to minimize the likelihood of unexpected failover, I recommended that they adjust certain cluster settings as outlined below:

1. Set the LeaseTimeout and HealthCheckTimeout values to 60000 in the Availability Group, as depicted below.

2. Raise the heartbeat delay and threshold values, as 1/2 * LeaseTimeout should be lower than SameSubnetThreshold * SameSubnetDelay, by executing the following PowerShell commands:

3. Bring the AG group offline and subsequently online, or perform a switchover, to apply the changes.

Gather the Usage and Size of all Indexes across every database

EXEC sp_MSforeachdb 'USE [?];

SELECT DB_NAME() AS DB,

    OBJECT_NAME(i.[object_id]) AS [TableName],

    i.name AS [IndexName],

    i.index_id,

    s.user_seeks,

    s.user_scans,

    s.user_lookups,

    s.user_updates,

    i.type_desc AS [IndexType],

    s.last_user_seek,

    s.last_user_scan,

    s.last_user_lookup,

    s.last_user_update,

    SUM(ps.used_page_count) * 8 / 1024 AS [Used Space (MB)],

    SUM(ps.reserved_page_count) * 8 / 1024 AS [Reserved Space (MB)]

FROM 

    sys.dm_db_index_usage_stats AS s

    INNER JOIN sys.indexes AS i

        ON s.[object_id] = i.[object_id] AND s.index_id = i.index_id

    INNER JOIN  sys.dm_db_partition_stats AS ps

        ON ps.object_id = i.object_id AND ps.index_id = i.index_id

WHERE 

    OBJECTPROPERTY(s.[object_id], ''IsUserTable'') = 1

    AND s.database_id = DB_ID()

GROUP BY 

    OBJECT_NAME(i.[object_id]),

    i.name,

    i.index_id,

    s.user_seeks,

    s.user_scans,

    s.user_lookups,

    s.user_updates,

    i.type_desc,

    s.last_user_seek,

    s.last_user_scan,

    s.last_user_lookup,

    s.last_user_update

ORDER BY 

    [TableName], [IndexName];

';

 

Assessing the Latency of Availability Group Database Synchronization

The SQL script provided below can be utilized to assess the real-time latency in seconds of SQL Server AlwaysOn synchronization.

;WITH

AG_Stats AS

(

SELECT

AR.replica_server_name,

AG.name AS AGName,

HARS.role_desc,

DB_NAME(DRS.database_id) AS DBName,

DRS.last_commit_time

FROM sys.dm_hadr_database_replica_states DRS

INNER JOIN sys.availability_replicas AR ON DRS.replica_id = AR.replica_id

INNER JOIN sys.dm_hadr_availability_replica_states HARS

ON AR.group_id = HARS.group_id AND AR.replica_id = HARS.replica_id

INNER JOIN sys.availability_groups AG ON AG.group_id = AR.group_id

),

Pri_CommitTime AS

(

SELECT

replica_server_name,

AGName,

DBName,

last_commit_time

FROM AG_Stats

WHERE role_desc = 'PRIMARY'

),

Sec_CommitTime AS

(

SELECT

replica_server_name,

AGName,

DBName,

last_commit_time

FROM AG_Stats

WHERE role_desc = 'SECONDARY'

)

SELECT

p.replica_server_name AS PrimaryReplica,

p.AGName,

p.DBName AS DatabaseName,

s.replica_server_name AS SecondaryReplica,

DATEDIFF(SECOND, s.last_commit_time, p.last_commit_time) AS Sync_Latency_Secs

FROM Pri_CommitTime p

LEFT JOIN Sec_CommitTime s

ON s.DBName = p.DBName AND s.AGName = p.AGName;

Example output as shown below:

Snapshot Replication Agent Profile's Parameter BcpBatchSize

This week, a customer of mine shared the image below, mentioning that their Snapshot Replication is experiencing bad performance.

The customer informed me that the snapshot replication was operating correctly. I then inquired whether the tables being replicated had increased in size, to which they responded affirmatively. Consequently, I recommended that they utilize a smaller BcpBatchSize value in the Snapshot Agent Profile. I recommended that they reduce the BcpBatchSize setting from its default value of 100000 to 50000 rows. This can be done by creating a new agent profile derived from the default profile, adjusting the BcpBatchSize value to 50000, and subsequently assigning this new profile to the Snapshot Publication.

Lowering it can reduce memory usage and improve stability for large datasets, though it may slow down snapshot generation.

Streamline the MSSQL High Availability to improve business continuity

This short document proposes an exploration into the possibility of refining the SQL Server high availability architecture to improve business continuity.

The issue we aim to address

Since March 2025, the Production SQL Server has faced outages 2 times. DBA team determined that the cause was the Windows Server Failover Cluster (WSFC) losing its quorum, which resulted in both the primary SQL availability replica in the US and the secondary SQL availability replica in the DK site being in a "Resolving" state, rendering them inaccessible for both reading and writing. The DBA team cannot fix this issue alone and needs help from the IT team to recover the WSFC quorum.

Current architecture for production SQL Server High Availability

Displayed below is the current WSFC architecture of the production SQL Server.

To mitigate the performance effects of network latency between the US and DK sites, the primary SQL replica and the secondary replica are configured to synchronize data in Asynchronous-Commit mode. This configuration limits the SQL availability group to Manual Failover only, where Automatic Failover is not supported, a fact that should already be known and accepted by us.

The table below has been taken from Microsoft Online regarding WSFC Quorum:

Based on the information provided by Microsoft, we can derive the following disaster recovery scenarios for our existing configuration:

Primary Server	Cloud Witness	Secondary Server	Database accessibility within the availability group
Up	Up	Up	The primary database server allows read and write access.
Up	Up	Down	The primary database server allows read and write access.
Up	Down	Up	The primary database server allows read and write access.
Up	Down	Down	According to dynamic quorum behavior of WSFC, if the secondary server and the witness are taken down one at a time, the quorum continues to exist, and the primary database server is still accessible. However, if both the secondary server and the witness go down simultaneously from the primary server's perspective—imagine the internet connection to both is severed—the databases in the primary server's availability group will become inaccessible, and immediate IT support is needed to recover the quorum. If the secondary server is connected to the witness, we might be able to reinstate the database services on the secondary server through a forced manual failover.
Down	Up	Up	DBA must conduct a manual failover to the secondary server to resume read-write access to the databases in availability group.
Down	Down	Up	Quorum is lost; the databases are inaccessible, and immediate IT support is needed to recover the quorum.

 

The new architecture we present aims to tackle the highlighted scenarios.

The proposed architecture – Clusterless Availability Group

SQL Server 2017 introduces read-scale availability groups without a cluster.

In the same way as a conventional clustered availability group, a Read-Scale availability group (often referred to as Clusterless; the term Read-Scale is merely a marketing label by Microsoft) allows for data synchronization between primary and secondary replicas to be configured in either Synchronous-Commit mode or Asynchronous-Commit mode at any time without causing service interruptions. A clusterless availability group only allows for manual failover, and does not support automatic failover. Using a clusterless availability group offers the benefit of removing the necessity for WSFC quorum and the cloud witness, thereby simplifying maintenance. Moreover, the disaster recovery scenarios for our SQL Servers will be less complex as described below:

Primary Server	Secondary Server	Database accessibility within the availability group
Up	Up	Primary database server is Read-Write accessible.
Up	Down	Primary database server is Read-Write accessible.
Down	Up	DBA must conduct a Manual Failover to the Secondary Server to resume Read-Write access to the databases in Availability Group.

 

Following Actions

DBA team is looking to obtain a new set of SQL Server VMs, one in the US and the other in DK, to set up a lab environment for assessing the clusterless availability group, focusing on disaster recovery and data integrity. After we secure satisfactory testing results, we can develop a migration plan for the clusterless architecture or include it in the forthcoming SQL Server upgrade initiative (upgrading from version 2019 to 2022/2025).

Out of Sync status for the AlwaysOn Availability Group Replica

 Last week, a customer mentioned to me that their MSSQL server is running slowly. I verified that the CPU usage of the SQL Server was low, and the memory showed a high Page Life Expectancy value. This application is a vendor package software, with a low workload. Before informing the customer about the health of their SQL Server, I executed Paul Randal's Wait Statistics script. This script indicates that the primary wait type is HADR_SYNC_COMMIT, which signifies that the server is waiting for transaction commit processing for the synchronized secondary databases to harden the log. The screenshot below indicates that 75% of the wait time is attributed to HADR_SYNC_COMMIT.

The customer stated that the Availability Group consists of two replicas: the primary one in Shanghai and the secondary one in Hong Kong, with a network bandwidth of 10Mb/s. They requested that I demonstrate how frequently the AAG Replicas are out of sync. I provided them with the hadr_db_partner_set_sync_state event from the alwayson_health Extended Event session, as shown in the two screenshots below (sync_state NOT indicate out-of-sync, while sync_state LOG signifies resumed):

Given that the out-of-sync situation arises often, I can easily show how to recognize it using the Availability Group Dashboard, as depicted in the image below.

To address this issue, I recommended that the customer switch the AAG sync mode from Synchronous mode to Asynchronous mode; however, the trade-off is that the AAG will not be able to perform automatic failover.