To Keep everyone informed we will be updating this with status updates thru the day today. Full Incident report will be added later...
This is an ongoing issue that began Monday morning: Heavy Table locking
After recommended restart last night 3-8-2021, the issue returned this morning 3-9-2021.
3-9-2021
8:00 AM - Microsoft (MS) re-engaged on issue with premier support critical severity ticket
8:35 AM - Tom (DKIT) requested XPO of code change from Sunday
8:45 AM - Microsoft analysis began on SQL debug from night of 3-8-2021. waiting deep analysis results.
8:46 AM - Tom (DKIT) received XPO of Sunday code and began analysis.
9:00 AM - Tom (DKIT) updated indexes on tables identified in analysis as being attached to blocking issue.
9:00 AM - RockSolid Also updating indexes and disabled Batch Jobs - other trouble shooting steps may have been performed but we do not have report on that at this time.
9:10 AM - MS requested new trace for comparison -
9:15 AM - Blocking has stopped, monitoring for return.
10:00 AM - Reports of Printing issues continue to be submitted.
10:15 AM - Restart of SSRS service cleared most printing issues, some others remain and are being worked.
12:00 PM - Ongoing printing related issue source has been identified. Print driver corruption on one RD server is causing user sessions to hang on print load. We have a process to fix this but it requires moving all users off the server. I have removed the server (PRDRD02) from load balancing to keep new users from connecting to it, anytime a user disconnects they should move to one of the other 3 servers in the farm. They can then print. I will repair the server later today.
3-10-2021
8:00 AM - Reports of ax problems with receiving, login issues, and memory overruns or outages submitted
8:54 AM - Tom (DKIT) identified locks against accountingevents table and cleared all locks from that table then everything else cleared and system appears to be running ok right now.