• Talking Book Library
• For the Public
• For State Employees
• For Librarians
• About MSL
• Jobs
• Home

MSL Home > For Librarians > Montana Shared Catalog > Members > Incident Display

Incident Display

Summary Statistics

Start Date/Time:			4/22/2006 00:00
Todays Date/Time:			12/3/2008 15:50
Total Elapsed Time:			7M,  13D,  15H,  50 Min.		100.00%
Total Up Time:			7M,  3D,  20H,  43 Min.		98.98%
Total Incident Down Time:			9D,  19H,  7 Min.		1.02%

Incident Detail

Inc. #	Date Down	Time Down	Date Up	Time Up	Per Incident Down Time	Cause of Down Time	Scheduled?	Description	Symptoms	Comment
30	10/21/2008	10:45	10/21/2008	12:30	1H,  45 Min.	State Network	No	ITSD experienced a network outage. This outage was caused when their distribution switch’s processors experienced extreme high utilization. ITSD is still investigating the root cause of this issue	Workflows and iBlink not available. MSC staff unable to send/receive email.
29	9/8/2008	18:00	9/8/2008	20:30	2H,  30 Min.	MSC PRD Server	No	Large reports file in /tmp directory reduced available disk space until no longer useable. Service restored when file was deleted and performed HIR.	Server response times slowed and no access to reports section of Workflows. Search errors in iBlink.
28	8/17/2008	06:00	8/17/2008	16:00	10H,  0 Min.	State Network	No	Originally an ITSD scheduled 5 minute network outage for maintenance resulted in total network outage for most of the day.	Workflows and iBlink not available
27	7/11/2008	22:00	7/12/2008	05:00	7H,  0 Min.	MSC PRD Server	Yes	Reindexing required after removal of Ronan holdlings from the catalog.	Workflows and iBlink not available.
26	6/26/2008	07:25	6/26/2008	07:45	0H,  20 Min.	MSC PRD Server	Yes	GL 3.1 patch update	Workflows and iBlink not available	This was an update to fix a big introduced in patch 5e.
25	5/18/2008	12:30	5/18/2008	14:30	2H,  0 Min.	MSC PRD Server	No	Large reports file in /tmp directory reduced available to disk space until no longer useable. Service restored when file was deleted and performed HIR.	Server response times slowed and no access to reports section of Workflows. Search errors in iBlink.
24	4/29/2008	12:30	4/29/2008	15:15	2H,  45 Min.	MSC PRD Server	No	Production server powered off. Cause unknown. Had to do a cold reboot of server and then bring back up Workflows and the Apache web server.	Workflows and iBlink not available	Suspect someone unplugged the server or the servers power strip and then plugged it back in.
23	4/25/2008	21:00	4/27/2008	16:00	1D,  19H,  0 Min.	MSC PRD Server	Yes	Scheduled Down Time for yearly load of Authority records and changed Bib records	Access to Workflows and production iBistro unavailable. Access to test instance of iBistro available for patron searching.
22	4/12/2008	08:06	4/12/2008	08:20	0H,  14 Min.	MSC PRD Server	Yes	GL 3.1 Patch 5e upgrade	Workflows and iBlink not available during upgrade.
21	4/4/2008	13:15	4/4/2008	14:30	1H,  15 Min.	MSC PRD Server	No	Production server slowed to a halt. Required server reboot to restore system. Cause unknown.	Workflows and iBlink not available.
20	9/13/2007	08:07	9/13/2007	08:25	0H,  18 Min.	MSC PRD Server	Yes	GL 3.1 Patch 5d upgrade	Workflows and iBlink not available during upgrade.
19	7/19/2007	08:00	7/19/2007	15:00	7H,  0 Min.	MSC PRD Server	Yes	GL3.1 Upgrade
18	7/10/2007	08:30	7/11/2007	15:20	1D,  6H,  50 Min.	MSC PRD Server	Yes	Oracle Upgrade.	Workflows and iBistro down for this scheduled upgrade.
17	5/15/2007	12:00	5/15/2007	14:00	2H,  0 Min.	MSC PRD Server	Yes	This was a scheduled reboot of the server. Two administrative reports were beginning to show a slowdown in completing weekly or daily and a reboot may help. In the course of rebooting, there were unexpected difficulties when users attempted to connect following the reboot. Issue was resolved by a second reboot of the server.	System down: no access to Workflows or to iBistro.
16	4/21/2007	07:45	4/21/2007	19:55	12H,  10 Min.	Power	Yes	Scheduled downtime at entire Montana State Library building, to install new electrical panel in preparation to connect new backup generator.	No power during installation time at MSL: no access to Workflows client or to iBistro, both production and development.
15	12/17/2006	11:00	12/18/2006	10:30	23H,  30 Min.	MSC PRD Server	Yes	Upgrade to GL3.0 platform	Workflows and iBistro unavailable. Standalone/Offline and test iBistro used during scheduled down time
14	12/15/2006	15:45	12/15/2006	17:30	1H,  45 Min.	MSC PRD Server	No	MSL Storage Area Network failed. MSC Production and MSC Development servers lost connection with the SAN	Workflows and iBistro unavailable
13	11/4/2006	10:15	11/4/2006	12:15	2H,  0 Min.	MSC PRD Server	No	Power outage in Helena. State down. Back up power to MSC server eventually failed. Had to do a cold reboot of server and then bring back up Workflows and the Apache web server.	MSC libraries could not connect to WF or iBistro.
12	10/3/2006	09:50	10/3/2006	11:40	1H,  50 Min.	N/A	No	SIP2 server generating "ghost sessions" since the 10/01 AIX upgrade. These sessions numbered in the thousands and caused Oracle errors and brought down the system. Decision was made to halt the SIP2 server until Sirsi can deliver a fix for our Unicorn version.	No access to Workflows or to iBlink.
11	10/1/2006	09:15	10/1/2006	11:00	1H,  45 Min.	MSC PRD Server	Yes	Upgrade to production server's AIX operating system, to AIX 5.2. Server and system down during upgrade.	Workflows and iBlink not available during upgrade.
10	9/13/2006	15:40	9/13/2006	15:50	0H,  10 Min.	State Network	No	State Internet unavailable	No network connection	Quickly resolved by State
9	8/28/2006	09:00	8/28/2006	11:25	2H,  25 Min.	State Network	No	State of Montana experienced network problems that appeared to primarily internal network issues.	Some agency libraries lost connectivity to the system. The MLN site was unavailable both inside and outside of the network.
8	8/24/2006	08:45	8/24/2006	09:15	0H,  30 Min.	MSC PRD Server	Yes	Due to extreme slowness reported by multiple MSC libraries over 48 hours, a reboot of the server was done. The reboot appeared to resolve the issue.	System unavailable during the reboot process.
7	8/17/2006	17:00	8/17/2006	17:40	0H,  40 Min.	State Network	No	Due to State power supply problems, the State Network was brought down incrementally to prevent a hard shutdown. Email was initially affected around 3:30 p.m. and network access to the system was lost at 5:00. Network access was restored around 5:40 p.m.	No Internet access	The phone tree was initiated because email was unavailable.
6	8/16/2006	06:30	8/16/2006	06:45	0H,  15 Min.	MSL Network	Yes	MSL Network switch maintenance		Except for minimal network downtime during early morning hours which caused some regularly running reports to be delayed, this incident did not disrupt the system.
5	7/27/2006	13:00	7/27/2006	15:10	2H,  10 Min.	State Network	No	Power failure in Helena. Affected entire state government campus and parts of Helena. Cause of power failure: squirrel.	Interrupted all access to state servers. No access to MSC servers or to MSC email.
4	7/12/2006	07:15	7/12/2006	08:15	1H,  0 Min.	MSC PRD Server	Yes	Mike Price upgrading AIX operating system. First step in process
3	6/9/2006	20:00	6/11/2006	15:00	1D,  19H,  0 Min.	MSC PRD Server	Yes	Scheduled Down Time for yearly load of Authority records and changed Bib records	Access to Workflows and production iBistro unavailable. Access to test instance of iBistro available for patron searching.
2	5/28/2006	02:00	5/28/2006	09:00	7H,  0 Min.	State Network	No	Power outage within state's network. Part of state network access unavailable	No access to MSC servers, inside and outside of state's network.
1	5/22/2006	02:00	5/23/2006	02:00	1D,  0H,  0 Min.	State Network	No	State of Montana network failure.	No access to MSC servers, inside and outside of state's network.