tag:status.warwick.ac.uk,2005:/historyUniversity of Warwick Status - Incident History2024-03-29T07:52:31+00:00University of Warwicktag:status.warwick.ac.uk,2005:Incident/199308702024-02-07T15:04:06+00:002024-02-07T15:04:06+00:00Issue with timetabling information in Tabula<p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>15:04</var> GMT</small><br><strong>Resolved</strong> - Issue with timetabling is now fixed.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>14:21</var> GMT</small><br><strong>Investigating</strong> - We're currently investigating a problem with the source that Tabula gets its timetabling information from. <br /><br />We'll update here when we know anything more.</p>tag:status.warwick.ac.uk,2005:Incident/183912672023-09-05T07:02:56+01:002023-09-05T07:02:56+01:00Network maintenance<p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>07:02</var> BST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>06:55</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>06:54</var> BST</small><br><strong>Scheduled</strong> - There will be some disruption to web pages this morning as a networking change is carried out. Please return later and you should be able to access the website.</p>tag:status.warwick.ac.uk,2005:Incident/158180212023-01-11T11:55:49+00:002023-01-11T12:30:53+00:00Tabula performance issues<p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>11:55</var> GMT</small><br><strong>Resolved</strong> - Service should now be normal. We continue to investigate performance improvements and other ways to avoid service issues when Tabula is very busy.</p><p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>11:53</var> GMT</small><br><strong>Update</strong> - System seems stable now and pages are loaded quickly. We did restart system a few times.</p><p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>11:24</var> GMT</small><br><strong>Investigating</strong> - There are some known performance issues with Tabula caused by a large influx of students signing up for small groups. We are working on remediating this and are already working on Tabula performance improvements to avoid this sort of situation.<br /><br />If you are currently trying to sign up for small groups and it is slow, please leave the application for a few minutes as users refreshing pages will only exacerbate the problem.</p>tag:status.warwick.ac.uk,2005:Incident/157285102022-12-28T12:03:03+00:002022-12-28T12:03:03+00:00Website slow to respond<p><small>Dec <var data-var='date'>28</var>, <var data-var='time'>12:03</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>28</var>, <var data-var='time'>11:27</var> GMT</small><br><strong>Investigating</strong> - We are investigating a follow-on issue where the website is very slow to respond.</p>tag:status.warwick.ac.uk,2005:Incident/157274492022-12-28T10:19:59+00:002023-01-04T15:12:15+00:00Website unavailability<p><small>Dec <var data-var='date'>28</var>, <var data-var='time'>10:19</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>28</var>, <var data-var='time'>10:13</var> GMT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Dec <var data-var='date'>28</var>, <var data-var='time'>07:49</var> GMT</small><br><strong>Investigating</strong> - We are aware that the website is currently unavailable. The issue should be resolved in the next few hours.</p>tag:status.warwick.ac.uk,2005:Incident/156858752022-12-21T19:45:36+00:002022-12-21T20:01:33+00:00Website unavailability from 19:00 GMT<p><small>Dec <var data-var='date'>21</var>, <var data-var='time'>19:45</var> GMT</small><br><strong>Completed</strong> - The upgrade work is completed and the remaining issues have been resolved. Apologies for the slightly longer than advertised lack of service.</p><p><small>Dec <var data-var='date'>21</var>, <var data-var='time'>19:17</var> GMT</small><br><strong>Update</strong> - The unavailability period has started.</p><p><small>Dec <var data-var='date'>21</var>, <var data-var='time'>18:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>21</var>, <var data-var='time'>10:03</var> GMT</small><br><strong>Scheduled</strong> - The Warwick website will be unavailable for around 45 minutes from 7pm GMT on Wednesday 21st December as part of a planned hardware upgrade. This will also affect access to other web applications such as Tabula, and you will not be able to sign into your Warwick account on the web during this time.<br /><br />We aim to keep the outage as brief as possible, and have planned it out of normal working hours to minimise the impact on visitors to our site. The upgrade will provide long-term stability and support for our web applications in the future.</p>tag:status.warwick.ac.uk,2005:Incident/156866262022-12-21T12:49:06+00:002022-12-21T12:58:15+00:00Single sign-on issues<p><small>Dec <var data-var='date'>21</var>, <var data-var='time'>12:49</var> GMT</small><br><strong>Resolved</strong> - A fix has been applied and signing in should now be working as normal.</p><p><small>Dec <var data-var='date'>21</var>, <var data-var='time'>12:38</var> GMT</small><br><strong>Identified</strong> - There are known issues with signing in for many users. A fix is currently being applied and service should be restored within 30 minutes.</p>tag:status.warwick.ac.uk,2005:Incident/127755612022-11-10T18:02:20+00:002022-11-10T18:02:32+00:00Website unavailability<p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>18:02</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>16:08</var> GMT</small><br><strong>Identified</strong> - We have identified a slow operation that could potentially exhaust resources for other requests, and are looking at improving the efficiency of this operation.</p><p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>11:55</var> GMT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>11:54</var> GMT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>11:52</var> GMT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>11:52</var> GMT</small><br><strong>Investigating</strong> - The warwick.ac.uk website was briefly unavailable. We are looking into the cause.</p>tag:status.warwick.ac.uk,2005:Incident/99579242022-05-11T14:46:20+01:002022-05-17T14:57:20+01:00Applications not available<p><small>May <var data-var='date'>11</var>, <var data-var='time'>14:46</var> BST</small><br><strong>Resolved</strong> - There has been no recurrence of the issue. Root cause investigations continue and applications continue to be monitored as usual.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>16:41</var> BST</small><br><strong>Monitoring</strong> - All services that were unavailable have now been restored. We continue to monitor the situation and investigate the root cause.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>15:29</var> BST</small><br><strong>Update</strong> - Service has been restored on some applications and we are working to restore the remaining. We are still investigating the root cause.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>13:42</var> BST</small><br><strong>Investigating</strong> - We are aware of an issue affecting various web applications and are currently investigating.</p>tag:status.warwick.ac.uk,2005:Incident/90806422022-01-14T12:16:31+00:002022-01-14T12:16:32+00:00Degraded performance due to seminar sign-ups<p><small>Jan <var data-var='date'>14</var>, <var data-var='time'>12:16</var> GMT</small><br><strong>Resolved</strong> - Error rates continue to be low and requests are rendering successfully so we are considering this issue to be resolved. We will continue to work to reduce the impact of seminar sign-ups going forward.</p><p><small>Jan <var data-var='date'>14</var>, <var data-var='time'>11:35</var> GMT</small><br><strong>Monitoring</strong> - Due to a large number of simultaneous seminar sign-ups, some parts of Tabula are currently experiencing degraded performance. Pages may be slow to load, and actions may take longer than usual to complete. We are continuing to monitor the application closely, but if you encounter problems you may wish to consider trying again later.</p>tag:status.warwick.ac.uk,2005:Incident/74814952021-07-12T11:52:07+01:002021-07-12T11:52:08+01:00Module marks not appearing for finalists<p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>11:52</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>11:31</var> BST</small><br><strong>Update</strong> - We have a workaround for this issue which we're hoping to deploy shortly.</p><p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>11:02</var> BST</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>10:42</var> BST</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>10:12</var> BST</small><br><strong>Identified</strong> - We're aware of an issue showing individual module marks to finalists. We are aware of the root cause and working to fix the problem. You should still be able to see progression decisions.</p>tag:status.warwick.ac.uk,2005:Incident/71592072021-06-03T19:15:48+01:002021-06-03T20:33:58+01:00AEP unavailable<p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>19:15</var> BST</small><br><strong>Resolved</strong> - The issue is resolved and AEP is fully available.</p><p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>19:04</var> BST</small><br><strong>Update</strong> - More healthy nodes are now returning to the load balancer pool ready to pick up incoming HTTP requests. We will continue monitoring the site as it recovers full operational capacity but we believe that AEP use should have been possible after 18:13 this evening (with some brief additional downtime at around 19:00).</p><p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>18:13</var> BST</small><br><strong>Monitoring</strong> - We have implemented a workaround to restore access to AEP whilst we resolve the root cause. This workaround has allowed us to serve incoming HTTP requests.</p><p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>18:06</var> BST</small><br><strong>Identified</strong> - We're currently looking into issues with the AEP site.</p>tag:status.warwick.ac.uk,2005:Incident/71511552021-06-02T22:32:15+01:002021-06-02T22:32:15+01:00Issues with single sign-on<p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>22:32</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>22:25</var> BST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>22:13</var> BST</small><br><strong>Identified</strong> - We're currently investigating issues with the authentication system. University members might have trouble signing in or using internal applications until these problems are resolved. We have identified the cause.</p>tag:status.warwick.ac.uk,2005:Incident/70759692021-05-24T20:03:19+01:002021-05-25T09:44:08+01:00Sitebuilder inaccessible<p><small>May <var data-var='date'>24</var>, <var data-var='time'>20:03</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>24</var>, <var data-var='time'>20:00</var> BST</small><br><strong>Investigating</strong> - We are aware that the warwick.ac.uk site is currently unavailable and are investigating the cause.</p>tag:status.warwick.ac.uk,2005:Incident/70759522021-05-24T19:30:00+01:002021-05-24T19:58:36+01:00Sitebuilder unavailable<p><small>May <var data-var='date'>24</var>, <var data-var='time'>19:30</var> BST</small><br><strong>Resolved</strong> - warwick.ac.uk inaccessible</p>tag:status.warwick.ac.uk,2005:Incident/69929012021-05-20T16:54:11+01:002021-05-20T16:54:12+01:00SiteBuilder read-only Thursday 20th May 1pm<p><small>May <var data-var='date'>20</var>, <var data-var='time'>16:54</var> BST</small><br><strong>Completed</strong> - Our maintenance on the Sitebuilder database is now complete. Thanks for bearing with us!</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>16:35</var> BST</small><br><strong>Verifying</strong> - We are currently verifying the results of our maintenance, and expect to restore editing shortly.</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>16:04</var> BST</small><br><strong>Update</strong> - Sitebuilder maintenance continues to progress well and is ahead of schedule. We currently expect to restore editing at or before 4:30pm</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>15:08</var> BST</small><br><strong>Update</strong> - Sitebuilder maintenance continues to progress well. We expect to restore access to editing before 5pm.</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>13:01</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>13</var>, <var data-var='time'>15:05</var> BST</small><br><strong>Scheduled</strong> - We're currently planning a period of read-only time for SiteBuilder to allow us to perform essential maintenance. We'll perform this work on Thursday 20th May at 1pm, and we expect the work to take around 4 hours.<br /><br />During the read-only period, it won't be possible to edit web pages, receive form submissions, post on forums or manage resource bookings, news or calendar pages in SiteBuilder (amongst others). Page viewing will not be affected.</p>tag:status.warwick.ac.uk,2005:Incident/70314672021-05-19T10:30:46+01:002021-05-19T10:30:48+01:00External Users briefly read-only<p><small>May <var data-var='date'>19</var>, <var data-var='time'>10:30</var> BST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>19</var>, <var data-var='time'>10:00</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>18</var>, <var data-var='time'>16:39</var> BST</small><br><strong>Scheduled</strong> - It will not be possible to create, edit, or delete external users for 30 minutes, starting at 10am on Wednesday. Read-only operations will still be available.<br /><br />External users will be unable to change their password during this period. They will be able to sign-in as normal.<br /><br />This change is to allow us to perform essential maintenance on the External Users database to improve performance and stability.<br /><br />For advice or support, please contact us: https://warwick.ac.uk/helpdesk</p>tag:status.warwick.ac.uk,2005:Incident/64061982021-02-26T16:56:09+00:002021-02-26T16:56:09+00:00SMS messages not being delivered<p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>16:56</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>15:14</var> GMT</small><br><strong>Update</strong> - SMS delivery continues to be unavailable due to a major outage with our third party SMS provider https://status.twilio.com/incidents/zmz5c9prz8mm</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>13:53</var> GMT</small><br><strong>Identified</strong> - Our third party SMS provider, Twilio, are experiencing a major outage https://status.twilio.com/incidents/zmz5c9prz8mm</p>tag:status.warwick.ac.uk,2005:Incident/61924822021-02-04T14:20:08+00:002021-02-04T15:47:16+00:00Slow response times on University website<p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>14:20</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>14:02</var> GMT</small><br><strong>Monitoring</strong> - We've made some changes to the nodes in the load-balancer pool which has improved page performance. We will continue to monitor the site.</p><p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>13:54</var> GMT</small><br><strong>Investigating</strong> - We are seeing an increase in response times and error rates for pages on the University website and are currently investigating the root cause</p>tag:status.warwick.ac.uk,2005:Incident/61650172021-02-04T13:30:29+00:002021-02-04T13:19:41+00:00SiteBuilder read-only Thursday 4th February 1pm<p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>13:30</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>13:02</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>15:37</var> GMT</small><br><strong>Scheduled</strong> - We're currently planning a period of read-only time for SiteBuilder to allow us to perform essential maintenance. We'll perform this work on Thursday 4th February at 1pm, and we expect the work to take around 15 minutes.<br /><br />During the read-only period, it won't be possible to edit web pages, receive form submissions, post on forums or manage resource bookings, news or calendar pages in SiteBuilder (amongst others). Page viewing will not be affected.<br /><br />This maintenance will also affect the publishing of alerts, activities and news to My Warwick for a shorter period of time (approximately 2 minutes).</p>tag:status.warwick.ac.uk,2005:Incident/60061642021-01-13T11:10:50+00:002021-01-13T12:38:23+00:00Tabula database performance issues<p><small>Jan <var data-var='date'>13</var>, <var data-var='time'>11:10</var> GMT</small><br><strong>Resolved</strong> - The issues with Tabula performance have now been fixed.</p><p><small>Jan <var data-var='date'>13</var>, <var data-var='time'>09:58</var> GMT</small><br><strong>Monitoring</strong> - We've made a change to reduce the load on Tabula's database, which appears to have been effective. We are currently monitoring the results.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>19:00</var> GMT</small><br><strong>Update</strong> - We have increased available computing power, and are monitoring while we continue to investigate Tabula database underperformance.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>16:04</var> GMT</small><br><strong>Investigating</strong> - We're currently investigating performance problems with the Tabula database, which will also affect My Warwick updates.</p>tag:status.warwick.ac.uk,2005:Incident/58142562020-12-17T10:15:30+00:002020-12-17T10:15:31+00:00WebGroups briefly read-only<p><small>Dec <var data-var='date'>17</var>, <var data-var='time'>10:15</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>17</var>, <var data-var='time'>10:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>09:23</var> GMT</small><br><strong>Scheduled</strong> - It will not be possible to create, edit, or delete web groups for 10 - 15 minutes, starting at 10am on Thursday. Read-only operations will still be available. Applications that use WebGroups to, for example, control access by web group membership, will be unaffected.<br /><br />This change is to allow us to perform essential maintenance on the WebGroups database to improve performance and stability.<br /><br />For advice or support, please contact us: https://warwick.ac.uk/helpdesk</p>tag:status.warwick.ac.uk,2005:Incident/58058502020-12-14T14:00:00+00:002020-12-15T10:36:46+00:00Slowness or Issues uploading files<p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>14:00</var> GMT</small><br><strong>Resolved</strong> - We again observed some slowness since 2:00 PM yesterday(14th Dec) and later some issues today uploading files this morning. We think it has been resolved now since 10:05 AM as suggested from logs.</p>tag:status.warwick.ac.uk,2005:Incident/57961862020-12-14T10:32:29+00:002020-12-14T10:32:30+00:00Large file upload problems<p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>10:32</var> GMT</small><br><strong>Resolved</strong> - We are confident that issues with large file uploads have been resolved.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>10:05</var> GMT</small><br><strong>Monitoring</strong> - A fix has been implemented for large file uploads and we are monitoring performance closely.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>19:30</var> GMT</small><br><strong>Identified</strong> - We are continuing to experience problems with uploads for large files. We have identified the infrastructure component at fault and are working with our software vendor to try and resolve the issue.<br /><br />If you are having trouble submitting coursework to Tabula or assessments to the AEP, please also submit your work via an alternative mechanism such as by email to your department as proof that you submitted on-time. For AEP submission, we record the time that you start uploading your work, so please do not cancel your upload once it has started.<br /><br />There are also problems where large files are generated in the background, such as with downloading zip files of multiple files from Tabula or Files.Warwick.<br /><br />We are working to resolve this problems as soon as possible, and will post regular updates as we have them.</p>tag:status.warwick.ac.uk,2005:Incident/57790952020-12-11T19:37:58+00:002020-12-11T19:37:59+00:00Reported slowness or issues uploading files<p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>19:37</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>19:07</var> GMT</small><br><strong>Monitoring</strong> - We've implemented a fix for slow or failing uploads and are continuing to monitor performance.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>17:20</var> GMT</small><br><strong>Update</strong> - We are continuing to investigate problems with extremely slow processing speed for uploaded files. This is causing large files to cause timeouts and may mean the file isn't accepted at all.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>17:19</var> GMT</small><br><strong>Update</strong> - We are continuing to investigate problems with extremely slow processing speed for uploaded files. This is causing large files to cause timeouts and may mean the file isn't accepted at all.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>16:21</var> GMT</small><br><strong>Identified</strong> - We have isolated the cause of uploads issues to a storage system and are liaising with the support provider to investigate further.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>15:12</var> GMT</small><br><strong>Update</strong> - We are continuing to investigate problems with file uploads as a matter of urgency. This is currently sporadically affecting uploads to Tabula, AEP, Sitebuilder, Files.Warwick, Blogs, Scholarships, Photos and the Wellbeing Portal.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>20:45</var> GMT</small><br><strong>Investigating</strong> - We are investigating reported issues uploading files to Tabula. This is failing intermittently; while we investigate please retry your uploads.</p>