Postmortem: # Background

Banner printing is the by which receipts, vendor checks, transcripts, and several other documents are printed. When banner printing is not working it holds up several business processes. The printing setup has multiple components that include an on-premise print application, a web end point to query for queued printing, and several configurations in the core product. The web end point, part of Banner 9 General Self Service, has to be configured for guest authentication in order to be successful.

Break Down of the Problem

On June 30th, at approximately 9:49 AM the first report of Banner printing not working for a receipt printer came in. The responding technician evaluated and determined the scope was all of Banner printing that relied on the restful endpoint. A priority 1 ticket was put in with the vendor to evaluate the current environment’s configuration for the restful endpoint. A UA technician’s evaluation of local log data for the printer application determined that Banner printing for production was unavailable since June 27 at 10:08pm.  During the outage time, a responding technician implemented a work around to the production process not functioning, which was done by manual execution that pulled down the rendered PDF of the print job and sent it to the destination printer. At approximately 8:34pm the vendor working with UA technicians was able to resolve the issue with the web endpoint. 

Target State / Goal

Banner printing services should be available to customers 24/7 except during pre-defined maintenance windows.

Root Cause Analysis

During a recent planned change activity starting on June 27th at 9pm, certificates were being updated across the Ellucian hosted production environment. During this update, the web endpoint was redeployed. After the endpoint returned to service, the process used to retrieve print jobs failed to be able to retrieve them with an HTTP 401 error indicating that the account wasn’t authorized. Technicians validating the change did not notice this as the web location was validated via other means. Technicians escalated to the vendor when first noticed on June 30th at 9:50am, and soon after shared with the vendor their expectation that a setting for guest authentication wasn’t correctly configured. Vendor believed they verified the setting was in place, and continued to evaluate other items. Vendor determined at 8pm that the setup parameter was incorrect and changed around April 10, for an undetermined reason. The parameter was corrected to match the same configuration in the TEST environment and the web end point redeployed restoring service. 

Develop Countermeasures

• Engage vendor to prevent future similar failures for their deployments.
• Require technicians to validate that after a Banner 9 General Self Service Deployment that the Banner Print Application can connect to the web endpoint. 
• Review ability to set up a monitor for immediately alerting around failures for the Banner Print Application to connect to the Banner 9 General Self Service endpoint.

Implementation of Countermeasures

• Jul 1 11:00 AM: Engage vendor about how they will be able to prevent this from occurring in the future. 
• July 16: Create documentation for validators to follow to ensure the Banner 9 General Self Service deployments are correctly accepting connections from the Banner Print Application. 
• July 16: Complete review of ability to set up a monitor for immediately alerting technicians at the first stage of failure for the Banner Print application to connect to the Banner 9 GeneralSSB endpoint.

Follow Up / Review

• July 2: Ensure vendor is working through an internal procedure to ensure this issue is prevented in the future.
• July 18: Verify documentation of validation was created for post validation of Banner 9 General SSB 
• July 18: Review/Discuss feasibility of monitoring the connection status for immediately alerting. If feasible, define follow up tasks for improvement.

Resolved: The OIT Enterprise Applications Services team has resolved the issue with Banner Printing. After working with the vendor, it was determined there was an issue on the vendor's side, who will be reviewing how the error occurred. We appreciate everyone's patience while this issue was being resolved.

Monitoring: A fix has been implemented, and the Banner Printing outage appears to be resolved. We are actively monitoring the issue, but pending prints are now flowing and being processed. Thank you for your patience during this incident.

Identified: There is no new information available at this time, however Banner print jobs should still be processing with the manual workaround implemented by EAS, and print jobs will continue to be manually pushed throughout the night. This will be the last update posted tonight, and we will begin updating this status page again tomorrow at 7:30am. Thank you for your continued patience during this incident.

Identified: The OIT EAS team is continuing to work with the vendor to resolve this issue, and there is no new information available at this time. Banner print jobs should still be processing with the manual workaround implemented by EAS. Print jobs will continue to be manually pushed throughout the night. Thank you again for your patience as we work to resolve this incident.

Identified: The OIT EAS team is continuing to work with the vendor to resolve this issue, and there is no new information available at this time. Banner print jobs should still be processing with the manual workaround implemented by EAS. Print jobs will continue to be manually pushed throughout the night. Thank you again for your patience as we work to resolve this incident.

Identified: The OIT EAS team is continuing to work with the vendor to resolve this issue, and there is no new information available at this time. Banner print jobs should still be processing with the manual workaround implemented by EAS. Thank you again for your patience as we work to resolve this incident.

Identified: The OIT EAS team is continuing to work with the vendor to resolve this issue, and there is no new information available at this time. Banner print jobs should still be processing with the manual workaround implemented by EAS. Thank you again for your patience as we work to resolve this incident.

Identified: The OIT EAS team is continuing to work with the vendor to resolve this issue, and there is no new information available at this time. Banner print jobs should still be processing with the manual workaround implemented by EAS. Thank you again for your patience as we work to resolve this incident.

Identified: The OIT EAS team is continuing to work with the vendor to resolve this issue, and there is no new information available at this time. Banner print jobs should still be processing with the manual workaround implemented by EAS. Thank you again for your patience as we work to resolve this incident.

Identified: The OIT EAS team is continuing to work with the vendor to resolve this issue, and there is no new information available at this time. Banner print jobs should still be processing with the manual workaround implemented by EAS. Thank you again for your patience as we work to resolve this incident!

Identified: The OIT Enterprise Applications Services team is continuing to work with the vendor to resolve this issue. OIT EAS has implemented a manual workaround and users should start seeing their expected printed documents within the next several minutes, if they're not already there. Thank you for your continued patience.

Identified: The OIT Enterprise Applications Services team is still working with the vendor to resolve this issue. OIT EAS has implemented a manual workaround and users should start seeing their expected printed documents within the next several minutes, if they're not already there. Thank you for your continued patience.

Identified: The OIT Enterprise Applications Services team is still working to resolve this issue. Thank you for your continued patience.

Identified: The OIT EAS team is aware of a problem stopping all Production Banner Printing from working correctly. EAS believes they have identified the issue, and are working with the vendor to resolve it. Thank you for your patience, we will try to have an update every thirty minutes.

Resolved: NTS CLS reports that this issue has been resolved. A Kaltura accessibility feature intended for regular video players was conflicting with live stream content.

Thank you again for your time and patience as we resolved this issue!

Investigating: Kaltura has indicated that they are investigating this issue and expect a resolution within 24 hours. NTS CLS is continuing to monitor their channels of communication with them, and we will provide updates as they're given and at 3-hour intervals.

Thank you again for your patience and understanding as we work towards a full restoration of service!

Investigating: The NTS CLS team reached out to the vendor earlier this morning and is currently awaiting a response. We will continue to report any updates as they come and at 3-hour intervals.

Thank you again for your patience as we continue to work this issue.

Investigating: We have received reports that the Livestreaming function of Kaltura is currently experiencing issues. The NTS Collaboration and Learning Spaces team is currently investigating; we will be making updates as more information is made available and at 3-hour intervals.

Thank you for your patience during the resolution of this issue!

Resolved: The OIT Networking team has resolved the issue with internet connectivity at the Kodiak Marine Science Center and determined the fix to be stable. Thank you again for your patience while this issue was resolved!

Monitoring: The Networking team has instituted a fix, and connectivity should now be restored at the Kodiak MSC. We will continue monitoring the situation for stability for a time. Thank you for your patience during the resolution of this issue.

Investigating: We have been notified of a Wi-Fi outage affecting the Kodiak Marine Science Center, and the OIT Networking Team is currently investigating. Thank you for your patience during the resolution of this issue, and we will try to post updates every hour.