Gradebook Failing to Update Grades

Incident Report for Top Hat

Resolved

Queued grades have now been successfully processed for all question types and attendance as of 7:09pm ET today.

There are a small number of secure attendance grades that will be reprocessed early next week to account for geolocation. This means we expect a small subset of secure attendance grades to be corrected early next week to account for geo-verification - this is expected behaviour.
Posted Oct 03, 2025 - 19:18 EDT

Update

Work is still underway to process the queue of grades (inclusive of classic questions and attendance grades) that were impacted. We are already 50% complete and expect queued grades to be fully processed by close of business today. New Builder Questions (or Advanced Questions) have already been appropriately processed.
Posted Oct 03, 2025 - 13:22 EDT

Monitoring

Our system upgrade successfully completed at 6:40am ET. We are currently monitoring our system as our backlog queued grades continue to process.
Posted Oct 03, 2025 - 07:55 EDT

Update

Database migration is well underway. During this period, users can expect significant service degradation, including increased latency and slower load times. We currently estimate that the migration will take several hours to complete and will provide an update as the migration completes in the early morning.
Posted Oct 02, 2025 - 23:53 EDT

Update

Platform latency issues have been resolved for now after the team increased the size of the database cluster, which has improved overall performance.

To minimize disruption to ongoing evening classes, we plan to begin the database migration for the US server at 8:00 PM EST. During this period, users can expect significant service degradation, including increased latency and slower load times. We currently estimate that the migration will take several hours to complete. However, the Engineering team is also exploring a potential solution that could help reduce downtime.

We will continue to provide updates as the migration progresses.
Posted Oct 02, 2025 - 15:39 EDT

Identified

We’ve confirmed that the Gradebook issue is preventing instructors from updating grades. While students can still submit responses, their grades will not appear right away. To resolve this, our engineering team will be migrating the database. During this time, all submissions will be safely queued and processed once the migration is complete.

This issue is also contributing to increased latency and slower load times across the platform. We expect the migration to take several hours, after which all queued grades will be updated. We’ll continue to share updates as progress continues.
Posted Oct 02, 2025 - 12:35 EDT

Investigating

We are currently investigating an issue with the Gradebook where grades are failing to update. The problem began around 9:30 AM ET this morning. Our engineering team is actively working to identify the cause and restore full functionality. We’ll provide an update as soon as more information becomes available.
Posted Oct 02, 2025 - 11:29 EDT
This incident affected: Top Hat (Top Hat API).