tag:sondehub.statuspage.io,2005:/historySondeHub Status - Incident History2024-03-28T22:41:47ZSondeHubtag:sondehub.statuspage.io,2005:Incident/175655952023-06-13T19:00:00Z2023-06-13T22:02:17ZAPI outage<p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>19:00</var> UTC</small><br><strong>Resolved</strong> - AWS us-east-1 outage caused an outage to APIs which also impacted predictions, frontend and ingestion. The issue has since been resolved by AWS</p>tag:sondehub.statuspage.io,2005:Incident/161568832023-02-21T05:20:45Z2023-02-21T05:20:45ZIncreased demand<p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>05:20</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>03:20</var> UTC</small><br><strong>Monitoring</strong> - With recent interest with balloon tracking SondeHub is under significantly more load than usual. Please consider supporting our work on Patreon https://patreon.com/sondehub or PayPal https://sondehub.org/go/donate</p>tag:sondehub.statuspage.io,2005:Incident/159164822023-01-24T04:13:33Z2023-01-24T04:13:33ZPredictions slow + high error rate<p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>04:13</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>00:00</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:sondehub.statuspage.io,2005:Incident/131704162022-11-16T01:41:38Z2022-11-16T01:41:38ZIngestion issues<p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>01:41</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>01:29</var> UTC</small><br><strong>Identified</strong> - Issue identified with a code deployment to the ingestion pipeline is preventing uploads to SondeHub</p>tag:sondehub.statuspage.io,2005:Incident/116585832022-10-11T09:27:57Z2022-10-11T09:27:57ZSondehub tracker update - please refresh<p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>09:27</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>05:51</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>05:50</var> UTC</small><br><strong>Monitoring</strong> - We have recently (2022-10-8) made some performance improvements to the Sondehub and Sondehub-Amateur Trackers. Please clear your browser cache to update to this version. This notice will still be displayed even after you have updated.</p>tag:sondehub.statuspage.io,2005:Incident/97434632022-04-09T08:38:43Z2022-04-09T08:38:43ZDatabase issue<p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>08:38</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>05:04</var> UTC</small><br><strong>Monitoring</strong> - DB has been restored and now monitoring performance.</p><p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>04:11</var> UTC</small><br><strong>Update</strong> - Database restore is in progress and we are starting to see some shards available again.</p><p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>02:22</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>02:21</var> UTC</small><br><strong>Identified</strong> - Investigating loss of database backend</p>tag:sondehub.statuspage.io,2005:Incident/95615452022-03-18T01:23:00Z2022-03-18T01:23:00ZPredictions slow to update<p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>01:23</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>05:01</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>01:26</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:sondehub.statuspage.io,2005:Incident/87059202021-11-29T11:31:32Z2021-11-29T11:31:32ZPredictions timing out<p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>11:31</var> UTC</small><br><strong>Resolved</strong> - This issue should be now resolved but we will continue monitoring</p><p><small>Nov <var data-var='date'>28</var>, <var data-var='time'>02:21</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Nov <var data-var='date'>28</var>, <var data-var='time'>02:08</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov <var data-var='date'>28</var>, <var data-var='time'>01:09</var> UTC</small><br><strong>Investigating</strong> - Investigating failing predictions</p>tag:sondehub.statuspage.io,2005:Incident/86185642021-11-24T23:05:44Z2021-11-24T23:05:44ZElasticSearch performance upgrades<p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>23:05</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>00:48</var> UTC</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>00:48</var> UTC</small><br><strong>Update</strong> - Scheduled maintenance is still in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>21:51</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>21</var>, <var data-var='time'>09:58</var> UTC</small><br><strong>Scheduled</strong> - Maint on hold</p><p><small>Nov <var data-var='date'>21</var>, <var data-var='time'>08:11</var> UTC</small><br><strong>In progress</strong> - Started maintenance early</p><p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>08:38</var> UTC</small><br><strong>Update</strong> - In line with an ongoing support ticket with AWS we will be downgrading from OpenSearch to ElasticSearch to aid in troubleshooting performance issues of the previous upgrade. During this time historic data might not be available.<br /><br />This operation is slow and we appreciate your patience. If you are planning on using SondeHub for chasing we recommend using chasemapper (https://github.com/projecthorus/chasemapper) for offline use.</p><p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>08:35</var> UTC</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>01:30</var> UTC</small><br><strong>Scheduled</strong> - In line with an ongoing support ticket with AWS we will be downgrading from OpenSearch to ElasticSearch to aid in troubleshooting performance issues of the previous upgrade. During this time historic data might not be available.<br /><br />This operation is slow and we appreciate your patience. If you are planning on using SondeHub for chasing we recommend using chasemapper (https://github.com/projecthorus/chasemapper) for offline use.</p>tag:sondehub.statuspage.io,2005:Incident/84670182021-11-11T11:27:49Z2021-11-11T11:27:49ZPredictions outage<p><small>Nov <var data-var='date'>11</var>, <var data-var='time'>11:27</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>11</var>, <var data-var='time'>10:54</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:sondehub.statuspage.io,2005:Incident/81813072021-10-08T09:50:22Z2021-10-08T09:50:22ZSome minor ongoing infrastructure changes to clear up technical debt<p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>09:50</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>05:25</var> UTC</small><br><strong>Investigating</strong> - Cleaning up resources and configuration from legacy sondehub. Some short disruption might occur as clean up works proceed.</p>tag:sondehub.statuspage.io,2005:Incident/81336832021-10-01T23:59:58Z2021-10-01T23:59:58ZMaintenance to reconfigure some systems<p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>23:59</var> UTC</small><br><strong>Resolved</strong> - Maintenance complete. If you are still having issues try flushing your DNS and restarting your browser.</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>23:38</var> UTC</small><br><strong>Identified</strong> - Briefly change some settings that will break some aspects of the site for a few minutes.</p>tag:sondehub.statuspage.io,2005:Incident/81209602021-10-01T12:03:45Z2021-10-01T12:03:45ZOpenSearch outage<p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>12:03</var> UTC</small><br><strong>Resolved</strong> - The workaround appears to be working our DB vendor is working to resolve the problem.</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>09:32</var> UTC</small><br><strong>Monitoring</strong> - A workaround has been implemented and we are monitoring to investigate if it has worked</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>07:58</var> UTC</small><br><strong>Update</strong> - We are working with our DB vendor to resolve the issue however this may take sometime. Live view should continue to operate normally however predictions and history are likely not performing well.</p><p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>12:52</var> UTC</small><br><strong>Identified</strong> - Performance issues occurring from earlier opensearch upgrade</p>tag:sondehub.statuspage.io,2005:Incident/81171692021-09-30T02:35:30Z2021-09-30T02:35:30ZResizing cluster<p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>02:35</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>01:34</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>01:33</var> UTC</small><br><strong>Scheduled</strong> - Slight outages as the cluster resizes.</p>tag:sondehub.statuspage.io,2005:Incident/78066202021-08-20T13:41:16Z2021-08-20T13:41:16ZInvestigating performance issues querying the database for historic data<p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>13:41</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>12:54</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:sondehub.statuspage.io,2005:Incident/77900572021-08-18T11:00:00Z2021-08-18T11:26:23ZIngestion issues<p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>11:00</var> UTC</small><br><strong>Resolved</strong> - New filtering code incorrectly parsed and discarded data that was valid. This issue has now been resolved</p>tag:sondehub.statuspage.io,2005:Incident/76469832021-07-31T06:08:02Z2021-07-31T06:08:02ZPrediction model out of date<p><small>Jul <var data-var='date'>31</var>, <var data-var='time'>06:08</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>30</var>, <var data-var='time'>23:22</var> UTC</small><br><strong>Identified</strong> - The upstream service (Tawhiri) that we use for getting balloon predictions is currently unable to access NOAA GFS dataset and has an out of date prediction. As such predictions will be inaccurate!</p>tag:sondehub.statuspage.io,2005:Incident/76130242021-07-27T08:50:28Z2021-07-27T08:50:28ZAWS us-east-1 connectivity issues<p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>08:50</var> UTC</small><br><strong>Resolved</strong> - The issue seems to be resolved</p><p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>07:53</var> UTC</small><br><strong>Identified</strong> - AWS are currently having connectivity issues with us-east-1. This is outside our control but we are monitoring the situation.</p>tag:sondehub.statuspage.io,2005:Incident/75922982021-07-24T11:48:39Z2021-07-24T11:48:39ZCluster resizing<p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>11:48</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>11:48</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>11:19</var> UTC</small><br><strong>Identified</strong> - Cluster resizing operation in progress - ETA ~20 minutes</p>tag:sondehub.statuspage.io,2005:Incident/75414222021-07-19T02:59:03Z2021-07-19T02:59:03ZDegraded performance<p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>02:59</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>01:52</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>01:51</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>01:33</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>01:33</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>00:16</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented to resolve the database performance issues. We are monitoring and will likely do some infra changes later today to clean up.</p><p><small>Jul <var data-var='date'>18</var>, <var data-var='time'>22:27</var> UTC</small><br><strong>Identified</strong> - Backend database is having performance issues. We are investigating solutions.</p>tag:sondehub.statuspage.io,2005:Incident/73701082021-06-30T20:53:25Z2021-06-30T20:53:25ZSlow loading times<p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>20:53</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>12:50</var> UTC</small><br><strong>Investigating</strong> - We are investigating a performance issue with the sondehub website. Ingestion of data continues operating as normal.</p>tag:sondehub.statuspage.io,2005:Incident/73293062021-06-25T11:34:38Z2021-06-25T11:34:38ZSondeHub.org outage : ElasticSearch backend failure<p><small>Jun <var data-var='date'>25</var>, <var data-var='time'>11:34</var> UTC</small><br><strong>Resolved</strong> - The cluster has returned to a stable configuration. Measures to prevent this in the future are being investigated.</p><p><small>Jun <var data-var='date'>25</var>, <var data-var='time'>11:24</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jun <var data-var='date'>25</var>, <var data-var='time'>11:22</var> UTC</small><br><strong>Monitoring</strong> - AWS have resolved the core problem and we a monitoring the cluster to ensure it returns to a health state. Backlog of radiosonde data has been successfully processed.</p><p><small>Jun <var data-var='date'>25</var>, <var data-var='time'>10:38</var> UTC</small><br><strong>Identified</strong> - AWS is working to resolve a fault with our ElasticSearch cluster. During this time SondeHub.org website is not be up to date and may fail.<br /><br />WebSockets and ingestion are still functioning normally and data will be processed once the ElasticSearch issue is resolved.</p>