tag:status-tokyo.sendbird.com,2005:/historySendbird Tokyo Status - Incident History2024-03-29T20:54:42+09:00Sendbird Tokyotag:status-tokyo.sendbird.com,2005:Incident/203906762024-04-03T07:30:00+09:002024-03-29T04:08:15+09:00Major version upgrade of the database (Dashboard)<p><strong>THIS IS A SCHEDULED EVENT Apr <var data-var='date'>3</var>, <var data-var='time'>07:30</var> - <var data-var='time'>07:35</var> JST</strong></p><p><small>Mar <var data-var='date'>29</var>, <var data-var='time'>04:08</var> JST</small><br><strong>Scheduled</strong> - Date and Time<br />* Tue Apr 2nd 22:30 - 22:35 UTC<br />* Tue Apr 2nd 15:30 - 15:35 PDT<br />* Wed Apr 3rd 07:30 - 07:35 KST<br /><br />Duration of downtime: Up to 5 minutes during the maintenance window<br /><br />Purpose: Major version upgrade of the database (Dashboard)<br /><br />Impact:<br />* The Dashboard will be unavailable<br />* The Desk service will be impacted during the maintenance window, as Desk Agents will be unable to log into Dashboard to handle tickets.<br /><br />What this means:<br />* Customers will not be able to access the Sendbird Dashboard. Calls dashboard, Moderation dashboard, and Desk agent dashboard<br />* Specifically for Desk customers’ users will still be able to create tickets, however agents will not be able to answer tickets during the actual maintenance time. We recommend our customers set working hours within the Dashboard to send auto messages, during the maintenance window<br />* NOTE: Sendbird Calls, Chat, and Desk services themselves will not be interrupted during the maintenance activity</p>tag:status-tokyo.sendbird.com,2005:Incident/199895872024-02-26T07:00:56+09:002024-02-26T07:00:56+09:00All services impacted during bi-yearly maintenance<p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>07:00</var> JST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>06:00</var> JST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>15:54</var> JST</small><br><strong>Scheduled</strong> - * Date and Time<br /> * Feb 25 21:00 - 22:00 UTC<br /> * Feb 25 13:00 - 14:00 PST<br /> * Feb 26 06:00 - 07:00 KST<br />* Duration of downtime: 20 minutes during the maintenance window<br />* Purpose: major version upgrade of the database (bi-yearly)<br />* Service impact<br /> * Full outage (APIs will get 5xx errors) during the downtime<br /> * Announcements, GDPR will be stopped for an hour beforehand<br />* Affected product: Chat, Desk, and Calls</p>tag:status-tokyo.sendbird.com,2005:Incident/200246462024-02-20T09:04:18+09:002024-02-20T09:04:18+09:00Elevated API Errors<p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>09:04</var> JST</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>09:03</var> JST</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-tokyo.sendbird.com,2005:Incident/199458192024-02-09T16:55:37+09:002024-02-09T16:55:37+09:00Message Threading partially broken<p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>16:55</var> JST</small><br><strong>Resolved</strong> - Hotfix has been deployed and we have confirmed that threading is back to operational</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>16:16</var> JST</small><br><strong>Identified</strong> - We have identified that the message threading functionality is partially broken. <br />Creating the first threaded message works, but adding further messages to existing threads have been broken since Feb 8, 10 am PST<br />We are rolling out the hotfix immediately (Feb 8 11 pm PST)</p>tag:status-tokyo.sendbird.com,2005:Incident/160204072023-02-04T05:04:01+09:002023-02-04T05:04:01+09:00Elevated API Errors<p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>05:04</var> JST</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>05:02</var> JST</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-tokyo.sendbird.com,2005:Incident/157063872022-12-25T05:32:03+09:002022-12-25T05:32:03+09:00Elevated API Errors<p><small>Dec <var data-var='date'>25</var>, <var data-var='time'>05:32</var> JST</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Dec <var data-var='date'>25</var>, <var data-var='time'>05:29</var> JST</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-tokyo.sendbird.com,2005:Incident/97129552022-04-05T21:37:04+09:002022-04-05T21:37:04+09:00Elevated API Errors<p><small>Apr <var data-var='date'> 5</var>, <var data-var='time'>21:37</var> JST</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Apr <var data-var='date'> 5</var>, <var data-var='time'>21:36</var> JST</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-tokyo.sendbird.com,2005:Incident/85543292021-11-17T07:40:47+09:002021-11-17T07:40:47+09:00Partial outage of chat service for 20 minutes.<p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>07:40</var> JST</small><br><strong>Resolved</strong> - There was an incident at 3:30am JST, Nov 17, 2021 for 20 minutes. <br /><br />Cause: We're still investigating it.<br /><br />Customers impacted: Chat servers were partially out during that time, users could not make a WebSocket connection.<br /><br />Resolution:<br />- It happened during a few deployments, we rollbacked immediately when we found it first.<br />- The system became operational at 3:50am JST, Nov 17, 2021.</p>tag:status-tokyo.sendbird.com,2005:Incident/82626852021-10-25T00:25:36+09:002021-10-25T00:25:36+09:00Chat unavailability during planned system maintenance<p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>00:25</var> JST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>00:00</var> JST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>18:19</var> JST</small><br><strong>Scheduled</strong> - When: Monday, 10/25 00:00am - 01:00am KST/JST (Sunday, 10/24 15:00pm - 16:00pm GMT)<br /><br />Impact: Chat is estimated to be unavailable for a 15 minute block during the maintenance period.<br /><br />Background: As a follow up action from the previous upgrade, we are updating again database components in our AWS infrastructure in preparation for an upcoming end-of-life (EOL) event.</p>tag:status-tokyo.sendbird.com,2005:Incident/81073542021-10-09T00:33:29+09:002021-10-09T00:33:29+09:00Chat unavailability during planned system maintenance<p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>00:33</var> JST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>00:00</var> JST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>10:00</var> JST</small><br><strong>Scheduled</strong> - When: Friday, Oct 9 00:00am - 01:00am, KST/JST (Friday, Oct 8 15:00pm - 16:00pm, GMT)<br /><br />Impact: Chat is estimated to be unavailable for a 15 minute block during the maintenance period.<br /><br />Background: We are updating several components in our AWS infrastructure including an Aurora database update in preparation for an upcoming end-of-life (EOL) event.</p>tag:status-tokyo.sendbird.com,2005:Incident/76695142021-08-03T07:58:45+09:002021-08-03T14:44:14+09:00Significant network issue on chat service for 43 minutes.<p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>07:58</var> JST</small><br><strong>Resolved</strong> - There was a significant issue with network connectivity.<br /><br />* Resource affected: AP-1 (Tokyo)<br />* When: 7:11 am – 7:58 am, Aug 3, 2021 KST (10:11 PM - 10:58 PM, Aug 2, 2021 UTC)<br />* Possible cause: We deployed a new image but it turned out not to configure CLB Nginx setting right, hence Nginx failed to get requests from CLB.<br />* Customer impact: Customers couldn’t connect to Sendbird and get a proper response either.<br />* Countermeasures and Next step:<br /> - We rollbacked to the previous version, and the service was back to operational.<br /> - We are still investigating why it happened and follow up through a postmortem.</p>tag:status-tokyo.sendbird.com,2005:Incident/75075322021-07-15T05:13:31+09:002021-07-15T05:13:31+09:00Elevated API Errors<p><small>Jul <var data-var='date'>15</var>, <var data-var='time'>05:13</var> JST</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Jul <var data-var='date'>15</var>, <var data-var='time'>05:12</var> JST</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-tokyo.sendbird.com,2005:Incident/74984152021-07-14T09:13:46+09:002021-07-14T09:13:46+09:00Elevated API Errors<p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>09:13</var> JST</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>09:13</var> JST</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-tokyo.sendbird.com,2005:Incident/73547882021-06-29T04:42:40+09:002021-06-29T04:42:41+09:00Elevated API Errors<p><small>Jun <var data-var='date'>29</var>, <var data-var='time'>04:42</var> JST</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Jun <var data-var='date'>29</var>, <var data-var='time'>04:41</var> JST</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-tokyo.sendbird.com,2005:Incident/73351212021-06-26T11:44:51+09:002021-06-26T11:45:06+09:00Response delay on Chat service<p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>11:44</var> JST</small><br><strong>Resolved</strong> - [Incident Report]<br />There were significant issues on AP-1, but we identified it late as it wasn't by Health Checker.<br /><br />Resource affected: AP-1 (Tokyo)<br /><br />When: <br />- 7:20 AM ~ 7:40 AM, June 23 2021 UTC (4:20 PM ~ 4:40 PM, June 23 2021 JST) <br />- 5:20 AM ~ 5:40 AM, June 24 2021 UTC (2:20 PM ~ 2:40 PM, June 24 2021 JST)<br /><br />Impact: Customers in ap-1 may have experienced chat response delay or failure.<br /><br />Possible cause: There was an update for the defect of the open channel event, but it caused a heavy load on DB. <br /><br />Next step:<br />- The issue has been fixed at 14:40 on June 24 KST, and the system backed to normal immediately.<br />- We will follow up on this through postmortem soon if there is any potential risk or not.</p>tag:status-tokyo.sendbird.com,2005:Incident/71410322021-06-02T13:07:40+09:002021-06-02T13:07:40+09:00Chat issue with ad blockers when using JavaScript SDK<p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>13:07</var> JST</small><br><strong>Resolved</strong> - This issue has been resolved. Sendbird has been removed from the EasyList blacklist.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>02:45</var> JST</small><br><strong>Investigating</strong> - Impact: Chat unavailable when using JavaScript SDK along with ad blockers that have privacy tracking enabled<br /><br />When: Starting Friday, May 28th around 11:30p EDT<br /><br />Cause:<br />We’ve recently been made aware of a change made to EasyList, a filter list provider for extensions such as Adblock Plus and uBlock Origin, where our WebSocket domain has been blacklisted. As a result, end users utilizing Adblock Plus, uBlock Origin or any similar service that uses EasyList as a source will be unable to connect to Sendbird.<br /><br />This issue impacts JavaScript SDK implementations only when used with ad blockers that have privacy tracking enabled.<br /><br />Next Step:<br />We are actively engaged with EasyList to address this and hope to resolve it as soon as possible. In the meantime, end users can utilize some potential workarounds below:<br />- Whitelist the sendbird.com domain (configuration varies from ad blocker to ad blocker; suggested pattern: @@||*.sendbird.com/*)<br />- Turn off the blocker for the page/site where chat is utilized.<br /><br />Sendbird does not utilize any user information for retargeting or any advertisement.<br /><br />Also refer to the following community post for updates: https://community.sendbird.com/t/ad-blocker-issues-in-using-js-sdk/2383.</p>tag:status-tokyo.sendbird.com,2005:Incident/67991422021-04-21T06:01:42+09:002021-04-21T06:01:44+09:00Elevated API Errors<p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>06:01</var> JST</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>05:55</var> JST</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-tokyo.sendbird.com,2005:Incident/67100712021-04-09T11:25:38+09:002021-04-09T11:25:38+09:00Elevated API Errors<p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>11:25</var> JST</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>11:23</var> JST</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-tokyo.sendbird.com,2005:Incident/66447832021-03-31T08:29:32+09:002021-03-31T08:29:32+09:00Elevated API Errors<p><small>Mar <var data-var='date'>31</var>, <var data-var='time'>08:29</var> JST</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Mar <var data-var='date'>31</var>, <var data-var='time'>08:27</var> JST</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-tokyo.sendbird.com,2005:Incident/66447262021-03-31T08:23:33+09:002021-03-31T08:23:33+09:00Elevated API Errors<p><small>Mar <var data-var='date'>31</var>, <var data-var='time'>08:23</var> JST</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Mar <var data-var='date'>31</var>, <var data-var='time'>08:20</var> JST</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-tokyo.sendbird.com,2005:Incident/66282022021-03-28T23:09:34+09:002021-03-28T23:09:34+09:00Elevated API Errors<p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>23:09</var> JST</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>23:04</var> JST</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-tokyo.sendbird.com,2005:Incident/63396912021-02-19T23:31:21+09:002021-02-20T14:07:59+09:00Partial outage on chat service.<p><small>Feb <var data-var='date'>19</var>, <var data-var='time'>23:31</var> JST</small><br><strong>Resolved</strong> - There was an incident from AWS affecting our ap-1 region.<br /><br />When: <br />23:12 ~ 23:31, Feb 19, 2021 KST (14:12 ~ 14:31, Feb 19, 2021 UTC)<br /><br />Impact:<br />We observed multiple failures during that time in ap-1.<br /> <br />Possible cause:<br />AWS experienced an increase in ambient temperatures within a section of a single Availability Zone within the AP-NORTHEAST-1 Region. Starting at 6:03 AM PST, some EC2 instances were impaired and some EBS volumes experienced degraded performance as a result of the increase in temperature. The root cause was a loss of power to the cooling system within a section of the affected Availability Zone.<br /><br />Next step:<br />We excluded instances affected and newly rolled them out. The system has been operational at 23:31 Feb 19, for almost all instances in the region. Finally, all have been operational including the rest of the customer's instances.</p>tag:status-tokyo.sendbird.com,2005:Incident/58218382020-12-17T14:04:38+09:002020-12-17T14:04:40+09:00Elevated API Errors<p><small>Dec <var data-var='date'>17</var>, <var data-var='time'>14:04</var> JST</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Dec <var data-var='date'>17</var>, <var data-var='time'>14:03</var> JST</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-tokyo.sendbird.com,2005:Incident/56567582020-11-26T06:06:39+09:002020-11-26T06:06:39+09:00Elevated API Errors<p><small>Nov <var data-var='date'>26</var>, <var data-var='time'>06:06</var> JST</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Nov <var data-var='date'>26</var>, <var data-var='time'>06:05</var> JST</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status-tokyo.sendbird.com,2005:Incident/54297762020-10-28T10:11:54+09:002020-10-28T10:11:54+09:00Elevated API Errors<p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>10:11</var> JST</small><br><strong>Resolved</strong> - The servers have returned operational.</p><p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>10:11</var> JST</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>