Permissiongeddon MADness! What happened in Salesforce —  The Timeline

Update 9:00 AM Eastern March 20

If you are one of the lucky ones that continued just like another day using your Salesforce platform, then you are blessed. Gov Cloud, Asia Pacific, and Australia customers are the lucky ones. Few North America and EU customers have also dogged the bullet.

Others were not fortunate to land on their feet. As the day began, some users started seeing the data that they should not have access to, and by 11:00 AM Eastern users experienced login issues! The issue was first reported by our customers at 5:30 AM eastern when ComplianceSeal observed the anomalies in the profiles and permission sets.

Disruptions

The first update of the incident was reported on trust.salesforce.com on March 17, 10:30 AM Eastern. At the same time,

  • Customers started to experience performance degradations and issues with users logins (especially community users) including Sandbox environments
  • Standard Salesforce users started seeing access issues to their data.
  • SysAdmins were unable to edit profiles and permissions to restore them to the original state.

Issue

Around 12:00 PM Eastern Salesforce arranged a call acknowledging the problems users are experiencing.

Salesforce communicated that at 9:56 AM UTC, a DML Operation (data update) was made to the Salesforce Metadata for the Partdot users.

At actual DML Change was made 2:00 AM UTC. This change to the Salesforce platform automatically modified Profiles and Permissionssets, giving unwanted elevated access. Users got “Modify All Data” (MADness) permission to all Salesforce data irrespective of they are authorized or not.

If you haven’t taken action against your impacted orgs, then your data may have been exposed for unauthorized access.

A̵s̵ ̵o̵f̵ ̵t̵h̵e̵ ̵w̵r̵i̵t̵i̵n̵g̵ ̵o̵f̵ ̵t̵h̵i̵s̵ ̵a̵r̵t̵i̵c̵l̵e̵ ̵a̵t̵ ̵7̵:̵3̵0̵ ̵P̵M̵,̵ ̵S̵a̵l̵e̵s̵f̵o̵r̵c̵e̵ ̵d̵o̵e̵s̵ ̵n̵o̵t̵ ̵h̵a̵v̵e̵ ̵a̵n̵ ̵E̵T̵A̵ ̵t̵o̵ ̵r̵e̵s̵o̵l̵v̵e̵ ̵t̵h̵i̵s̵ ̵i̵s̵s̵u̵e̵ ̵c̵o̵m̵p̵l̵e̵t̵e̵l̵y̵.̵

A̵t̵ ̵8̵:̵3̵0̵ ̵A̵M̵ ̵E̵a̵s̵t̵e̵r̵n̵,̵ ̵M̵a̵y̵ ̵1̵8̵ ̵S̵a̵l̵e̵s̵f̵o̵r̵c̵e̵ ̵r̵e̵s̵o̵l̵v̵e̵d̵ ̵a̵l̵l̵ ̵t̵h̵e̵ ̵i̵s̵s̵u̵e̵s̵ ̵w̵i̵t̵h̵ ̵L̵o̵g̵i̵n̵.̵ ̵T̵h̵e̵y̵ ̵a̵r̵e̵ ̵w̵r̵i̵t̵i̵n̵g̵ ̵t̵h̵e̵ ̵g̵u̵i̵d̵a̵n̵c̵e̵ ̵t̵o̵ ̵t̵h̵e̵ ̵c̵u̵s̵t̵o̵m̵e̵r̵s̵ ̵o̵n̵ ̵t̵o̵ ̵”̵f̵i̵x̵”̵ ̵t̵h̵e̵ ̵p̵r̵o̵f̵i̵l̵e̵s̵ ̵a̵n̵d̵ ̵p̵e̵r̵m̵i̵s̵s̵i̵o̵n̵ ̵s̵e̵t̵s̵

At 9:00 AM Eastern, May 19, Salesforce is running restoration scripts to get back the profiles to the last known state.

How is Salesforce fixing the issue

Salesforce is working on two workstreams

  • Work Stream 1: Salesforce is working on a remediation script to revert the change. Once this change is made, users will be able to login to their Salesforce. — Completed Successfully. We have seen our customers able to login successfully. If you are still having issues, please reach out to your success manager and open case and they should be able to provide you an update.
  • W̵o̵r̵k̵ ̵S̵t̵r̵e̵a̵m̵ ̵2̵:̵ ̵T̵h̵i̵s̵ ̵i̵s̵ ̵a̵n̵ ̵a̵c̵t̵i̵v̵e̵ ̵w̵o̵r̵k̵s̵t̵r̵e̵a̵m̵ ̵t̵h̵a̵t̵ ̵w̵i̵l̵l̵ ̵b̵e̵ ̵w̵o̵r̵k̵i̵n̵g̵ ̵o̵n̵ ̵r̵e̵s̵t̵o̵r̵i̵n̵g̵ ̵t̵h̵e̵ ̵a̵f̵f̵e̵c̵t̵e̵d̵ ̵p̵e̵r̵m̵i̵s̵s̵i̵o̵n̵s̵ ̵a̵n̵d̵ ̵p̵r̵o̵f̵i̵l̵e̵s̵ ̵t̵o̵ ̵t̵h̵e̵i̵r̵ ̵o̵r̵i̵g̵i̵n̵a̵l̵ ̵s̵t̵a̵t̵e̵.̵ ̵T̵h̵i̵s̵ ̵w̵o̵r̵k̵s̵t̵r̵e̵a̵m̵ ̵w̵i̵l̵l̵ ̵s̵t̵a̵r̵t̵ ̵w̵o̵r̵k̵i̵n̵g̵ ̵a̵s̵ ̵s̵o̵o̵n̵ ̵a̵s̵ ̵t̵h̵e̵ ̵W̵o̵r̵k̵S̵t̵r̵e̵a̵m̵1̵ ̵c̵o̵m̵p̵l̵e̵t̵e̵s̵ ̵t̵h̵e̵i̵r̵ ̵r̵e̵m̵e̵d̵i̵a̵t̵i̵o̵n̵ ̵s̵c̵r̵i̵p̵t̵.̵ This is completed.
  • Restoration Scripts: The restoration scripts are running currently as we speak. It is possible for the scripts to run for several hours. Please look at the setup audit trail. The restoration scripts are also completed for most orgs
  • Sandbox: Salesforce is coming up with a plan to fix the issues, but it is recommended for customers to fix the themselves
  • Link for Issue Tracker: https://success.salesforce.com/issues_view?Id=a1p3A000001SHDl
  • Link on the trust page: https://status.salesforce.com/incidents/3815

Salesforce Responses for some of the Critical Questions:

  • Managed Packages Question: It is not known if customers will be able to restore the managed package permission sets and profile.
    Salesforce Response: Salesforce was able to resolve the issues with managed packages. Two of the solutions that are suggested if you are not seeing the fixes from Salesforce is to ope
  • Standard Objects and Custom Objects Permissions: Administrators cannot fully restore yet especially with Standard objects. 
    Salesforce Response: 89% of the restoration scripts were successful. But some of the restoration scripts failed. You have two options: 
    a) Open a support ticket with Salesforce so they can run the restoration script for your org
    b) Ask your administrator to restore the script from an org that is not impacted or from your backup.
  • Timeline: Exact timeline window when the change happened
    Salesforce Response: Salesforce will update after complete the RCA on request from the customer
  • Pardot Sync: Pardot Sync is not yet restored as of this writing.
  • Question: Why was made to simultaneously to the Disaster Recovery, when the DML change side effect was so big 
    Salesforce Response: Salesforce will update after complete the RCA on request from the customer

Control the Unknown with ComplianceSeal!

ComplianceSeal customers were able to proactively work on the issue from being escalated as they got alerts as soon as profiles and permission sets were modified. This helped them to take corrective action and protect the data and privacy of their customers. ComplianceSeal users were quickly able to restore the permission set and profiles to some extent.

Communication is key

Today’s Salesforce outage by far is the most significant we have experienced in the past nine years. Salesforce should continue updating customers through the Sev0 bridge line as well as post latest updates on https://trust.salesforce.com.

Previous Post
5 Reasons to Attend the Salesforce World Tour, DC

Related Posts

No results found

Leave a Reply

Your email address will not be published. Required fields are marked *

Fill out this field
Fill out this field
Please enter a valid email address.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Menu