Importing a complex data structure in Salesforce DX

Introduction

When Salesforce DX was announced, I was excited, finally Developers can operate like true developers, using files to build apps instead of going through a UI. However, one feature that came out that excited every DBA (database administrator) bone in my body was the ability to use Data Trees to import data into scratch orgs. It's pretty simple, you create your data in a scratch org, then you export it to JSON files. Then you import it into the next scratch org. The idea is you can build test data for the apps you build with Salesforce DX.
However, when I started toying around with this for my Resume Builder app, it wasn't clear how to relate multiple objects to one another, it looked like only 1 parent object and multiple children, and you can only go down 1 level. So I made a video and this post on how to export an app's data with multiple data levels.

Check out my youtube video on how to manage Salesforce DX data for scratch orgs.

Resume App Schema

Below is the schema of my Resume Builder App. You have the Resume object, with child Achievement, Experience, and Technical Skill records. The Resume looks up to a Contact record (which in turn looks up to an Account), and the Experience records also look up to an Account record.


When writing a query, unfortunately the Data Tree Export can only support going one level up and down, meaning I could export the resume and all the child records, but could not include the Contact records. Similarly, if I tried to export the Contact record, I could go down to the Resume, but couldn't go down to the children of that.

Installing the App

My Resume Builder App is available on Github for you to clone (or fork) to your local machine. Once you've done that, you can register for a DevHub org. Then you can create a scratch org and push the app to it. Please refer to trailhead to familiarize yourself with SFDX, and my Github Wiki to install the app. You can even push test records included in the Repo.

Managing Test Data

Now we're ready to export and import our data.

Exporting the Data

With Salesforce DX, you can write queries to get the representation of the data you want. In this query, I am returning all my Account records, with the fields that I need for the app.

sfdx force:data:soql:query -u DataSource -q “SELECT Id, Name, RecordTypeId, BillingStreet, BillingCity, BillingState, BillingPostalCode, BillingCountry, Phone, Website, Industry, Ownership, Description FROM Account”

Since we don’t really use the Description field, we can drop from query. We are also going to drop record type since it is not support and we will just get an annoying error.

1. Now we can run the Export command to export Account records. We are also requesting a plan to make life easier when we map our data:
   

   sfdx force:data:tree:export -u DataSource -q “SELECT Id, Name, BillingStreet, BillingCity, BillingState, BillingPostalCode, BillingCountry, Phone, Website, Industry, Ownership FROM Account” --prefix demo --outputdir TestData --plan
   

2. We can repeat a similar export for Contact records.
   

   sfdx force:data:tree:export -u DataSource -q “SELECT Id, AccountId, LastName, FirstName, Salutation, MailingStreet, MailingCity, MailingState, MailingPostalCode, MailingCountry, MobilePhone, Email, Title, LinkedIn_Profile__c, Trailhead_Profile__c, Personal_Website__c FROM Contact” --prefix demo --outputdir TestData --plan
   

3. So we got a warning about not having an Account record - we’ll ignore for now and I’ll show you how to over come this.
4. Since Achievements, Experiences, and Technical Skills are detail records of Resume, we are going to export them all together.
   

   sfdx force:data:tree:export -u DataSource -q "SELECT Name, Summary__c, Active__c, (SELECT Job_Title__c, Degree_Earned__c, Completion_Date__c, Start_Date__c, End_Date__c, Current_Position__c, Resume__c, Description__c, Location__c, Course_Name__c, Id FROM Experiences__r),(SELECT Name, Completion_Date__c, Sort_Order__c, Subtitle__c FROM Achievements__r),(SELECT Name, Sort_Order__c, Skills_List__c FROM Technical_Skills__r) FROM Resume__c" --prefix demo --outputdir TestData --plan
   

5. Now that we have all our records exported, we can may them all together so we can import all these at once.

Data Plan

When we open the folder containing our SFDX project, we'll see the test data folder. When we open that folder, we'll see a plan JSON file for Accounts, Contacts, and one for Resume will all the child objects. We'll open that one and work with it. Open demo-Resume__c-Experience__c-Achievement__c-Technical_Skill__c-plan.json and then save the file as resume-data-plan.json.

1. In our editor, open the demo-account-plan.json and copy the block enclosed with curly brackets.
2. Back in the resume json, we'll paste the block and add a comma at the end to separate the object.
3. Now, open demo-contact-plan.json, and copy the block enclosed with curly brackets, and paste in the resume json after the account block, also with a comma to separate.
4. Now let's configure the plan to allow the records to associate to each other. For Contact, change "resolveRefs" to true. What does this do? Well this tells the import process that we want to relate the Contact record to other records that are part of the import. You'll see for Account, "saveRefs" is already true. When the import process creates the Account records, it will store a reference Id, think of it as an external Id; when the process creates the Contact record, we can now look at the Account reference Id to relate them.
5. For Resume, we also want to set "resolveRefs" to true so we can relate the Resume to the Contact record. For the other child records "resolveRefs" is true because they were part of the bigger export, we don't need to saveRefs set to true because they don't have child records.
6. Your plan file should look like the code below. Go ahead and save the file.
   

   [
       {
           "sobject": "Account",
           "saveRefs": true,
           "resolveRefs": false,
           "files": [
           "demo-Accounts.json"
           ]
       },
       {
           "sobject": "Contact",
           "saveRefs": true,
           "resolveRefs": true,
           "files": [
               "demo-Contacts.json"
           ]
       },
       {
           "sobject": "Resume__c",
           "saveRefs": true,
           "resolveRefs": true,
           "files": [
               "demo-Resume__cs.json"
           ]
       },
       {
           "sobject": "Experience__c",
           "saveRefs": false,
           "resolveRefs": true,
           "files": [
               "demo-Experience__cs.json"
           ]
       },
       {
           "sobject": "Achievement__c",
           "saveRefs": false,
           "resolveRefs": true,
           "files": [
               "demo-Achievement__cs.json"
           ]
       },
       {
           "sobject": "Technical_Skill__c",
           "saveRefs": false,
           "resolveRefs": true,
           "files": [
               "demo-Technical_Skill__cs.json"
           ]
       }
   ]
   

Map Records

With the plan saved, let's open the demo-Accounts.json, here we have all the data we want for each data record; in the first block, you see a "referenceId", this is the Id we want to use to map the contact record and experience records to.

1. Let's copy the referenceId for Codey Enterprises. 
2. Next, open the demo-Contacts.json. In the second set of brackets, above "LastName", let's insert the AccountId field, we'll also add an @ symbol to reference the referenceId of the Account record. You're code should look like this:
   

   {
       "records": [
           {
               "attributes": {
                   "type": "Contact",
                   "referenceId": "ContactRef1"
               },
               "AccountId": "@AccountRef1",
   ...
   

3. Copy the ReferenceId for the contact record ContactRef1
4. Next, open the demo-Resume__cs.json. In the second block, insert a new line and reference the Contact record: "Contact__c": "@ContactRef1",
   

   {
       "records": [
           {
               "attributes": {
                   "type": "Resume__c",
                   "referenceId": "Resume__cRef1"
               },
               "Contact__c": "@ContactRef1",
               "Name": "Salesforce Architect",
               "Summary__c": "<p>Innovative Salesforce Certified Application Architect with strong analytical, database, and technical skills. Led the implementation of three separate Salesforce systems as a customer, in the Nonprofit, Financial Services, and Software sectors. Five years experience in Salesforce administration and solutions. Over 10 years experience in requirements definition and refinement, and 4 years of Agile experience. Broad methodology knowledge including Agile Scrum, Scaled Agile Framework (SAFe) and DevOps practices. Achieved Ranger status on Salesforce Trailhead with 150 badges including 4 superbadges.</p>",
               "Active__c": true
           }
       ]
   }
   

5. Now we are going to associate the Experience records to the appropriate Accounts. Open demo-Experience__cs.json, for the first record we see the Degree record "B.S. Computer...", above that let's insert the Organization__c reference field that looks up to the Account, go to the Account record, and copy the ReferenceId for the "University of the United States", so your record should look something like: "Organization__c": "@AccountRef5"
   

   {
       "records": [
           {
               "attributes": {
                   "type": "Experience__c",
                   "referenceId": "Experience__cRef1"
               },
               "Organization__c": "@AccountRef5",
               "Degree_Earned__c": "B.S. Computer Management Information Systems",
   ...
   

6. We'll repeat the same for the other records, here's a rough mapping:
   

   Experience Record	Account
   "Job_Title__c": "Senior Consultant"	Codey Enterprises (ref1)
   "Job_Title__c": "Senior Systems Engineer"	Astro Consulting
   "Job_Title__c": "Salesforce Technical Lead"	Blaze Bank
   "Course_Name__c": "TrailheaDX Architect Bootcamp"	Salesforce
   "Course_Name__c": "Programmatic Development using Apex and Visualforce"	Salesforce
   "Course_Name__c": "Introduction to Object-Oriented Programming using Apex"	Salesforce

   
   
7. Make sure to save all your files, now all the records are mapped together.

Import the Data to a new scratch org

Back in our Command Prompt (or terminal), we are going to create a new scratch org, deploy the Resume Builder app, assign the permission set to the user, and import the data.

1. Create the scratch org with the alias DataTarget:
   sfdx force:org:create -f config/project-scratch-def.json --setalias DataTarget
2. Now that the org is created, we'll push the app to it:
   sfdx force:source:push -u DataTarget
3. With the app create, we need to assign the permission set. This ensures that the user has all the appropriate permissions to our app, including the objects and fields.
   sfdx force:user:permset:assign -u DataTarget --permsetname Resume_Builder
4. Now we're ready for the Data Import. Import Data with Plan:
   sfdx force:data:tree:import --plan -u DataTarget TestData/resume-data-plan.json
5. Open the org:
   sfdx force:org:open -u DataTarget
6. In the scratch org, click on the Menu button and select Resume Builder. Here we see that the Contact record has been associated to Codey Enterprises. When we click on the Resume tab, and open the resume record, we see that it is associated to the Contact record. 
7. When we go to the related tab, we see all the related records, however all the records have the same record type. So when we scroll down to the resume page, we see that the resume is not formatted the way we want. 

Fixing the records

The quickest way to fix our records, is to open the developer console and change the record type ids.

Contact Record

1. In Salesforce, open the Developer Console (gear icon > Developer Console).
2. Query the available record types:
   SELECT Id, DeveloperName, SobjectType FROM RecordType ORDER BY SobjectType
3. Copy the Id value for Resume_Contact.
4. Query the Contact Record:
   SELECT Id, Name, RecordTypeId FROM Contact
5. Replace the value of RecordTypeID with the one copied. If different, there will be a small record tag.
6. Click on Save Rows.

Account Record

1. In the Developer Console, select the RecordType tab.
2. Copy the Id value for Education_Organization
3. Query the Account Record:
   SELECT Id, Name, RecordTypeId FROM Account
4. Replace the value of RecordTypeID for "University of the United States" with the one copied. If different, there will be a small record tag.
5. Click on Save Rows.
6. The RecordTypeId for the other Accounts should already be the Professional_Organization, if not, copy the Id for Professional_Organization and paste it for all other records.

Achievement Record

1. In the Developer Console, select the RecordType tab.
2. Copy the Id value for Accomplishment
3. Query the Achievement Record:
   SELECT Id, Name, RecordTypeId FROM Achievement
4. Replace the value of RecordTypeID for "Guest Speaker..." with the one copied. If different, there will be a small record tag.
5. Click on Save Rows.
6. The RecordTypeId for the other Achievements should already be the Certification Id, if not, copy the Id for Certification and paste it for all other records.

Experience Record

1. In the Developer Console, select the RecordType tab.
2. Copy the Id value for Educational (for Experience__c SobjectType)
3. Query the Educational Experience Record:
   SELECT Id, RecordTypeId, Degree_Earned__c FROM Experience__c WHERE Degree_Earned__c != null
4. Replace the value of RecordTypeID for the degree with the one copied. If different, there will be a small record tag.
5. Click on Save Rows.
6. Copy the Id value for Professional Development (for Experience__c SobjectType)
7. Query the Professional Development Experience Record:
   SELECT Id, RecordTypeId, Course_Name__c FROM Experience__c WHERE Course_Name__c != null
8. Replace the value of RecordTypeId for the Courses with the one copied. If different, there will be a small record tag.
9. Click on Save Rows.
10. The RecordTypeId for the Professional experience records should already be the Professional Id, if not, copy the Id for Professional and paste it for all other records.
11. Open the Resume Builder App, click on the Resume tab and open the resume record. The PDF resume should render properly.

Summary

Now that all of our data looks good, it is worth noting that, especially with Experience records, we can see patterns in our data to determine record types. We could write some apex to automatically set the record type using a pattern. So if we had a large set of test data, it would be easier to handle.