D7 to D9 in an Hour with AMA

Goal

Learn how to migrate a Drupal 7 site to Drupal 9 using Acquia Migrate: Accelerate, and efficiently investigate migration messages.

Prerequisites

• https://dev.acquia.com/blog/acquia-migrate-accelerate-now-open-source

Overview

Let me take you on a tour of actually using AM:A.

(And if you've never done a migration from Drupal 7 to Drupal 8/9/10 before … you'll get to see what a whirlwind it is 🌪️😅)

Requirements

• Generate a Drupal 9 site

  Follow the steps outlined in the "Want it now?" section of "Acquia Migrate: Accelerate — now open source!" 🧑‍💻

  You should see something like this:

  

  Now let's actually use it! 🥳

Warning
The FAQ says we should run the generated AM:A project using PHP 7.4. Primarily most migrations were written against PHP 7, and PHP 8 made non-strict (==) comparisons behave differently, which is used a lot in migrations! Unambiguous signs that this is the case: error messages starting with Deprecated function: or PHP Warning:. I didn't do that here. Let's find out how far I get.

• Start migrating: 4 clicks to >50% 😮

  As you may have noticed in the terminal output, this is my personal site. I do not want to think about which data to migrate. I just want to see how far AM:A would get me. I also don't want to bother with switching from PHP 8.1 to PHP 7.4.

  Seems to have worked fine? 🤷‍♂️ Because this is what I see:

  

  So, I just went ahead and did 2 clicks:

  

  Not bad, those two clicks triggered migrations that took half a minute and got me to 645/3466 rows imported, or 53.86%:

  

  The number of errors crept up from 3 tot 23 … but that seems like a low enough number to continue … so let's just continue — I'm too curious to see how far we can get, and besides, we can easily reset to the original state. I've spent less than 10 minutes on this so far anyway! 😁

  So: let's repeat those 2 clicks from earlier! The result (after another half a minute):

  
Note
🤨 Some of you spotted that 645/3466 is a ratio much lower than 54%. And 37/3466 is definitely not 12% either. So why does the AM:A UI report these progress counts? The number of rows that has been migrated is an incomplete assessment. The number of migrations that have already been fully completed is also important to take into account. For each migration, typically a few patterns emerge across the failing rows. Once you figure out the pattern, it's relatively simple to get all rows in that migration to work! The AM:A UI's progress indicator conveys relative total progress.
Warning
PHP warnings/errors aside, even just seeing "⚠️ Dependencies need review" is sufficient to warrant an investigation.AM:A shows this when dependencies of this migration were imported. There's little point in forcefully executing this migration: that would just lead to an avalanche of more migration errors.

• Switch to PHP 7.4 ⇒ composer install + Drupal reinstall

  My preferred development environment is the one with the least layers of abstractions: an AMP stack in macOS directly, as described in this guide. That makes switching PHP versions trivial for me:

  copy

  sphp 7.4

  Hopefully it's equally simple for you!

  Packages must be reinstalled now that the environment has changed. First remove your composer.lock, to ensure composer chooses the right packages to install. Then run composer install manually (in step 1 above, acli app:new:from:drupal7 did that for us):

  copy

  rm composer.lock
  composer install

  This yields mostly the same packages, but some did change — hopefully these will help us be more successful in the migration:

  copy

   - Downgrading symfony/finder (v6.3.3 => v5.4.27): Extracting archive
   - Downgrading symfony/filesystem (v6.3.1 => v5.4.25): Extracting archive
   - Downgrading league/uri (6.8.0 => 6.7.2): Extracting archive
   - Downgrading symfony/string (v6.3.2 => v5.4.26): Extracting archive

  Finally: execute the commands in step 7 again:

  copy

  vendor/bin/drush site-install✂️
  jq -r '.installModules✂️
  vendor/bin/drush state:set✂️

  This will automatically wipe the Drupal 9 database too, and will set us up for a fresh new attempt. If you're used to switching PHP versions, this probably took you less than a minute. (If not, well, congrats, you've just figured out something that'll be useful in the future anyway!)

• Repeat 4 clicks … same result

  After arriving at the freshly installed D9-on-7.4 site's AM:A dashboard, I see exactly the same numbers in the top right.

  Repeating the first 4 clicks … still exactly the same numbers.

  Despite my site being so simple, I am unfortunately running into a hard blocker 😔 (Quite possibly this is happening because it was originally a Drupal 5 site.)

  The only way forward is to dive right in!

Caution
Special care is necessary for Drupal 7 sites whose database is actually an upgraded Drupal 5/6 site. In those days, Drupal provided automatic upgrade paths: from 5 to 6 and 6 to 7. But not from 7 to 8 — that's why AM:A even exists. Those upgrade paths worked … kinda 🧟 There are plenty of examples to be found about incomplete/inconsistent data in the upgraded databases. That's exactly why since Drupal 8, we made Drupal upgrades easy forever, and we achieved that by requiring that every update has test coverage. Unfortunately, for sites originally built on Drupal

• Investigate 🕵️

  Where to begin investigating? In the migration that has the highest absolute or relative failure rate. By either metric, that is definitely Document media items for me:

  
  
  

  Which migration plugins exist in this migration, which one seems likely to be the cause?

  I first click the "Document media items" link, and not the "255" link, to go to its Details tab:

  

  Apparently AM:A has clustered 8 migration plugins into this single migration (the ones with a colon in them are derived migration plugins). The first 5 do not have a (xx of yy) indicator at the end — that's because these provide the supporting configuration (these all refer to field type/formatter/widget-esque things.)
  The last 3 migration plugins do have those numbers, because they're doing the actual data migration.

  There are 255 errors and 254 imported rows for the d7_file_plain::public migration. That's suspiciously similar.

  Also … the :: looks out of place — the two other data migrations have a string between the colons.

  Do previews work at all?

• Testing the hunch

  My hunch is that:

  • rolling back the 3 "media items" migrations
  • setting filemime="text/x-diff" on the rows without a value in the file_managed table in the Drupal 7 database
  • vendor/bin/drush cr (to wipe all caches in the migration system — and forcing the plugin deriver to run again)
  • re-importing those three migrations

  … might be enough to unblock me!

  So let's roll back these migrations, which is easy in the AM:A UI:

  

  A second or so after I ran drush cr, the Details tab that I had already open, automatically updated and showed this:

  

  Succes! 🥳 From 299 rows to migrate down to 48. It looks like this really was just a hitherto undiscovered edge case! (And that makes sense: the Drupal 7 site's file_managed table only contains 271 rows — that was another sign.)

  210 messages are still there though. That's a good reminder that we don't really know what the indirect damage was of this unhandled edge case. I can't wait to see if this results in a successful migration, but I really should reset my environment again (see what we did to switch to PHP 7.4, but this time there's no need to reinstall packages), to avoid such side effects biting me later.

  Still, a quick sanity check of re-importing those 3 "media items" migrations results in this, 10 seconds later:

Note
When rolling back migrations, immediately after it finishes, you'll see a progress bar at the top of the dashboard that says "Re-importing supporting configuration (N) after a rollback…". That means the supporting configuration (such as field types, formatters, widgets, node types, translation settings …) for the migrations that were rolled back were automatically re-imported. That supporting configuration is necessary for previews to work. It's also virtually always very fast: there's typically far less configuration than actual content/data to migrate.

• Fixed D7 DB ⇒ Drupal reinstall

  I know for a fact that most of those 431 messages are now obsolete. And it's more work to figure out if the >100 new messages are legitimate or not.

  So: time to restart the migration environment again: reinstall Drupal 9, to have a clean destination slate.

  Execute the commands in step 7 again:

  copy

  vendor/bin/drush site-install✂️
  jq -r '.installModules✂️
  vendor/bin/drush state:set✂️

  … just like we did when switching to PHP 7.4.

  I personally like to keep things simple while reducing the probability of mistakes. So I like to concatenate repetitive commands with &&, to allow me to quickly find it in my terminal's history and re-execute it:

  copy

  vendor/bin/drush site-install✂️ && jq -r '.installModules✂️ && vendor/bin/drush state:set✂️

• drush ama:import --all

  Now that we can expect the migration to reach ~98% again with just 10 or so clicks (due to dependencies between migrations, we'd have to repeat 8 clicks spread across ~5 minutes.

  What if we could avoid those clicks?

  What if we could ask AM:A to just keep importing whatever the next set of migrations is whose dependencies are met?

  

  And while that is happening, I can following along in my browser:

  

  (On a bigger site, this would take hours instead of minutes, in that case I could investigate the messages of the current migration, live. Or investigate the messages of an already finished migration.)

• 98%! 🥳

  Yay:

  

  The last ETA update:

  copy

  🏁 

  3 minutes to re-migrate an entire site from Drupal 7 to 9. Not bad! 🚀

• Time check ⏱️

  This is the result, after:

  • 5 minutes to generate a Drupal 9 AM:A project from my Drupal 7 site
  • 2 minutes to get to ~54%
  • 1 minute to switch to PHP 7.4
  • 2 minutes to get to ~54% … again
  • 40 minutes to investigate a problem
  • 2 minutes to test the hunch
  • 3 minutes to re-run the entire migration

  That leaves me with a few minutes in my 1 hour to do what that screen is telling me! 🤓

• Review, step #1: migration messages

  There are 47 migration messages, of which 28 are entity validation errors (which I can fix manually, worst case), and 19 actual migration errors. Let's focus on those first.

  2 are for filters, and they have suggested solutions ("PHP code" filter 🙈). 1 is for a field formatter that no longer exists, and it also has a suggested solution. 15 are for menu links to views that do not exist (Views has no migration path, unfortunately — you'll have to recreate views manually!), 1 is for a vocabulary I do not want to migrate anyway.

  So: tweak the text formats by hand, I don't care about the formatter, I'll have to create the views anyway and I'll just delete the vocabulary ✅

• Review, step #2: missing rows

  Looking at the migrations that did not reach 100% of rows imported, for me those are:

  • User accounts: 1 row in d7_google_analytics_user_settings fails to migrate, but I do not want to continue using Google Analytics anyway.
  • Terms: 1 row in d7_taxonomy_term:vocabulary_2, but this was a demo vocabulary anyway, now obsolete (interestingly, the term name was null! 😮)
  • Block placements in my custom theme: 3 rows in d7_block:wimleers_v2:simple — of course Drupal cannot know which regions to map to! I'll fix this manually.
  • Site configuration: action_settings, d7_system_authorize and d7_action each had a row that did not import. None of these are actually important: I will uninstall the Actions module.

  The cruft you find! 🕸️ Also, delightfully, this means:

  • skipping migrations of data I no longer need in the initial "Data to migrate" screen
  • using the module auditor (see next)

  … should be enough to reach 100%! ✅

• Review, step #3: Module Auditor

  AM:A helps doing a 1:1 migration. It incorporates best practices (such as switching to Media + Media Library), but does not know which functionality you may no longer need.

  As long as it's pure functionality modules (rather than modules storing/managing/touching data), it's safe to just uninstall modules: the number of total rows to migrate will not change. But some will affect the migrated data. So it's better to uninstall them before going to the "Select data to migrate" screen in that case!

  This is what the Module Auditor screen looks like for me:

  

  This makes it simple for me to see which contrib modules AM:A automatically installed for me (AM:A will also "composer install" but not "Drupal install" modules without a vetted migration path — uncheck that checkbox at the top to see those too).

  In my case:

  • I don't need the "Insert" module anymore, even though an upgrade path is provided.
  • I do not want the Google Analytics module anymore, because I'd rather not expose readers of my site to Google's tracking empire.
  • And … to complicate matters: I adopted Markdown over a decade ago but now want to switch to CKEditor 5, so I'll have to do some extra work to convert that. 😅 That is definitely not a 1:1 migration anymore, so it falls out of the scope of this tutorial.

• Contribute your findings!

  I figured out the root cause for a problem. This means I should do two things:

  • create an issue against the module providing the migration: #3390454 — I do not have time right now to fix it, but just sharing my findings is low-effort. It may help another person!
  • create an issue against the AM:A project on Drupal.org to suggest the solution! The AM:A module includes a file called messages-solutions.yaml, which makes it easy to add more suggested solutions. If I suggest a solution for this specific UUID, the before (top) vs after (bottom) becomes clearly visible:
    
    
    
    That change means that we go from an obscure message to something actionable!

  🙏 Even if you don't have time/energy to create issues with root cause analyses, please gather all your findings and submit an issue with a merge request. That alone can be enough to save fellow Drupal users many hours!

  I just did so here: #3390697.

Conclusion

I did not want to release AM:A into the world without a real-world example of it. It is not a silver bullet.

I hope I've shown you that with a minimum of terminal commands, you can restart the migration process very quickly. Without deep programming expertise but if you have tenacity and a willingness to dig deeper, you can figure out many problems.

When you figure out solutions for those problems, please report them as issues, and feel free to suggest more solutions for migration messages, I will make sure they are merged and released swiftly.

Finally, for sites with so much data that they need hours or even days to migrate: that's what the "Refresh" tab at the top right is for: to only migrate new and changed rows.

P.S.: I applied the conclusions from my review above (pushing me slightly past the hour). Only the 3 irrelevant "Site configuration" rows are left:

🚀