--- title: "AMA Module: Heuristics & Hypermedia" date: "2023-12-12T14:32:30+00:00" summary: image: type: "article" url: "/acquia-cloud-platform/help/89336-ama-module-heuristics-hypermedia" id: "ab8ed52d-d203-4358-a0fe-7f5d28ec584d" --- Acquia Migrate: Accelerate has been open source for 2 months (see the [announcement](https://dev.acquia.com/blog/acquia-migrate-accelerate-now-open-source), [drupal.org project](https://www.drupal.org/project/acquia_migrate) and [tutorial](https://dev.acquia.com/tutorial/drupal-7-drupal-9-hour-using-ama)). Time for a peek behind the curtain now that the entire source code is visible how does the module _actually_ work? 🕵️ Heuristics ---------- A hard requirement from the beginning was to empower non-technical users to be able to perform migrations. The UI had to prioritize the _site builder_ persona as much as possible: * Good: content types, vocabularies and path aliases ✅ * Bad: migration plugins, source/progress/destination plugins, YAML without validation ❌ (Or as I like to put it: "if it could contain an underscore, it should require clicking on "Details".) How can this possibly be achieved? The Drupal [Migrate API](https://www.drupal.org/docs/drupal-apis/migrate-api/migrate-api-overview) has no infrastructure for this. In fact, migration source plugins and migration definitions are arguably not even developer-centric, but _database table_\-centric (which many developers don't need to know the details of): they're designed to get all rows from one Drupal 7 database table into the equivalent tables in Drupal 8/9/10. This is why for example the `d7_url_alias` migration is only able to migrate _all_ path aliases at once not only those for a certain content type, for example (AM:A's core patch to change that: #3122649). This makes sense for developers who know exactly what data is stored where (which indeed is necessary knowledge for complex migrations), but is not something every developer necessarily needs to know. It's _definitely_ something the site builder should not need to know. On top of that, the user of the Migration API is expected to know in which _order_ to execute migrations. One often hears about "migration scripts", which use Drush to execute one specific migration plugin after another. Again, this is fine to expect for complex migrations. But it should not be necessary in most cases, and it's completely unreasonable to ask a site builder. One that surprised us while working on AM:A was that migrations of entities with file/image fields do not depend on the migration of files, resulting in stubs getting created for _everything_ (AM:A's core patch to change that: #3123775). By applying patches to Drupal core that increase the granularity of migration plugins, as well as adding dependencies to some migrations that are missing it (and updating the ones that do exist to take into account the increased granularity), AM:A is able to achieve this: ![Migration dashboard showing document media items import progress at 84.95%, with 254 of 299 items imported.](https://acquia.widen.net/content/4thfcg1gri/web/url_a1e6013c954b32c2bb846d183bfe443d.png?v=31ef66f7-e01e-40e4-aee6-56e4719a0a06) What do you see here? Well: 1. We're looking at the "Document media items" migration. 2. AM:A is _redefining_ what "a migration" actually is: AM:A makes it into a higher level concept, one that we believe better matches the mental model of the site builder. (And honestly: we think it better matches the mental model of anybody working on a Drupal migration.) 3. The "underlying migrations" (_migration plugins_, to use the [correct terminology](https://www.drupal.org/docs/drupal-apis/migrate-api/migrate-api-overview#s-migrate-api-plugins)) are `d7_file_plain_source_field:document`, `d7_file_plain_type: document`, and so on. Note how the last 3 list a "(X of Y)" suffix. These are the "data migration plugins", all preceding ones are "supporting configuration migration plugins". The latter are responsible for configuring Drupal 9 in a way that Drupal can actually _store_ the data to be migrated. 4. This migration depends on the "Public files" migration because one of the underlying migrations depends on `d7_file`. As an end user, if I don't click the "Details" tab, I never see any of this. I would only ever see the dozens of "AM:A migrations", not the hundreds of migration plugins. How does AM:A figure out _what_ the underlying migrations should be if the Migrate API does not have infrastructure for this? Does it use core patches? No. Does it use magic? No. It uses a bunch of [heuristics](https://en.wikipedia.org/wiki/Heuristic) to achieve this it's definitely imperfect but works >95% of the time without the need for tweaks to its heuristics. (For the curious: there is a _very_ detailed comment at the top of [`MigrationClusterer.php`](https://git.drupalcode.org/project/acquia_migrate/-/blob/1.8.0/src/Clusterer/MigrationClusterer.php#L23-L89), and [`MigrationClusterer::getHeuristics()`](https://git.drupalcode.org/project/acquia_migrate/-/blob/1.8.0/src/Clusterer/MigrationClusterer.php#L211-L277) lists all heuristics, what they impact and their weight.) This hard requirement of a UX that is approachable for everyone is a key reason why AM:A has an intimidating [40 (!) core patches configured to be applied](https://git.drupalcode.org/project/acquia_migrate/-/blob/2023-09-27/curated.json?ref_type=tags#L1832-L1871): many of these patches add derivers to core's migration plugins (to increase granularity), , to allow more piecemeal migrations. Hypermedia & React ------------------ The AM:A UI is fully decoupled, and: * is written in React by [Peter "zrpnr" Weber](https://www.drupal.org/u/zrpnr) * is tightly integrated with Drupal's routing and theme system (did you notice above that the Drupal toolbar is still present and working as it otherwise would?) thanks to the [Decoupled Pages module](https://www.drupal.org/project/decoupled_pages) module written by [Gabe Sullice](https://www.drupal.org/u/gabesullice) * is fetching data and triggering migration operations through a JSON:API implementation provided by the `acquia_migrate` module also written by [Gabe Sullice](https://www.drupal.org/u/gabesullice) All of those aspects can be visualized in a single GIF: ![Dashboard showing migration status with progress bar, import details, and lists of items ready for import, unmet dependencies, and operations.](https://acquia.widen.net/content/9ssjteexcx/web/url_73a3b4f627fed99859a8c29ba5040a51.gif?animate=true&v=cf6f2e4e-9af1-49fe-8d9f-7d7bdeab7417) This module was _not_ written on top of [JSON:API Hypermedia](https://www.drupal.org/project/jsonapi_hypermedia) because it does not use Drupal entities (we _could_ in theory have modeled the "AM:A migrations" as some kind of computed entity, but that would have introduced more layers of complexity for little gain). The React UI [receives](https://git.drupalcode.org/project/acquia_migrate/-/blob/1.8.0/ui/src/dashboard-app.jsx) application bootstrapping data from the [route definition](https://git.drupalcode.org/project/acquia_migrate/-/blob/1.8.0/acquia_migrate.routing.yml#L225-L234), with the glue provided by the [Decoupled Pages module](https://www.drupal.org/project/decoupled_pages) module. The server side defines a number of [link relation types](https://git.drupalcode.org/project/acquia_migrate/-/blob/1.8.0/src/UriDefinitions.php). These describe the semantics of links ("to what concept does this link? what kind of operation does this trigger?"), enabling the AM:A React UI (or any competing UI!) to implement logic for each link relation type _once_ and then have it work "automatically" for any new links that trigger different operations but with the same semantics! For example, the `import`, `rollback`, `import-and-rollback` and `refresh` operations _all_ use the same link relation type: `https://drupal.org/project/acquia_migrate#link-rel-start-batch-process`. The React application only had to implement logic for how to [discover](https://git.drupalcode.org/project/acquia_migrate/-/blob/1.8.0/ui/src/lib/utils.js#L241-253), [present](https://git.drupalcode.org/project/acquia_migrate/-/blob/1.8.0/ui/src/lib/utils.js#L651-L684) and [request](https://git.drupalcode.org/project/acquia_migrate/-/blob/1.8.0/ui/src/lib/api.js#L54-L107) these links (including [tailored retry semantics](https://git.drupalcode.org/project/acquia_migrate/-/blob/1.8.0/ui/src/lib/api.js#L6-L13)) _once_. For each of those 4 operations. If we add a new "retry any rows with messages" operation, or an "erase messages" operation, then: * the server side would have to implement a route + server-side logic to generate a response * the server side would then have to a new link with an appropriate `title` to present to the end user, but use the same `rel` * the client side would need to make _zero_ changes * … and these new operations would show up in the AM:A UI! This is why initially, a lot of time was spent on the UI: both the client and server needed to grow the necessary basic infrastructure. But _then_ … we increasingly discovered that the building blocks we needed were the same, and we didn't need to build anything new except for refining the semantics of the link relation types we had defined (we had little to no experience actually implementing [Hypermedia](https://en.wikipedia.org/wiki/Hypermedia)). It ended up working _very_ well 🤩 Both for us (very low maintenance) and for the end user (great UX). If you want more details still, see Gabe's previous blog post about this at [https://dev.acquia.com/blog/tightly-integrated-loosely-coupled](https://dev.acquia.com/blog/tightly-integrated-loosely-coupled) and the post on his personal blog at [https://www.sullice.com/posts/2019/09/20/a-new-era-for-drupals-jsonapi/](https://www.sullice.com/posts/2019/09/20/a-new-era-for-drupals-jsonapi/).