And Migrate to the Future

The major theme of the next release of Migrate on Drupal 7, version 2.6, will be developing an infrastructure to make it easy for migration modules such as wordpress_migrate and migrate_d2d to build user-friendly interfaces for importing content, based on step-by-step wizards. This work includes not just adding a wizard API, but also presenting a simple interface for non-technical users to run and review imports while still giving migration developers all the information they need. Initial work has begun and is available in the current -dev releases of Migrate and WordPress Migrate (with some more rudimentary work in the wizard_api branch of migrate_d2d). The API is still in flux, so if you're interested in experimenting with this it will be important to update the modules in sync for the time being.

Wizard API

The new Wizard API is still evolving, so the details are likely to change, but the basic idea is to implement an extension of the MigrateUIWizard class which defines the forms and validation functions for each stage of your wizard, and register the name of your class via hook_migrate_api:

<?php 
function wordpress_migrate_migrate_api() {
 $api = array(
 'api' => 2,
 'wizard classes' => array('WordPressMigrateWizard'),
 );
}

class WordPressMigrateWizard extends MigrateUIWizard {
...
?> 

The base MigrateUIWizard class will manage the Next/Previous/Finish buttons as users go back-and-forth through your wizard, maintaining the raw form state at each step (so it can be restored on Previous). Your own class will maintain the high-level state as you move forward, so it is available to construct the concrete migrations in the final stage.

So, let's look at the anatomy of a wizard class. The one required method (other than the constructor) is getSourceName, returning the name of the source system/format to migrate from:

<?php 
public function getSourceName() {
 return t('WordPress');
}
?> 

This is used in the tab names and wizard titles in the Migrate UI:

Now, in your constructor, your primary job is to define the required steps in your wizard:

<?php 
public function __construct() {
 parent::__construct();
 $this->addStep(t('Overview'), 'overviewForm');
 $this->addStep(t('Upload blog'), 'sourceDataForm');
 $this->addStep(t('Select content to import'), 'contentSelectForm');
 $this->addStep(t('Review'), 'reviewForm');
}
?> 

The first argument to addStep() goes into the page title - in the screenshot above, you see how getSourceName() and the first addStep() produce the title "Import from WordPress, step 1: Overview".

The second argument to addStep() is the name of a function defined in your class, which will take the traditional form state as an argument and return a form in the usual manner:

<?php 
protected function overviewForm(&$form_state) {
 $form['overview'] = array(
 '#prefix' => '<p>',
 '#markup' => t('This wizard supports importing into your Drupal site from ' .
 'a WordPress blog. To be able to use this wizard, you must have the ' .
 'address and credentials of the WordPress blog, or an XML file exported ' .
 'from the blog.'),
 '#suffix' => '</p>',
 );
...
return $form;
}
?>

You are also required to define a validation function, with 'Validate' appended to the name of the form function, which will validate the form state and also save anything which will be needed in subsequent steps.

<?php 
protected function overviewFormValidate(&$form_state) {
 $this->groupName = trim($form_state['values']['group_name']);
 if (!preg_match('|^[[:alnum:]]+$|', $this->groupName)) {
 form_set_error('group_name',
 t('The name for your import, %name, must be alphanumeric.',
 array('%name' => $this->groupName)));
 }
}
?> 

Now, an important feature of the step management is that you can add steps dynamically. If you refer back the constructor, we added a contentSelectForm step, followed by the final reviewForm step. The content selection form looks like this:

Each of the content types needs to prompt for a bunch of options, but so far we haven't defined steps for this, because the user can choose to not import one of the content types. We will add those option steps as needed on validation:

<?php 
protected function contentSelectFormValidate(&$form_state) {
 $this->contentValues['page']['page_type'] = $form_state['values']['page_type'];
 if ($this->contentValues['page']['page_type']) {
 $this->addStep(t('Pages'), 'contentPageForm', $this->currentStep);
 }
 $this->contentValues['blog']['post_type'] = $form_state['values']['blog_post_type'];
 if ($this->contentValues['blog']['post_type']) {
 $this->addStep(t('Blog posts'), 'contentBlogPostForm', $this->currentStep);
 }
}
?> 

Notice that here we've got a third argument to addStep, $this->currentStep. Steps in the migration are implemented as instances of the MigrateUIStep class - the third argument to addStep is the step after which to add a new step. The default is to add the new step at the end of the list.

The last step of the wizard reviews what is to be imported and how:

While building the page, it is also informing the base MigrateUIWizard classes of what will be imported:

<?php 
protected function reviewForm(&$form_state) {
 $message = t('<p>Please review your migration configuration. When you submit this
 form, migration processes will be created and you will be left at the
 migration dashboard.</p>');

 $form['description'] = array(
 '#prefix' => '<div>',
 '#markup' => $message,
 '#suffix' => '</div>',
 );
 $migrations = '';
 $account = user_load($this->defaultAuthorUid);
 if ($this->authorMigration) {
 $migrations .= t('<li>WordPress authors will be imported.</li>');
 }
 else {
 $migrations .= t('<li>WordPress authors will not be imported. The default ' .
 'author for any content whose author can\'t be identified will be %default.</li>',
 array('%default' => $account->name));
 }
 $arguments = array(
 'default_author_uid' => $this->defaultAuthorUid,
 'import_users' => $this->authorMigration,
 );
 $this->addMigration('Author', 'WordPressAuthor', $arguments);
...
?>

addMigration takes the same arguments as MigrationBase::registerMigration - it simply saves them.The review form is the one exception to the rule of adding a Validate function for each form - the handler for the Finish button is MigrateUIWizard::formFinish(), which takes the information saved by addMigration and makes the calls to registerMigration, before redirecting you to the Migrate dashboard:

And I think looking at this makes it clear why we'll need additional UX work to make the overall experience work for someone who just wants to import their WordPress blog.

Further work

I feel good about where we are now on the Wizard API itself - the main thing I'm looking at now is if there are any additional services to provide in MigrateUIWizard that are commonly needed. One thing in particular is the Overview and Review steps - to help provide a consistent UI among different import modules, I think most of the work on these could be moved into the general class, and in most cases the import module should only need to implement the domain-specific steps in between. Another thought is to eliminate the Overview step, and automatically generate the group name.

UX improvements

The biggest challenge here is serving both non-technical people wanting to easily import, and heavy-duty migration developers needing deep information on the state of their migrations. As a starting point, to the existing 'migration information' permission I've added 'advanced migration information', and hidden some details from users without the advanced permission, but there is a long way to go here. Also, the long-requested organization of the UI by groups has been implemented.

One big issue is terminology - with more serious UX attention on Migrate, we want to present clear non-technical terms to end users. But, to preserve good DX, the terminology developers see ideally should reflect that used in the API. A primary example is the word "Migrate" itself - for the end-user scenarios we're looking at, "Import" is preferable, and even from a development scenario in the long run thinking of this API as providing import services would be preferable - but for now, with Migrate 2, the word Migrate is thoroughly embedded everywhere, and it would be confusing for developers to deal with things that are called "migrations" in the code and, say, "importers" on the front end. It may be, though, that we will ask the developers to bite the bullet on this one, because trying to maintain two front-end terminologies will be even uglier.

Groups

When the concept of migration groups was introduced, I made them a class, MigrateGroup. All we've done with groups up to now is give them names, so having the class has been overkill. I think now is the time to start leveraging groups as objects in themselves. First off, group names up to now have been treated both as machine names (unique identifiers) and display names (shown in drush output, and now in the UI). We should have distinct machine and display names - and the wizard API (which currently prompts for a group name) should set them automatically. For example, in the WordPress case, the group display name could be derived from the blog title and URL in the source file - e.g., "My WordPress Blog (myblog.wordpress.com)", while the machine name (not displayed in the UI, mainly of use for power users issuing drush commands) would be a scrubbed version of the blog title as it is now - mywordpressblog.

More DX than UX, I also want to start using the MigrateGroup class to store information common to the migrations in the group. Currently, wordpress_migrate stores the input filename in the arguments of all migrations, while migrate_d2d stores connection info. Ideally we should normalize this information by storing it in the group, which implies we introduce a group table. Since apart from the machine name (which would be the primary key) and display name, the group data would vary according to the implementing module, so would be stored as a serialized array. Perhaps we could also store state information (when the last import for the group was run, what was the total running time, etc.).

Which brings us back to the UX side. The non-technical user is going to be thinking of the migration process in terms of the group, not the individual migrations. By default, they should only see the group and its state. To what extent do we provide access to per-migration details?

Migration deregistration

Another request in the queue is a UI for deregistering migrations. This is becoming more necessary for developers with a greater emphasis on explicit registration of migrations, and the use of dynamic migrations for the scenarios we're working with here. For the non-technical user, WordPress Migrate has had this functionality presented as "Remove migration bookkeeping" (or in the latest release, "Delete tracking"). Here is one of those language discrepancies I mentioned, since "deregistration" is the precise and accurate description of what is happening technically, but it will not be a meaningful word to most users.

Field mappings

One of the biggest UX challenges will be dealing with field mappings, particularly in the Drupal-to-Drupal case, where there are likely to be rich and divergent content type and field architectures on both sides of the migration. If we are to allow users to configure how fields are mapped between source and destination, that's a lot of information to present, and a complex UI to implement. It may be that the first implementation in migrate_d2d will do its best to automatically map fields - map source fields to any destination fields with the same name, and for any others automatically create the fields on the destination to match how they were configured on the source, which would address at least the upgrade and direct migration scenarios. Ultimately, we would want to see a field mapping UI in Migrate itself on the detail pages where the mappings instantiated in code are currently presented.

Background import

One problem with presenting import operations through the UI is that core Drupal does not provide a good, reliable way to perform lengthy (perhaps hours-long) operations through the UI. For developers, running migrations through Drush is an option (and strongly encouraged), but that is not directly available to someone simply wanting to import their WordPress blog. wordpress_migrate attempts to address this by allowing configuration at the server end (through setting appropriate variables manually) to run imports by execing Drush in the background. We need to put this, or some equivalent, background import ability into Migrate itself. Whether we exec Drush, or leverage system queues in some way, this is essential for supporting large imports.

It's time (if not past time) to consider Migrate in Drupal 8. The idea of incorporating at least some of the Migrate API into core for Drupal 8 to support upgrades has been around for a while - it may be a little late in the game to get it in, but we should do what we can. My plan is to begin this work soon in a new 8.x-3.x branch of Migrate (or perhaps a sandbox project), and once the above work for Migrate 2.6 is done most of my contrib time will be devoted to this effort. Of course, the Wizard API work (specifically the migrate_d2d implementation) will serve as a proof-of-concept for the upgrade UI.

Some thoughts on what Migrate 3 for Drupal 8 might look like are on g.d.o.