Migrating from Drupal 7 to Drupal 8

Authored on

02/11/2015

Changed

08/11/2015

Introduction and prerequisites

It was thanks to an article on the metaltoad blog that I initiated the module: http://www.metaltoad.com/blog/migrating-content-drupal-8

I also looked at the migration examples and real templates present in core.

The source code is available on my GitHub account: https://github.com/FlorentTorregrosa/migrate_ftorregrosa

As prerequisites, using Migrate plus and Migrate upgrade helps with development. The user interface was not complete at the time of development, but it is useful for getting an overview of migration statuses.

The migration was performed from a database as the source. In the settings.php file, the following connection information needs to be added:

$databases['migrate']['default'] = array(
  'database' => 'database',
  'username' => 'username',
  'password' => 'password',
  'prefix' => '',
  'host' => 'localhost',
  'port' => '3306',
  'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
  'driver' => 'mysql',
);

The important part is the "migrate" key.

Structure

With Migrate Drupal 7, to perform a migration you write a migration class that links the source to the destination (even if source and destination classes already exist) and also contains pre-processing, processing, and post-processing functions.

With Migrate Drupal 8, the migration information; migration name, source, destination, field mappings, etc. is contained in a configuration entity created via a YAML configuration file placed in the config/install directory.

In order to perform additional processing beyond what Migrate provides and to retrieve data, it is necessary (at least in my case) to write migration source classes.

To run your migrations, there are several methods. The one initially used was the migrate manifest method. This is a YAML file called manifest.yml placed in the module. It lists the migrations to execute in file order. To run the migration, the following drush command is used:

drush migrate-manifest /home/florent/sites/drupal8/modules/migrate_ftorregrosa/manifest.yml --legacy-db-url=""

Issues encountered

First, the issues that slowed down development:

The fact that migrations are configuration entities with no graphical interface for editing them means that if something needs to change in the YAML files, the module must then be uninstalled and reinstalled.
Earlier in development, uninstalling did not remove the configuration entities, the migrate tables in the database, or the content, requiring manual deletions.
Initially I followed the migrate manifest method, except that it does not allow re-running migrations in "update" mode to refresh already-imported content, hence the creation of a small script.
For the migration of "Drupal books" (the book content type), there was a problem in the book structure table when creating content that served as a book root. There was no way to get the correct nid inserted into it, despite having configured Migrate to look up the nid created by the migration itself. I had also started using migrate events, but there was no way to update the data. When re-running the migration in import mode, Migrate would this time insert the correct nid, but MySQL errors occurred due to table constraints, instead of updating existing entries, it was attempting to insert new ones. To work around this, I ran the book migration once, did a TRUNCATE on the table (yes, it's brute force but it works), then re-ran the migration in update mode, and the book structure was correct.
My site is (at the time of writing this article at least) in French only) French must be enabled on the Drupal 8 site. It sounds obvious, but because of this, article bodies were migrated but not displayed.

Detail of a migrate configuration file

Example of migrate.migration.ftorregrosa_website.yml (inline comments):

id: ftorregrosa_website
label: ftorregrosa website nodes
dependencies:                      # Module dependency for this migration.
  enforced:
    module:
      - language
      - migrate_ftorregrosa
migration_group: ftorregrosa      # Feature added by migrate_plus, allows grouping migrations in the back office.
source:
  plugin: ftorregrosa_website     # The migration class (plugin) serving as the source.
destination:
  plugin: entity:node             # The migration class (plugin) serving as the destination. Provided by core.
  type: website
  bundle: website
migration_dependencies:           # Migrations that must be run beforehand.
  required:
    - ftorregrosa_taxonomy_term
    - ftorregrosa_user
process:                         # The field mapping.
#  nid: nid
  vid: vid
  type: type
  langcode:
    plugin: static_map
    bypass: true
    source: language
    map:
      und: en
  title: title
  uid:
    -
      plugin: migration         # Instructs Migrate to look up the uid created from the source uid in the ftorregrosa_user migration.
      migration: ftorregrosa_user
      source: uid
  status: status
  created: created
  changed: changed
  promote: promote
  sticky: sticky
  'body/value': body_value
  'body/summary': body_summary
  'body/format':
    plugin: static_map
    bypass: true
    source: body_format
    map:
      1: plain_text
      2: restricted_html
      3: full_html
      4: full_html
  field_website_type:
    -
      plugin: migration
      migration: ftorregrosa_taxonomy_term
      source: field_website_type
  field_website_technology:
    -
      plugin: migration
      migration: ftorregrosa_taxonomy_term
      source: field_website_technology
  field_website_image: field_website_image
  field_website_dev_date_start: field_website_dev_date_start
  field_website_dev_date_end: field_website_dev_date_end
  field_website_link: field_website_link

Detail of a migration source class

Example of Website.php (full file on GitHub):

Annotation to declare a Migrate plugin:

/**
 * ftorregrosa website node source plugin.
 *
 * @MigrateSource(
 * id = "ftorregrosa_website"
 * )
 */

Example of body retrieval:

$nid = $row->getSourceProperty('nid');
// Body (compound field with value, summary, and format).
$result = $this->getDatabase()->query('SELECT fdb.body_value, fdb.body_summary, fdb.body_format FROM {field_data_body} fdb WHERE fdb.entity_id = :nid ', array(':nid' => $nid));
foreach ($result as $record) {
  $row->setSourceProperty('body_value', $record->body_value);
  $row->setSourceProperty('body_summary', $record->body_summary);
  $row->setSourceProperty('body_format', $record->body_format);
}

Example of preparing the data array for link fields:

// field_website_link.
$result = $this->getDatabase()->query('SELECT fdfwl.field_website_link_url, fdfwl.field_website_link_title FROM {field_data_field_website_link} fdfwl WHERE fdfwl.entity_id = :nid ORDER BY fdfwl.delta ASC ', array(':nid' => $nid));
// Create an associative array for each row in the result. The keys // here match the last part of the column name in the field table.
$links = [];
foreach ($result as $record) {
  $links[] = [
    'uri' => $record->field_website_link_url,
    'title' => $record->field_website_link_title,
  ];
}
$row->setSourceProperty('field_website_link', $links);

Date processing: converting from storage format yyyy-MM-dd HH:mm:ss (where HH:mm:ss equals 00:00:00) to yyyy-MM-dd:

// field_website_dev_date.
$result = $this->getDatabase()->query('SELECT fdfwdd.field_website_dev_date_value, fdfwdd.field_website_dev_date_value2 FROM {field_data_field_website_dev_date} fdfwdd WHERE fdfwdd.entity_id = :nid ', array(':nid' => $nid));
// Create an associative array for each row in the result. The keys
// here match the last part of the column name in the field table.
// Source: yyyy-MM-dd HH:mm:ss (where HH:mm:ss equals 00:00:00)
// Target: yyyy-MM-dd
foreach ($result as $record) {
  $row->setSourceProperty('field_website_dev_date_start', substr($record->field_website_dev_date_value, 0, 10));
  $row->setSourceProperty('field_website_dev_date_end', substr($record->field_website_dev_date_value2, 0, 10));
}

For images, the title, alt, height, and width attributes need to be retrieved and prepared in the same way as for links. To retrieve the migrated fid, $this->setUpDatabase connects to the Drupal 8 site database, whereas $this->getDatabase connects to the source migration database:

// Images.
// Site using file_entity:
// alt text => field_data_field_file_image_alt_text
// title text => field_data_field_file_image_title_text
$result = $this->getDatabase()->query('SELECT ...
// Create an associative array for each row in the result. The keys
// here match the last part of the column name in the field table.
$images = [];
foreach ($result as $record) {
  // Retrieve the migrated fid from ftorregrosa_file migration.
  $migrated_fid = $this->setUpDatabase(array('key' =>'default', 'target' => 'default'))
    ->select('migrate_map_ftorregrosa_file')
    ->fields('migrate_map_ftorregrosa_file', array('destid1'))
    ->condition('sourceid1', $record->field_website_image_fid)
    ->execute()
    ->fetchField();
  // Skip file if not migrated yet.
  if (!is_null($migrated_fid)) {
    $images[] = [
      'target_id' => $migrated_fid,
      'alt' => $record->field_file_image_alt_text_value,
      'title' => $record->field_file_image_title_text_value,
      'width' => $record->field_website_image_width,
      'height' => $record->field_website_image_height,
    ];
  }
}
$row->setSourceProperty('field_website_image', $images);

Notes

For contributed files and for simplicity, it is enough to copy/paste the contributed files into the files directory of your Drupal 8 site, the migration will then populate the contributed file tables. At least there is no need to make requests to the source site to retrieve the files.

I did not process rich text content to handle internal links. I will handle those manually during the actual migration. However, such processing can be done in the prepareRow function.

Conclusion

This module allowed me to get familiar with Migrate D8 and to see in a real-world context what it looks like with actual content. I can always refine it further :).

For example, handling the case of an account that already exists on the destination site (e.g. the admin account), the account created by the migration is disabled, but logging in with the original account is not possible.

EDIT: For the case of a username that already exists on the destination site, this is now "handled". Migrate will raise an error and will not migrate the source user. I believe this is nonetheless configurable to specify how Migrate should handle already-existing entities.

Tags

Archive