Texas Camp: Migration - How to Tame the Data

Migration has become the standard method to upgrade Drupal sites between major versions and external sources. In this session you will learn how to set up a migration and how to overcome common gotchas. Finally you will learn the current state of migration in Drupal 8 and the differences from Drupal 7. You will leave this session with an improved understanding of how to approach migrations and how to plan for the road ahead.

Methods

  • Drupal API
  • Feeds
  • Migration module (d7)
  • Migrate API (d8)

Method: Drupal API

  • Write custom code to parse files or connect to database.
  • Use drupal api like, node_save(), user_save(), taxonomy_term_save(), taxonomy_vocabulary_save(), entity_create with entity_metadata_wrapper.
  • Advantage: You are in total control. Not waiting on contrib modules.
  • Disadvantage: Time consuming
    • Write your own database query or file parser.
    • Write your own loops.
    • Detailed preparation of each object and its settings and fields.
    • Lots of opportunity to make typos and mistakes.
    • No easy way to roll back.
    • Manage large imports by incorporating Batch API.

Method: Drupal 7 API - create node

Drupal Migration api node create

Method: Drupal 8 API - create node

Drupal Migration api node create

Method: Drupal 7 API - create terms

Drupal Migration api term create

Method: Drupal 8 API - create terms

Drupal Migration api term create

Method: Drupal 7 API - create user

Drupal Migration api user create

Method: Drupal 8 API - create user

Drupal Migration api user create

Method: Feeds

  • UI for importing content without writing code.
  • Several parsers to extend feeds.
    • Default: RSS, CSV, OPML, Sitemap
    • XPath XML
    • QueryPath XML
    • JSONPath
    • JMESPath (JSON)
    • SQL
  • Feeds Tamper: UI plugin for altering data.
  • Does provide hooks for data alteration.

Method: Feeds hook_feeds_after_parse()

Alter data before it is processed.

Example of altering a country codes.

Drupal Migration hook feeds after parse

Altar just before pre-save.

Drupal Migration hook feeds presave

Method Feeds: Demo

Demo Current D8 Feeds Interface

Method: Migration Module (d7)

  • Define migration in Object oriented code.
  • Extend migration classes in custom module (developers tool)
  • Source support for PDO, XML, CSV, JSON, MSSQL, ORACLE
  • Destination support for node, user, taxonomy term, comment, file.
  • Map sources to destination.
  • Manage dependencies.
  • UI interface to track mapping of fields and perform migration.
  • Roll back support for easy development and testing.
  • Memory management for large running migration.
  • Alter data with prepare before it is saved and complete after saved.
  • Extend with extra modules: migrate_d2d, wordpress_migrate

Method: Migration Module (d7)

  • Register your custom class
  • Define your class that extends a migration class.
  • Define key mapping
  • Define columns for mapping.
  • Define source
  • Map fields
  • Unmap source fields and destination fields.
  • Perform operations if needed on prepare and complete.

Method: Migrate extending Migration

Drupal Migration extend Migration class

Method: Migrate API (d8)

Migration API added to d8 core with several contrib modules

  • Migrate Core: Provides Core API
  • Migrate Drupal: Provides classes to migrate configuration and content d6->d8 & d7->d8.
  • Drupal Upgrade: Contrib being added to core in 8.1.x
  • Migrate Tools: provides optional Drush commands
  • Migrate Plus: Extends API with grouping, source manipulation and examples (Beer).
  • Migrate Manifest: Provides drush command migration from manifest file.

Method: D6/D7 to D8 Issues

  • D6 has Story type vs Article type
  • Views migration: Views 6.x-2.x needs to upgrade to 6.x-3.x
  • Date Formats: Only the default short, medium and long formats are migrated.
  • Text/Input Formats: if input format is not recognized it will be imported as filter_null
  • Timezones: D6 uses timezone offset. D7 & D8 use a timezone name. Could cause problems depending on server setup.
  • Node, User, Entity Reference fields not supported.
  • Multilingual Content
  • D6 to D8 supports:Core, CCK, Link, Email, Phone, Imagecache modules
  • D7 to D8 supports:Only content, users, taxonomy, blocks, menus, filters

Method: Migrate Extended

  • To customize a d6 migration you will need to create your own module.
  • Include the proper Namespace and classes.
  • Create a migration template
  • Drupal Migration template

    • Reference: core/modules/node/migration_templates
  • Create your own template with your own ID and source id.

Method: Migrate Extended

  • Create your PHP file to extend D6node.
  • Include the proper namespaces and classes
  • Update the MigrateSource id to match your template migrate source id.
  • Use function fields() to add fields.
  • Use function query() to access the database.
  • Use function prepareRow(Row $row) to prepare data.

Method: D6/D7 to D8

UI

  • Enable appropriate module and navigate to /upgrade in D8
  • Rerun
  • Rollback

Drush

  • drush migrate-upgrade --legacy-db-url=mysql://user:@127.0.0.1/drupal6
  • With migrate_tools you can use additional drush commands
    • Drush help to view options

Method Migrate - D6 to D7 Demo

DEMO d6->d8 Migration