Cloud Platform
Decoupling Drupal 8: Serialization Module
If you have decided to decouple Drupal, after conducting due diligence with regard to assessing the risks and rewards of decoupling Drupal and understanding how best to decouple Drupal in 2018, it is essential to understand how web services in Drupal 8 core and their dependencies undergird all decoupled Drupal architectures, regardless of the application you are implementing.
In this Experience Express column, we'll explore the process of decoupling Drupal through the lens of core modules in Drupal 8.
In Drupal 8 core, four key modules, when enabled, collectively facilitate the provisioning of RESTful APIs out of the box: HAL, Serialization, REST, and Basic Authentication.
Some of these modules provide foundations that underpin important contributed modules also useful for Drupal such as JSON API, RELAXed Web Services, and GraphQL. For instance, while the JSON API contributed module depends on Serialization, it does not depend on REST.
After examining these modules, in a forthcoming post, we'll configure Drupal 8 to be an effective web service provider and back end before turning to the contributed ecosystem of web services modules. Later, we'll also take a look at authentication methods for RESTful APIs in Drupal 8 like Basic Authentication, OAuth2, and JSON Web Tokens as well as other useful contributed modules.
Though web services are available off the shelf with Drupal 8, there are contributed solutions available for Drupal 7, for those who wish to remain on that version. We'll handle those in a future column. Because Drupal 8 core is the most common point of entry for those experimenting with Drupal as a decoupled back end, we'll start there.
For Drupal to truly embrace the future web, we need to fundamentally rethink how Drupal [responds] to an incoming HTTP request. We need to treat HTML pages as what they are: A particularly common form of REST response, but only one among many. To that end, Drupal needs to evolve, and quickly, from a first-class web CMS into a first-class REST server that includes a first-class web CMS. Larry Garfield, WSCCI lead 1
Thanks to the Web Services and Context Core Initiative (WSCCI), Drupal 8 is now a capable off-the-shelf REST server with the ability to manipulate Drupal content entities including nodes, users, taxonomy terms, and comments through typical CRUD (create, read, update, delete) operations via HTTP requests.
Drupal was originally a product of an era in which static pages dictated application state and content management systems were by and large monolithic. However, in recent years, the advent of dynamic web applications and other channels for content has highlighted the growing demands on Drupal with regard to serving requests originating from diverse sources.
Such requests can include partial page requests (for example for edge side includes), RESTful API calls, and even queries originating from command line tools like Drush and Drupal Console.2 Nonetheless, these sorts of requests were ill-suited for Drupal at the time, which focused on composing and rendering full HTML pages rather than data in other formats. Even Drupal 7 core only benefited from an XML-RPC layer rather than a true web services layer.3
In 2011, during the Drupal 7 release cycle, the Web Services and Context Core Initiative (WSCCI) was launched before becoming an official initiative for Drupal 8. Because of the need to sidestep the HTML rendering process in Drupal, WSCCI's goal was to modernize how Drupal handles requests and serves pages to anticipate the many use cases that would emerge later in the Drupal 8 development cycle.
In order to handle the vast variety of responses that Drupal would eventually need to serve, the Symfony HTTPFoundation component, an HTTP request processing library for PHP, was incorporated into Drupal core, in conjunction with other elements of Symfony.
Due to the monumental scope of the initiative, in 2012, the initiative was scaled down to concern itself with web services only.4,5
Alongside the introduction of the Serialization module, WSCCI made it possible for Drupal to serialize, or transform data structures into a storable format (like a file or a transmission across a network), and deserialize, or construct data structures from the Drupal data.
At the time, only content entities were available for serialization into XML, JSON, and the HAL+JSON format based on the HAL normalization, which relies on the Hypertext Application Language (HAL) specification (the Drupal community chose to replace JSON-LD with HAL in 2013).
The Serialization module allows modules depending on it (such as the REST module) to use its serializers to convert Drupal data into consumable formats. Built on top of the Symfony Serializer component, Serialization offers an API for developers to provide additional serialization formats through the installation of other modules. A few prominent examples of modules that make use of the Serialization API include HAL in Drupal 8 core, which includes support for the HAL+JSON format; the CSV Serialization contributed module, which supports data in CSV format; and the Acquia Content Hub contributed module, which integrates with Acquia's Lift with Content Syndication product.
The Serialization module also handles normalization when different standards enter the picture. The HAL module's HAL+JSON format employs different data structures and exposes a different set of information from the default JSON encoding, such as links to other resource URIs that satisfy the HAL specification. As such, the Serialization module includes a default serializer and a default normalizer. Other modules can provide their own encoders or normalizers. Meanwhile, encoders for JSON and XML are built into the Symfony Serializer component.
The diagram below, adapted from a diagram in the Symfony Serializer documentation, illustrates how serializers provide consumable data in JSON and XML.
As you can see, while most developers will opt to use the serializer as-is, each format is actually comprised of both normalization (plus denormalization) and encoding (plus decoding). This allows for a separation of concerns between the normalization of an object into a nested array and the encoding of that nested array into the desired format.
The diagram below demonstrates the flow of data and how normalization must occur before encoding to ensure that certain elements of specifications like HAL are part of the encoded format. In this example, a node is normalized into a HAL-compatible structure before being encoded into HAL+JSON.
When the Serializer::serialize() method is called, the Serializer will iterate through the available Normalizer services to determine which it should use, calling Normalizer::supportsNormalization($object, $format) on each Normalizer (from highest priority to lowest) until it encounters a Normalizer that returns TRUE.
The Serializer undertakes precisely the same process to select the correct Encoder, iterating through the Encoders with invocations of EncoderInterface::supportsEncoding($format) until it lands on the Encoder that supports the provided format.
To add a new encoding entirely, outside of the core-supported JSON, XML, and HAL+JSON, you can add an Encoder, assuming that the data structure provided by core's default Normalizers works for your encoding as well.
First, create an encoder that implements EncoderInterface and define the obligatory encode() and decode() methods. Then, you can register the encoder using a *.services.yml file in your module. This example is taken from the HAL module's hal.services.yml file:
services:
# ...
serializer.encoder.hal:
class: Drupal\hal\Encoder\JsonEncoder
tags:
- { name: encoder, priority: 10, format: hal_json }
# ...There are several key APIs that can aid you in constructing new serializers and normalizers and in handling entities with references to other entities using entity resolvers.
You can use the serialize() and deserialize() methods in Drupal 8's serializer service (\Symfony\Component\Serializer\SerializerInterface) to either serialize an entity into JSON or XML output or to deserialize incoming JSON or XML into a Drupal entity:
$output = $this->serializer->serialize($entity, 'json');
$entity = $this->serializer->deserialize($output, \Drupal\node\Entity\Node::class, 'json');In the first line above, an entity is serialized into JSON output. In the second, a JSON data structure is deserialized into an entity that can then be manipulated normally by Drupal code.
Each serializer implementation makes use of an encoder (\Symfony\Component\Serializer\Encoder\EncoderInterface) and decoder (\Symfony\Component\Serializer\Decoder\DecoderInterface) which can be used to add support for encodings to and decodings from new serialization formats (such as CSV or something else).
As an educational example, the encode() implementation in the CSV Serialization module identifies the type of input data and encodes that data into CSV that can then be consumed outside of Drupal's own understanding of entities.
In order to reconcile the differences between a particular encoding and a specific normalization, such as the distinctions between the raw JSON data structures and the HAL+JSON normalization, the Symfony Serializer component introduces normalizer (\Symfony\Component\Serializer\Normalizer\NormalizerInterface) and denormalizer (\Symfony\Component\Serializer\Normalizer\DenormalizerInterface) interfaces.
In Drupal, the default normalization is close to an identical replication of the object data as possible and merely apply the JSON and XML encoders to the default normalization ('json' and 'xml' formats). Other normalization formats written by developers may wish to apply specific constraints to incoming object data, such as the omission of local IDs in favor of UUIDs or the addition of new metadata which satisfies specifications like JSON-LD or HAL.
The normalize() implementation in the HAL module, which will be discussed further shortly, demonstrates how adherence to specifications must be captured by normalizers and not encoders, whose responsibilities do not include the inclusion of metadata such as that required by the HAL specification. In this example, the normalization is prepended with a _links key that begins every HAL response.
Whereas content entities are the most common data structures to be serialized, you may also find that a particular entity references other entities. When that is the case, as is common in complex Drupal content models, you can resolve those referenced entities by invoking resolve() with their UUIDs (\Drupal\serialization\EntityResolver\UuidResolver) or local Drupal identifiers (\Drupal\serialization\EntityResolver\TargetIdResolver).
As you can see, the history of web services in core is complex and indicates an early emphasis on their availability in the Drupal 8 development cycle. Thanks to the foundation established by the Web Services and Context Core Initiative, web services are now a key component of Drupal 8 core. With the Serialization module, you can access a rich array of features that allow you to handle new encodings and normalizations and make use of the Serialization API to add new serializers.
In the next installment of Experience Express, we're stopping in Austin, where SXSW Interactive was in full swing last week. I teamed up with Muneeb Ali of Blockstack, Sara M. Watson of the Berkman Klein Center, and Andrei Sambra of Qwant to talk about starting the internet all over again by decentralizing the web. Check back here for a SXSW Interactive roundup with insight into how innovators both on and off the web are finding novel ways to engage with emerging technologies like blockchain and virtual reality.
As always, you can ping mailto: [email protected] or @prestonso on Twitter with ideas for what you'd like to see in this column. All aboard!
Special thanks to Wim Leers for his feedback during the writing process and to Lin Clark, who is the original author of much of the documentation used for this column.
Works cited:
Cloud Platform