REST API for file upload in Drupal 8 (multipart/form-data)

By Ronald van Belzen | November 30, 2018

This article is related to a former article in which I claimed that "there is no support to directly upload images using REST in Drupal 8". Since Drupal 8.6 this in no longer true (https://www.drupal.org/node/1927648). You can read how to use this new functionality at https://wimleers.com/blog/api-first-drupal-file-uploads and https://wimleers.com/blog/api-first-drupal-8.6.

This article is, however, about an entirely different method to upload a file in Drupal 8 for the somewhat unusual use case of passing the uploaded file to a third party REST api. Not that the method described here will be limited to that use case, but in most cases the new Drupal 8.6 functionality will be the preferred choice.

The situation was that for a decoupled set-up the front-end needed to upload a file to a third party REST api. For that upload the third party needed info about the user present in Drupal that the front-end was not allowed to retrieve or expose in any way. Also, as a temporary solution Drupal needed to perform a virusscan on the uploaded files. The front-end strongly preferred the use of a multipart/form-data content type for the POST request.

So, the uploaded file is not meant for storing in the Drupal application and not meant to be saved as File object. The method shown below does do that anyway, because the clamav module, used for the virus scan, requires it.

As usual I start with a new module:

# my_api.info.yml

name: My REST Service
type: module
description: "REST Service for passing file upload"
package: Web services
core: '8.x'

And since I need to create the end-point for this REST api myself, I create the route for this end-point:

# my_api.routing.yml

my_api.upload.file:
  path: '/service/upload'
  defaults:
    _controller: '\Drupal\my_api\Controller\UploadController::Upload'
    _title: 'File upload end-point'
  requirements:
    _access: 'TRUE'
    _method: 'POST'
    _csrf_request_header_token: 'TRUE'

The routing points to the Upload() function of the UploadController class. The requirements allow anyone to post requests to this end-point, but a X-CRSF-Token header with a valid token will be required. The UploadController will do the user validation, because that will allow us to give 403 responses in line with the way the third parties returns 4xx/5xx responses.

  /**
   * Handle the file upload.
   *
   * @return \Symfony\Component\HttpFoundation\JsonResponse
   */
  public function Upload() {
    try {
      $user = $this->ensureUserIsKnown();
      $this->agentID = $user->get('field_agentid')->value;
      $fileData = $this->checkForVirus();
      $response = $this->passUploadRequest($fileData);
    }
    catch (HttpException $e) {
      $response = [
        'timestamp' => date('Y-m-d\TH:i:sO'),
        'status' => $e->getStatusCode(),
        'error' => Response::$statusTexts[$e->getStatusCode()],
        'message' => $e->getMessage(),
        'path' => $this->path,
      ];
      return JsonResponse::create($response, $e->getStatusCode());
    }

    $data = NULL;
    $statusCode = 500;
    if ($response) {
      $statusCode = $response->getStatusCode();
      $data = $this->responseBody;
    }
    return JsonResponse::fromJsonString($data, $statusCode);
  }

The Upload() function starts with retrieving the user account. The ensureUserIsKnown() throws an HttpException when the user is not known or not authorized.

From the user account the value for the field_agentid is retrieved that will be used to pass along with the file to the third party REST api.

The checkForVirus() function also throws an HttpException when an virus is detected in the file, and when no virus is found returns the binary of the file. This binary is passed to the passUploadRequest() function that will pass the request to the third party and returns the response received from the third party REST api.

When an exception is caught the exception is used for returning the error to the requester as a JsonResponse. When the response from the third party REST api is received this response contains a json in the body that is returned to the requester.

I will include the full source of the UploadController at the end of this article, but let us first focus on the functions checkForVirus() and passUploadRequest() that do the heavy lifting.

  /**
   * Check for a virus in uploaded file.
   *
   * @return bool|string
   */
  protected function checkForVirus() {
    $files = $this->currentRequest->files;
    if ($files->count() == 0) {
      return FALSE;
    }
    $keys = $files->keys();
    /** @var \Symfony\Component\HttpFoundation\File\UploadedFile $uploadedFile */
    $uploadedFile = $this->currentRequest->files->get($keys[0]);


    $fileName = $uploadedFile->getClientOriginalName();
    if (empty($fileName)) {
      throw new HttpException(400, 'File name not found');
    }

    $path = $uploadedFile->getPathName();
    $fileData = file_get_contents($path, FILE_USE_INCLUDE_PATH);
    if (FALSE === $fileData) {
      throw new HttpException(400, 'File could not be processed');
    }

    if (\Drupal::service('module_handler')->moduleExists('clamav')) {
      $fileName = uniqid(rand());
      /** @var \Drupal\file\FileInterface $file */
      $file = file_save_data($fileData, 'public://' . $fileName , FILE_EXISTS_REPLACE);
      /** @var \Drupal\clamav\Scanner $clamAV */
      $clamAV = \Drupal::service('clamav');
      $result = $clamAV->scan($file);
      try {
        $file->delete();
      }
      catch (EntityStorageException $e) {
        $this->logger->error($e->getMessage());
      }

      if ($result == Scanner::FILE_IS_INFECTED) {
        throw new HttpException(403, 'Virus found');
      }
    }

    return $fileData;
  }

$this->currentRequest is an object of the class \Symfony\Component\HttpFoundation\Request. $files is an object of the class \Symfony\Component\HttpFoundation\FileBag. The latter is derived from the first and used to get the all important object of the UploadedFile class, which contains all information about the first uploaded file, in which I am interested.

The file is read from storage (file system) and this binary is returned from the function after performing a virus scan on the file. That is to say, it performs the virus scan, when the module clamav is enabled.

The binary of the file return by the checkForVirus() function is passed to the passUploadRequest(). If you do not want to pass the file to another REST service you could take the functionality from the checkForVirus() function, not delete the File object, but instead build a response that contains relevant information about the File in a json object.

But I was going to pass the uploaded file to the third party REST api. For that I use Guzzle, which is part of Drupal 8.

You can see that an api_key is retrieved from the configuration and the agent ID retrieved from the user account is added to the headers.

The only thing not shown is how the logging is handled. The only thing of importance is to notice that all requests that are passed can be logged. Whether they should be logged and how they should be logged is your decision, but I prefer to have that option available when negotiating requests and responses for other parties at all times.

  /**
   * @param $fileData
   *
   * @return \Psr\Http\Message\ResponseInterface
   */
  protected function passUploadRequest($fileData) {
    $method = $this->currentRequest->getRealMethod();
    $files = $this->currentRequest->files;
    $keys = $files->keys();
    $uploadedFile = $this->currentRequest->files->get($keys[0]);
    $fileName = $uploadedFile->getClientOriginalName();

    $url = $this->getEndpointUrl();
    $apikey = $this->config->get('api_key');
    $agentcode = $this->getAgentCode();
    $options = [
      'debug' => FALSE,
      'timeout' => '150.00', // Server timeout.
      'connect_timeout' => '5.00', // Client timeout.
      'headers' => [
        'apikey' => $apikey,
        'AgentID' => $agentcode,
        'Cache-Control' => 'no-store',
      ],
    ];
    $mimeType = $this->mimetypeFromFileName($fileName);

    $logRequest = [
      'url' => $url,
      'method' => $method,
      'headers' => $options['headers'],
      'body' => "File upload for file '{$fileName}'",
    ];

    $options['multipart'] = [
      [
        'name' => 'file',
        'filename' => $fileName,
        'contents' => $fileData,
        'headers' => ['Content-Type' => $mimeType],
      ]
    ];

    try {
      $response = $this->httpClient->request($method, $url, $options);
      $this->responseBody = $this->apiLog->log($logRequest, $response);
      return $response;
    }
    catch (GuzzleException $e) {
      /** @var \GuzzleHttp\Exception\RequestException $e */
      if ($e->hasResponse()) {
        $this->responseBody = $this->apiLog->log($logRequest, $e->getResponse());
        return $e->getResponse();
      }
      // No response indicates a connectivity problem.
      $this->apiLog->errorLog($logRequest, $e->getMessage() . ' [' . $e->getCode() . ']');
      throw new HttpException(503, 'Service Unavailable');
    }
  }

 

The complete UploadController class:

<?php

namespace Drupal\my_api\Controller;

use Drupal\clamav\Scanner;
use Drupal\Core\Config\Config;
use Drupal\Core\Controller\ControllerBase;
use Drupal\Core\Entity\EntityStorageException;
use Drupal\Core\Session\AccountProxyInterface;
use Drupal\user\Entity\User;
use Drupal\vlot_api\ApiLog;
use GuzzleHttp\ClientInterface;
use GuzzleHttp\Exception\GuzzleException;
use Psr\Log\LoggerInterface;
use Symfony\Component\DependencyInjection\ContainerInterface;
use Symfony\Component\HttpFoundation\JsonResponse;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\HttpKernel\Exception\HttpException;

/**
 * Class UploadController.
 */
class UploadController extends ControllerBase {

  /**
   * The watchdog log.
   *
   * @var \Psr\Log\LoggerInterface
   */
  protected $logger;

  /**
   * API Log object.
   *
   * @var \Drupal\vlot_api\ApiLog
   */
  protected $apiLog;

  /**
   * A current user instance.
   *
   * @var \Drupal\Core\Session\AccountProxyInterface
   */
  protected $currentUser;

  /**
   * The current request.
   *
   * @var \Symfony\Component\HttpFoundation\Request
   */
  protected $currentRequest;

  /**
   * A configuration object.
   *
   * @var \Drupal\Core\Config\Config
   */
  protected $config;

  /**
   * The http client.
   *
   * @var \GuzzleHttp\ClientInterface $http_client
   */
  protected $httpClient;

  /**
   * End-point path.
   *
   * @var string
   */
  protected $path = '/customer-portal/v1/forms/attachments';

  /**
   * The agency ID for the user.
   *
   * @var string
   */
  protected $agentID = 'dummy code';

  /**
   * The body is read from a stream.
   *
   * @var string
   */
  protected $responseBody;

  /**
   * {@inheritdoc}
   */
  public function __construct(
    LoggerInterface $logger,
    Config $config,
    AccountProxyInterface $current_user,
    Request $current_request,
    ClientInterface $http_client,
    ApiLog $api_log
  ) {
    $this->logger = $logger;
    $this->config = $config;
    $this->currentUser = $current_user;
    $this->currentRequest = $current_request;
    $this->httpClient = $http_client;
    $this->apiLog = $api_log;
  }

  /**
   * {@inheritdoc}
   */
  public static function create(ContainerInterface $container) {
    return new static(
      $container->get('logger.factory')->get('rest'),
      $container->get('config.factory')->get('vlot_api'),
      $container->get('current_user'),
      $container->get('request_stack')->getCurrentRequest(),
      $container->get('http_client'),
      $container->get('vlot_api.api_logger')
    );
  }


  /**
   * Handle the file upload.
   *
   * @return \Symfony\Component\HttpFoundation\JsonResponse
   */
  public function Upload() {
    try {
      $user = $this->ensureUserIsKnown();
      $this->agentID = $user->get('field_agentid')->value;
      $fileData = $this->checkForVirus();
      $response = $this->passUploadRequest($fileData);
    }
    catch (HttpException $e) {
      $response = [
        'timestamp' => date('Y-m-d\TH:i:sO'),
        'status' => $e->getStatusCode(),
        'error' => Response::$statusTexts[$e->getStatusCode()],
        'message' => $e->getMessage(),
        'path' => $this->path,
      ];
      return JsonResponse::create($response, $e->getStatusCode());
    }

    $data = NULL;
    $statusCode = 500;
    if ($response) {
      $statusCode = $response->getStatusCode();
      $data = $this->responseBody;
    }
    return JsonResponse::fromJsonString($data, $statusCode);
  }

  /**
   * @param $fileData
   *
   * @return \Psr\Http\Message\ResponseInterface
   */
  protected function passUploadRequest($fileData) {
    $method = $this->currentRequest->getRealMethod();
    $files = $this->currentRequest->files;
    $keys = $files->keys();
    $uploadedFile = $this->currentRequest->files->get($keys[0]);
    // @todo Test $uploadedFile for not being NULL
    $fileName = $uploadedFile->getClientOriginalName();

    $url = $this->getEndpointUrl();
    $apikey = $this->config->get('api_key');
    $agentcode = $this->getAgentCode();
    $options = [
      'debug' => FALSE,
      'timeout' => '150.00', // Server timeout.
      'connect_timeout' => '5.00', // Client timeout.
      'headers' => [
        'apikey' => $apikey,
        'AgentID' => $agentcode,
        'Cache-Control' => 'no-store',
      ],
    ];
    $mimeType = $this->mimetypeFromFileName($fileName);

    $logRequest = [
      'url' => $url,
      'method' => $method,
      'headers' => $options['headers'],
      'body' => "File upload for file '{$fileName}'",
    ];

    $options['multipart'] = [
      [
        'name' => 'file',
        'filename' => $fileName,
        'contents' => $fileData,
        'headers' => ['Content-Type' => $mimeType],
      ]
    ];

    try {
      $response = $this->httpClient->request($method, $url, $options);
      $this->responseBody = $this->apiLog->log($logRequest, $response);
      return $response;
    }
    catch (GuzzleException $e) {
      /** @var \GuzzleHttp\Exception\RequestException $e */
      if ($e->hasResponse()) {
        $this->responseBody = $this->apiLog->log($logRequest, $e->getResponse());
        return $e->getResponse();
      }
      // No response indicates a connectivity problem.
      $this->apiLog->errorLog($logRequest, $e->getMessage() . ' [' . $e->getCode() . ']');
      throw new HttpException(503, 'Service Unavailable');
    }
  }

  /**
   * Get the url of the endpoint to pass the request to.
   *
   * @return string
   */
  public function getEndpointUrl() {
    $url = $this->config->get('endpoint');
    $url .= $this->path;

    $query = $this->currentRequest->getQueryString();
    $query = str_replace(['&_format=json', '_format=json&', '_format=json', ], '', $query);
    if (!empty($query)) {
      $url .= '?' . $query;
    }

    return $url;
  }

  /**
   * Get the agency ID for the user.
   *
   * @return string
   */
  public function getAgentCode() {
    return $this->agentID;
  }

  /**
   * Validate user authorization.
   *
   * @return \Drupal\user\Entity\User
   *
   * @throws \Symfony\Component\HttpKernel\Exception\HttpException
   */
  protected function ensureUserIsKnown() {
    if ($this->currentUser->isAnonymous()) {
      throw new HttpException(403, 'Anonymous user');
    }

    if ($user = User::load($this->currentUser->id())) {
      if ($user->isBlocked()) {
        throw new HttpException(403, 'User is blocked');
      }
      return $user;
    }
    else {
      throw new HttpException(403, 'User not found');
    }
  }

  /**
   * Check for a virus in uploaded file.
   *
   * @return bool|string
   */
  protected function checkForVirus() {
    $files = $this->currentRequest->files;
    if ($files->count() == 0) {
      return FALSE;
    }
    $keys = $files->keys();
    /** @var \Symfony\Component\HttpFoundation\File\UploadedFile $uploadedFile */
    $uploadedFile = $this->currentRequest->files->get($keys[0]);


    $fileName = $uploadedFile->getClientOriginalName();
    if (empty($fileName)) {
      throw new HttpException(400, 'File name not found');
    }

    $path = $uploadedFile->getPathName();
    $fileData = file_get_contents($path, FILE_USE_INCLUDE_PATH);
    if (FALSE === $fileData) {
      throw new HttpException(400, 'File could not be processed');
    }

    if (\Drupal::service('module_handler')->moduleExists('clamav')) {
      $fileName = uniqid(rand());
      /** @var \Drupal\file\FileInterface $file */
      $file = file_save_data($fileData, 'public://' . $fileName , FILE_EXISTS_REPLACE);
      /** @var \Drupal\clamav\Scanner $clamAV */
      $clamAV = \Drupal::service('clamav');
      $result = $clamAV->scan($file);
      try {
        $file->delete();
      }
      catch (EntityStorageException $e) {
        $this->logger->error($e->getMessage());
      }

      if ($result == Scanner::FILE_IS_INFECTED) {
        throw new HttpException(403, 'Virus found');
      }
    }

    return $fileData;
  }

  /**
   * Maps a file name with extension to a mimetype.
   *
   * @param string $fileName The file file name.
   *
   * @return string|null
   * @link http://svn.apache.org/repos/asf/httpd/httpd/branches/1.3.x/conf/mime.types
   */
  protected function mimetypeFromFileName($fileName) {
    $splitName = explode('.', $fileName);
    if (count($splitName) < 2) {
      return NULL;
    }

    $mimetypes = [
      '7z' => 'application/x-7z-compressed',
      'aac' => 'audio/x-aac',
      'ai' => 'application/postscript',
      'aif' => 'audio/x-aiff',
      'asc' => 'text/plain',
      'asf' => 'video/x-ms-asf',
      'atom' => 'application/atom+xml',
      'avi' => 'video/x-msvideo',
      'bmp' => 'image/bmp',
      'bz2' => 'application/x-bzip2',
      'cer' => 'application/pkix-cert',
      'crl' => 'application/pkix-crl',
      'crt' => 'application/x-x509-ca-cert',
      'css' => 'text/css',
      'csv' => 'text/csv',
      'cu' => 'application/cu-seeme',
      'deb' => 'application/x-debian-package',
      'doc' => 'application/msword',
      'docx' => 'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
      'dvi' => 'application/x-dvi',
      'eot' => 'application/vnd.ms-fontobject',
      'eps' => 'application/postscript',
      'epub' => 'application/epub+zip',
      'etx' => 'text/x-setext',
      'flac' => 'audio/flac',
      'flv' => 'video/x-flv',
      'gif' => 'image/gif',
      'gz' => 'application/gzip',
      'htm' => 'text/html',
      'html' => 'text/html',
      'ico' => 'image/x-icon',
      'ics' => 'text/calendar',
      'ini' => 'text/plain',
      'iso' => 'application/x-iso9660-image',
      'jar' => 'application/java-archive',
      'jpe' => 'image/jpeg',
      'jpeg' => 'image/jpeg',
      'jpg' => 'image/jpeg',
      'js' => 'text/javascript',
      'json' => 'application/json',
      'latex' => 'application/x-latex',
      'log' => 'text/plain',
      'm4a' => 'audio/mp4',
      'm4v' => 'video/mp4',
      'mid' => 'audio/midi',
      'midi' => 'audio/midi',
      'mov' => 'video/quicktime',
      'mp3' => 'audio/mpeg',
      'mp4' => 'video/mp4',
      'mp4a' => 'audio/mp4',
      'mp4v' => 'video/mp4',
      'mpe' => 'video/mpeg',
      'mpeg' => 'video/mpeg',
      'mpg' => 'video/mpeg',
      'mpg4' => 'video/mp4',
      'oga' => 'audio/ogg',
      'ogg' => 'audio/ogg',
      'ogv' => 'video/ogg',
      'ogx' => 'application/ogg',
      'pbm' => 'image/x-portable-bitmap',
      'pdf' => 'application/pdf',
      'pgm' => 'image/x-portable-graymap',
      'png' => 'image/png',
      'pnm' => 'image/x-portable-anymap',
      'ppm' => 'image/x-portable-pixmap',
      'ppt' => 'application/vnd.ms-powerpoint',
      'pptx' => 'application/vnd.openxmlformats-officedocument.presentationml.presentation',
      'ps' => 'application/postscript',
      'qt' => 'video/quicktime',
      'rar' => 'application/x-rar-compressed',
      'ras' => 'image/x-cmu-raster',
      'rss' => 'application/rss+xml',
      'rtf' => 'application/rtf',
      'sgm' => 'text/sgml',
      'sgml' => 'text/sgml',
      'svg' => 'image/svg+xml',
      'swf' => 'application/x-shockwave-flash',
      'tar' => 'application/x-tar',
      'tif' => 'image/tiff',
      'tiff' => 'image/tiff',
      'torrent' => 'application/x-bittorrent',
      'ttf' => 'application/x-font-ttf',
      'txt' => 'text/plain',
      'wav' => 'audio/x-wav',
      'webm' => 'video/webm',
      'wma' => 'audio/x-ms-wma',
      'wmv' => 'video/x-ms-wmv',
      'woff' => 'application/x-font-woff',
      'wsdl' => 'application/wsdl+xml',
      'xbm' => 'image/x-xbitmap',
      'xls' => 'application/vnd.ms-excel',
      'xlsx' => 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
      'xml' => 'application/xml',
      'xpm' => 'image/x-xpixmap',
      'xwd' => 'image/x-xwindowdump',
      'yaml' => 'text/yaml',
      'yml' => 'text/yaml',
      'zip' => 'application/zip',
    ];

    $extension = end($splitName);
    $extension = strtolower($extension);

    return isset($mimetypes[$extension]) ? $mimetypes[$extension] : NULL;
  }

}

 

Add new comment