#composerphp

The require property MUST be provided and it MUST contain the requirement to TYPO3 with a version constraint.

It should be very easy to understand, that TYPO3 extensions only work with TYPO3 and thus TYPO3 needs to be added as dependency to the composer.json of extensions. But let me elaborate what I recommend as concrete version constraint.

If your extension is compatible with the two latest LTS releases, the version constraint is pretty straightforward: ^6.2 || ^7.6

If your extension relies on a certain bugfix of a certain bugfix version, you can specify that minor release easily: ^6.2.13 || ^7.6.11

Things get a little bit more tricky, if your extension supports current TYPO3 sprint releases. It is important to know that minor versions bumps of TYPO3 sprint releases may contain breaking changes. If your extension is compatible with version 8.0.0, it does not necessarily mean it is compatible with 8.1.0 or 8.2.0.

If you now specify a version constraint of ^8.0.0 or < 8.99.99 at the time when version 8.0.0 is released, then you declare that this exact same version, this exact same code base, will be compatible with all future sprint releases, which in fact you can not, unless you can forsee the future.

If you nevertheless use such a loose version constraint, you are making a bet. If you are lucky, nothing bad happens and your code magically works with a future sprint release. If you are not so lucky, than your incompatible extension version might be installed with the latest TYPO3 sprint release and might be the reason, why a complete TYPO3 installation is broken.

I suggest not to bet, but specify exactly with which sprint release version your extension is compatible: ^7.6 || >=8.3.0 <8.6 would be a perfect version constraint, when 8.5.0 has been released and your extension version is compatible with that version, but needs at least 8.3.0. If 8.6.0 is released and your code still works, just bump the version constraint with one commit and release a new version that is compatible with the latest TYPO3 sprint release. If your code does not work, you made sure, broken code is not installed.

It SHOULD contain at least a requirement to typo3/cms-core. It SHOULD NOT contain a requirement to typo3/cms directly to avoid confusions as extensions cannot be used as root package with TYPO3.

Now what does this mean? The package typo3/cms-core isn't available on Packagist, so why would I want to add a dependency to it? Well, here is the deal: typo3/cms is available on Packagist and declares to be a replacement for typo3/cms-core and all other system extensions of the current TYPO3 core (typo3/cms-backend, typo3/cms-info, …). Currently if you specify typo3/cms-core, your extension composer.json file cannot act as a root package in the sense that you cannot go to the extension directory and simply type composer install. If you do so, you'll get an error message. This is important, because TYPO3 cannot deal with extensions that are not in the typo3conf/ext directory anway.

You may want to install and test multiple TYPO3 version in your CI setup. But even then I find it useful to not specify typo3/cms directly in the extension composer.json and do composer install during CI, because then composer will always only install the latest TYPO3 version your extension is compatible with.

So you would end up doing composer require typo3/cms $TYPO3_VERSION anyway, which would overwrite any previously specified dependency to typo3/cms, but will add a depedency to it alongside letting composer check whether the newly required TYPO3 version is actually compatible with what the extension specifies. Believe me or, this setup caught quite some errors for me and some other people I know. It is not a strict requirement to use typo3/cms-core instead of typo3/cms, but it gives enough benefits to recommend it.

It COULD contain requirements to other individual TYPO3 system extensions, to properly specify which TYPO3 extensions this extension depends on. This is useful especially in combination with TYPO3 Console, but also provides information, which will be useful in the future.

What I mean with that is, that individual core extensions might be split out of the big TYPO3 repository, making it possible to install only these TYPO3 system extensions, that are strictly required for TYPO3 or your project. For this to work properly, extension (or composer package) authors would need to start specifying which exact TYPO3 system extensions their code depends on. I did this for TYPO3 console for example. Doing so now comes with no downside and no side effect. It only enables better compatiblity for the future and allows TYPO3 Console to automatically detect which TYPO3 system extensions need to be active in a project.

How to read this specification

I have created an example with a minimal setup for a TYPO3 extension. In the following I will describe each property in the composer.json using the modifiers MUST, SHOULD, COULD. MUST means that it will break hard without specifying that configuration property or if specifying a wrong value. SHOULD means that not providing that configuration property might lead to faulty or inconsistent and confusing behaviour. COULD means that this is completely optional, but a nice addition and therefore recommended.

The following rules apply in case the extensions are released publicly. For private extensions the requirements are less strict, but they SHOULD be followed nevertheless.

Configuration properties

"name"

The name property MUST be provided and it MUST have a vendor name and a package name (separated by a slash /). Vendor name and package name MUST only contain lowercase alphanumerical characters, dashes or underscores (regex of the complete name: [a-z0-9]([_.-]?[a-z0-9]+)*/[a-z0-9]([_.-]?[a-z0-9]+)*).

The vendor name MUST not already be registered on Packagist. If it is registered, you MUST be owner of this vendor name on Packagist. If it is not registered you MUST register at least one package with this vendor name on Packagist.

"description"

The description property is one line describing the extrension and MUST be provided.

"keywords"

The keywords property COULD be added to tag your extension with some keywords.

"homepage"

The homepage property COULD be added to provide an URL to the homepage of the extension.

"authors"

The authors property SHOULD be added to provide information who created the extension.

"license"

The license property SHOULD be added to specify the license under which the extension is published

"type"

The type property MUST be provided and it MUST exactly have the value typo3-cms-extension

"version"

The version property SHOULD NOT be provided, if you are using TYPO3 version 7.6 and above. However for TYPO3 version 6.2 it SHOULD be provided to avoid issues during extension activation in TYPO3 extension manager.

"require"

The require property MUST be provided and it MUST contain the requirement to TYPO3 with a version constraint. It SHOULD contain at least a requirement to typo3/cms-core. It SHOULD NOT contain a requirement to typo3/cms directly to avoid confusions as extensions cannot be used as root package with TYPO3. It COULD contain requirements to other individual TYPO3 system extensions, to properly specify which TYPO3 extensions this extension depends on. This is useful especially in combination with TYPO3 Console, but also provides information, which will be useful in the future.

For TYPO3 version 6.2, the require section MUST NOT contain any other dependencies than the TYPO3 version.

For TYPO3 version 7.6, the require section SHOULD NOT contain any dependencies than the TYPO3 version. If composer packages are added, then non composer usage of the extension SHOULD be considered. If TYPO3 extensions are added, then the composer name from Packagist SHOULD be used. If the dependent extension is only available in TER, then the TER composer name SHOULD be used.

"replace"

The replace property MUST be provided if the composer package name part does not match the extension name and is highly recommended thus SHOULD be provided in any case. If it is provided, the first replace value MUST match exactly the extension key and the second replace value SHOULD match the TER package key (extension key prefixed with "typo3-ter/" and with underscores (_) replaced with dashes (-). The version constraint MUST exactly be self.version

"autoload"

The autoload property MUST be provided if the extension provides PHP classes. The autoload definition MUST be psr-4 or classmap and MUST NOT contain some other autoload type. The folder name for psr-4 is recommended to be Classes/, but can be any other folder as well. The PHP namespace prefix for psr-4 MUST only have two parts, where the first MUST be your vendor name in upper camel case and the second MUST be your extension name in upper camel case. While the latter sounds very restrictive, it is essential especially when using Extbase, as the framework very often makes assumptions on the extension key derived from the namespace prefix.

Tools support

Currently there is no tool support, so you have to check the above specification manually. I would find it useful to have two things. One tool, that supports you with creating a composer.json for your TYPO3 extension. A second tool could validate existing composer.json files against the specification above. If you are willing to contribute such functionality, get in touch with me and I will mentor this effort. Thanks in advance.

Spread the word!

Please check your own extensions and fix the composer.json files accordingly and help others to fix their composer.json files. It'll help to improve the TYPO3 extension ecosystem.

The years B.C.

In the years B.C. (before composer), PHP code was handled differently. It was randomly copied from forums, publicly available code repositories or at best taken from PEAR and then burried into an extension and most of the time kept there untouched for ages. Updates or bugfixes in libraries weren't handled most of the time and sometimes the library code was changed arbitrarily for the needs of the extension, making updates of the library code close to impossible. But it worked well (enough). It worked well enough because nobody knew a better way to handle it.

The age of composer

Then some crazy people decided to change the PHP world and created a set of tools and kickstarted an ecosystem which since then thrives. You need a PHP library in your composer managed project? Go find it on Packagist, type composer require name-of-vendor/name-of-package in your project root and be done. Using the library code just works. Updating it works simply by typing composer update name-of-vendor/name-of-package. If the library needs another library, or anything needs a special PHP version, or another library needs another version of a library you intend to use, composer will resolve everything for you just nicely (most of the times).

The TYPO3 Extension Repository (TER)

The TER exists for years and it an integral part of the ecosystem that made TYPO3 thrive. However technically it is a bit aged. The TER and the TYPO3 Extension manager can handle updates of TYPO3 versions and even extensions. It can also handle dependencies to other extensions. However resolving these dependencies is less stable and capable than what composer does. Most importantly it can only handle dependencies from the TYPO3 ecosystem, not from the composer ecosystem. This means, if you want to publish an extension to TER, that needs code from a composer library, you need to take care and include it into your TYPO3 extension, like in the old B.C. days.

Bundling libraries in phar files

There is a project, which you can use to bundle any composer based library, or even a set of libraries as single phar file. I find it much nicer to ship one file with dependent third party code, instead of hundreds of files.

Benefits

More secure: The complete phar might be exposed but not single files.

Cleaner: It is much more clear which is a library and which is your code.

More reliable: Code scans or code formatters or linters in your CI environment will ignore phar files by default

Smaller: The phar files are smaller in size than the sum of all files

And probably more, but these are enough to convince me.

Howto

Install the tool globally and make sure $HOME/.composer/vendor/bin is set in your path:

composer global require clue/phar-composer

Create a directory in your extension and create the phar archive (symfony/process for example):

mkdir Libraries phar-composer build symfony/process Libraries/symfony-process.phar

In your extension code, before you want to use the libraries in question, include the autoloader file of your phar:

use TYPO3\CMS\Core\Utility\ExtensionManagementUtility @include 'phar://' . ExtensionManagementUtility::extPath('ext-key') . 'Libraries/symfony-process.phar/vendor/autoload.php';

Optimization

There are several things I have done to optimize this in my TYPO3 Console package:

I excluded the phar from the packages delivered by composer. In that case the phar is not needed.

I recreate the lib with the current version on every travis build, which means the TER upload will always include the current version

I set classmap-authoritative to true and prepend-autoloader to false for the phar class loader. That way I can just include the phar if present and the autoloader of the phar is only used when the class is not loadable.

Summary

These files do not replace a proper dependency management. It still is a relic from the old times to bundle third party code and especially when this becomes more public, we all will run into conflicts when one extension requires another incompatible version of the same library. But I consider this OK, to support legacy systems at least for a while.

But if you can, you should start setting up your TYPO3 projects with composer. Mid term the TER will also be replaced with something that speaks composer but not "Extension Manager".

Introduction

What I'm going to describe here is not a complete composer setup, which would also be perfectly possible with TYPO3 6.2, but the few steps you need to take to take advantage from composer class loading in your existing non composer TYPO3 6.2. projects. If you already use composer to setup your TYPO3 projects, you may as well want to have a peak into this article to make sure you get the most out of the possibilities.

Setup

A prerequisite for this to work is, that you put your TYPO3 sources in a folder, not a link, and this folder resides on the same level as your index.php like this (I skipped fileadmin, typo3conf, … directories here for better overview):

lrwxr-xr-x 1 19 Sep 28 10:29 index.php -> typo3_src/index.php lrwxr-xr-x 1 15 Sep 28 10:29 typo3 -> typo3_src/typo3 drwxr-xr-x+ 17 578 Sep 28 10:28 typo3_src

Step 1: Add composer.json

Add this composer.json file on the same directory level:

Step: 2: Adapt class loading registration in composer.json

The example composer.json file from the gist comes with class loading information for every TYPO3 core extension, including the class aliases for 4.x compatiblity. What you now need to do is add class loading information for every single extension in your installation. I prepared examples for typo3_console, news and tt_news, which should be self-explanatory. You basically need to add all paths in which the extensions in your installation have classes. Make sure not to forget to specify the relative path to the directory (seen from the composer.json file).

You can even try to only specify "classmap": ["typo3conf/ext"] as a single path, which saves you quite some time and works out in most cases, but can fail in some edge cases.

Step 3: Perform a composer install in this directory

Download composer if not done already, navigate to the directory with the composer.json file and perform a composer install. The result should look like this:

lrwxr-xr-x 1 19 Sep 28 10:29 index.php -> typo3_src/index.php drwxr-xr-x+ 3 102 Sep 28 10:22 Packages -rw-r--r--+ 1 8194 Sep 29 12:52 composer.json -rw-r--r--+ 1 2334 Sep 28 13:16 composer.lock drwxrwsr-x+ 4 136 Sep 28 13:22 fileadmin lrwxr-xr-x 1 19 Sep 28 10:29 index.php -> typo3_src/index.php lrwxr-xr-x 1 15 Sep 28 10:29 typo3 -> typo3_src/typo3 drwxr-xr-x+ 17 578 Sep 28 10:28 typo3_src -rwxr-xr-x+ 1 1279 Sep 28 10:32 typo3cms drwxrwsr-x+ 8 272 Oct 10 17:45 typo3conf drwxrwsr-x+ 24 816 Sep 29 12:48 typo3temp drwxrwsr-x+ 8 272 Sep 28 13:10 uploads

It is mandatory that the Packages folder, which is created during the composer install run is on the same level as the typo3_src folder.

Step 4: Set the TYPO3_COMPOSER_AUTOLOAD environment variable

Add the following to your apache vhost configuration:

SetEnv TYPO3_COMPOSER_AUTOLOAD "On"

or the following in your .htaccess file:

RewriteRule .? - [E= TYPO3_COMPOSER_AUTOLOAD:On]

For nginx, you can set a fcgi param like that:

fastcgi_param TYPO3_COMPOSER_AUTOLOAD On;

Verify that it worked

Remove all class loader cache files in typo3temp/Cache/Data/cache_classes and load the backend and frontend of your website. The website should show up and no cache files should be added to this directory. Do not delete the complete typo3temp/Cache because then, a few class cache entries are written anyway and you cannot confirm or deny whether you did everything correctly.

Optimize

If you do not have every single TYPO3 core extension installed, you should consider removing the lines from the composer.json file which point to core extensions you do not use in the project.

If you do not want to expose the TYPO3 sources directory and the Packages directory and your composer.json file, as they contain quite some information about your installation, you can create a web folder and link the necessary folders and files into it, e.g. like that:

$ ls -l web/ lrwxr-xr-x 1 12 Oct 10 17:43 fileadmin -> ../fileadmin lrwxr-xr-x 1 12 Oct 10 17:44 index.php -> ../index.php lrwxr-xr-x 1 8 Oct 10 17:44 typo3 -> ../typo3 lrwxr-xr-x 1 12 Oct 10 17:45 typo3conf -> ../typo3conf lrwxr-xr-x 1 12 Oct 10 17:44 typo3temp -> ../typo3temp lrwxr-xr-x 1 10 Oct 10 17:44 uploads -> ../uploads

Minimum setup

In this section I describe what you can do to get better class loading suport in TYPO3 7LTS. Everthing you find here, you can safely do without loosing compatibility with older TYPO3 versions!

Compatiblity mode

If you don't want to, you do not have to do anything! Regarding class loading your extension will work with TYPO3 7LTS anyway in 99,9% of the cases. TYPO3 scans the extension directory for class or interface files and registers every single one automatically to ensure backwards compatibility.

Important: If you add classes during development they will not be found by the class loader unless you update your class information files. If you have classes located e.g. in a tests folder which should not be availble in production, you should consider one of the following ways to configure class loading.

Extensions having one folder with classes

Considering you have an Extbase extension (or an extension where all classes and interfaces reside in a Classes folder) you can simply add the following to your ext_emconf.php file:

'autoload' => array( 'classmap' => array('Classes') ),

The folder Classes will be scanned and every class or interface found will automatically be registered, no matter if the classes use PHP namspaces or not.

Important: If you add classes during development they will not be found by the class loader unless you update your class information

Extensions using namespaces (possible since TYPO3 6.0)

TYPO3 introduced a way to load classes using namespaces, which was later standardized by the PHP framework interoperability group as PSR-4. If your extension has namespaced classes, then you can add the following to your ext_emconf.php file:

'autoload' => array( 'psr-4' => array('Vendor\\ExtName\\' => 'Classes') ),

Here you do not only specify a folder, but also a namespace prefix which all classes in this folder must have. The prefix must end with a backslash!

Note: If you do so, newly added classes during development can be resolved automatically, whithout any further action! There is no need to update your class information files!

Legacy extension which previously used an ext_autoload.php file

As outlined before, it is not necessary any more to specify the name and location of every single class. You can specify folders and TYPO3 scans for the classes in them. So for example to provide class information for tt_news, Rupi could add the following to the ext_emconf.php:

'autoload' => array( 'classmap' => array( 'pi', 'mod1', 'lib', 'class.tx_ttnews_compatibility.php', ) ),

Note: As you can see, it is perfectly fine to add multiple folders or even single files to the class map. However, just to stress it again, if you add classes during development they will not be found by the class loader unless you update your class information files manually.

Advanced setup

This section is a collection of further things you can do to tweak your development experience regarding class loading in TYPO3 7LTS.

Class loading information in composer.json

If your extension has a composer.json file already with a psr-4 or classmap autoload section, you are done. No further action is required. The section in the composer.json file can look like this:

"autoload": { "psr-4": { "Vendor\\ExtName\\": "Classes" }, "classmap": ["Resources/Private/PHP/SomeLib"] },

As you can see, you can combine both classmap and psr-4 sections (which is also possible of course in an ext_emconf.php file.

Fixture classes

The old TYPO3 class loader could automatically resolve classes inside a Tests folder, in case the namespace reflected the folder structure below. In TYPO3 7LTS this is not the case any more, which really is a good thing! If you do not want to use composer to execute your extension tests individually like I described in a previous article, but execute them in the context of your TYPO3 project installation, you can add an autoload-dev section in your ext_emconf.php file:

'autoload-dev' => array( 'psr-4' => array('Vendor\\ExtName\\Tests\\' => 'Tests') ),

Note: Just like composer, TYPO3 ignores an autoload-dev section in a composer.json file. So unlike the regular autoload section, the autoload-dev section is only evaluated if present in the ext_emconf.php file.

General notes

Update class loading information

The class loading information is updated every time you activate or deactivate an extension via extension manager. In the rare case you want to update it manually, you can run the following command from the command line:

typo3/cli_dispatch.phpsh extbase \ extension:dumpclassloadinginformation

Composer mode

It is possible to manage the source code for your TYPO3 project exclusively with composer (which I can highly recommend). In that case, all class loading information provided by extensions in the ext_emconf.php files is completely ignored! However as best practice and for the time being it is recommended to nevertheless provide this information in your extensions, so that it can be used in projects which are not set up using composer.

The challenge

We invested a lot of time to improve the situation and introduced some locking, but still a few main conceptual problems remained:

Besides having a registry with class name to file mapping, TYPO3 tried to resolve class files using several (partly conflicting) strategies.

To avoid doing these expensive operations again and again, the information is cached in files, but these files contain absolute paths, which makes the cache unusable if CLI access has different paths than web access. Warming up caches on CLI during deployment is impossible in such situations.

The file location of classes never change in an installation and could be known upfront, but TYPO3 tries to resolve the location of a class dynamically in a regular request. At the same time, it can never be determined whether the information is complete or not, because different code paths, could need different PHP classes that are then looked up dynamically.

In a production system the class information does not change unless an extension is installed. However this information was written into volatile caches, which are often flushed. This lead to repeatedly unnecessary re-calculation of these caches.

The solution

Finding a solution for these issues was brain twisting for me, until I realized that composer and standardization of class autoloading solved all problems for us at once. We only needed to implement composer more deeply into our system. Here is why composer class loading solves all the issues we had:

There is no magic involved in the composer class loader. How classes are loaded must be specified for each package (or in TYPO3 terms: each extension).

By specifying the class loading information, the class names in your extensions, must no longer be related to the extension name and are not bound to a Classes folder.

With this class loading configuration the composer class loader knows how to load every single class during build time. No assumptions, no magic, no additional class information lookup at runtime.

The class information is static. In a production (TYPO3) system, it is just there and never changes unless a package (extension) is added or removed. No failures due to broken caches during deployment, as there are no caches any more.

Class information configuration is simple. Just a few lines instead of long ext_autoload.php files.

Class information is stored with relative paths. So it can be generated on one system and deployed to another one without issues.

The implementation

We completely removed all TYPO3 class loading code in TYPO3 7 and now rely on the composer class loader to load all classes of TYPO3 and every installed extension. If you set up your TYPO3 installation (including extensions) with composer (not downloading a package and symlinking the source like in the old days), TYPO3 code does nothing regarding class loading, composer takes over that task completely.

Non composer mode

In case you want to set up your TYPO3 project traditionally without using composer, TYPO3 ships with the composer class loader and all necessary information to load all core classes, but additionally executes a very slim layer of code to detect and register classes of active extensions, so that the composer class loader can load them as well. If an extension does not provide class loading information, the complete extension directory is scanned for class files and every class found, will automatically be registered for the class loader. The ext_autoload.php file is completely ignored in TYPO3 7.

There might be some cases where a full scan of all directories is not desired, e.g. when third party libraries are shipped or test classes are provided with the extension. Because of that, extension authors can provide class loading information which will be evaluated by TYPO3. As of TYPO3 7.5 this information can be provided in an autoload section of the ext_emconf.php file, or in a composer.json file.

In contrast to composer, TYPO3 only evaluates the psr-4 and classmap sections. But we think this is more than enough and even easier to handle than before.

Benefits

We do not have a classes cache any more. So this cache cannot be corrupt, will never be flushed (and thus being unavailable) and must not be build during runtime. Besides that, we have a lot less code. The results are: 

Better performance

Higher reliability

Higher flexibility

Less maintenance costs

Implications

We do not generate class loading information during runtime any more, but change the class loading information every time an extension is added or removed from the installation. If you use composer, it takes over that task and updates the class loading information on every composer install or composer update. If you do not use composer, TYPO3 will update the class loading information every time you add or remove an extension with the extension manager or via command line (typo3/cli_dispatch.phpsh extbase extension:install realurl) or if you explicitly trigger TYPO3 to re-build the information from the command line: typo3/cli_dispatch.phpsh extbase extension:dumpclassloadinginformation

If you develop extensions and add classes, it is therefore highly recommended to follow the PSR-4 specification and add a psr-4 section to your ext_emconf.php or composer.json of your extension.

Combined TYPO3 6.2 and TYPO3 7 compatiblity

Since ext_autoload.php files are ignored in TYPO3 7 and an autoload section in ext_emconf.php is ignored in TYPO3 6.2, it is pretty easy to provide compatibility for both TYPO3 versions with your extension. Just add both, and your extension will work in both TYPO3 versions. If you follow the PSR-4 standard, you of course only need to add the according section in ext_emconf.php. Of course your classes and interfaces must reside in a Classes folder, as TYPO3 6.2 only looks up classes in that folder. My scaffold extension for example is compatible with both TYPO3 versions in that regard.

Conclusion

The listed benefits should be enough to justify this change. With this step we also step closer to the broader PHP community by using composer. It is something new to learn for TYPO3 community members, but we are convinced this is for the benefit of our product and both the TYPO3 and PHP community.