doc_core_insight

Improvements

In the past many extension developers already published their extensions to Packagist. If that is the case it is recommended to require these packages over the ones offered via composer.typo3.org (e.g. georgringer/news instead of typo3-ter/news).

It is recommended, because as a legacy service, composer.typo3.org does not expose all features extension developlers can use. Most notably, requirements to third party composer packages are ignored. Also class aliases are ignored.

However, to check whether an extension is also available on Packagist, involved quite some boring steps and an additional lookup - until now.

Tell users your extension is available on Packagist

To improve this situation, extension owners that have their extension published on Packagist, can now conveniently communicate this to their users, by just confirming a checkbox and adding the composer name of their extension directly on extensions.typo3.org.

Just log in and navigate to your extensions:

Then open settings for the extension by pressing “Edit”, tick the checkbox to confirm the extension is published, enter the composer name and press save.

If you do so, this information is automatically propagated to composer.typo3.org, where a nice red warning box is show, that requiring this extension through the legacy service is deprecated:

Additionally a warning message appears as well on command line, when requiring this extension through our legacy service:

Last but not least, this information will be presented directly on extensions.typo3.org (this feature is not ready yet, but will arrive soon).

Further improvements

Satis has been upgraded to the latest stable version

Abandoned extensions are shown as abandoned (in the same way, but without the notice which composer name to use instead)

All extensions have a link to their detail view on extensions.typo3.org included

Credits

Thanks a lot Thomas Löffler for your constant work on improving extensions.typo3.org and your help with reviewing, merging and deploying this feature.

Thanks a bunch mehrwert for sponsoring this feature.

Challenge

While I always wanted to have such a feature and I knew others were interested in it as well or even created some code already, I didn't have the time to work on a solution that was good enough to be implemented into TYPO3 Console.

The existing code by Christin Zenker worked for older TYPO3 installations, starting with TYPO3 7.6 the install tool code changed in a way, that it was even more coupled with a web request. This means the old code stopped working and (re-) implementing it was far from trivial.

Not only that the current install tool code expects $_GET to be filled, all user interaction was streamlined for the web, upgrade wizard IDs were changed to the fully qualified (namspaced) class names, which is a bit vebose and clunky to type on the command line and and most output from the wizards contain HTML, which makes it ugly or unreadable on cli.

Last but not least the install tool upgrade tool got more features like being able to re-run upgrade wizards, although they have been marked as executed already.

Two things were clear to me.

Having a good user experience from command line for this task and implementing a complete feature set, would require quite some conceptual work besides the required work for the implementation.

It was nothing I could accomplish im my (limited) free time.

Challenge accepted

A few months ago I was contacted by Mittwald. They asked if I could implement this feature, if they sponsor it. Only a few weeks later I was at a TYPO3 code sprint at Systime, where I talked to Kasper Ligaard about this topic. Systime also was very interested in this feature, because they were about to upgrade literally hunderds of TYPO3 installations from 7.6 to 8.7 and therefore offered to sponsor my work on that as well.

With such generous sponsoring I was able to prioritize this task, postpone other work and start with working on it a few weeks later.

Result

I quickly had some code ready to be able to execute a single upgrade wizard from the command line, but was a very small portion of what needed to be done.

It took me another three weeks and two TYPO3 Console releases (4.5 and 4.6) to get the whole feature set:

List all upgrade wizards with a short identifier

List wizards that need execution

Execute a single wizard using its short identifier

Execute multiple wizards by using their short identifier

Force execution of a single wizard, so that it is executed even when marked as done

Pass arguments to a single wizard, so that wizards that expect user interaction can be executed

Pass arguments for multiple wizards

Execute all wizards that are scheduled for execution, making it possible to perform a complete TYPO3 migration without user interaction

Check compatiblity of all third party extensions with a given TYPO3 version

Automatically disable third party extensions, that have broken (incompatible) code in ext_tables.php or ext_localconf.php

A progress bar when executing all wizards, so that progress for long running upgrades is made visible

Every action is covered with functional tests

Profit

It will change the way TYPO3 upgrades are done in the future.

With these features now present in TYPO3 Console, I personally will never do TYPO3 upgrades using the install tool any more. Given the success of TYPO3 Console, I predict that others will jump on that wagon, too. This feature will change the way TYPO3 upgrades are done in the future. Here is why:

Upgrades are done by executing a single command typo3cms upgrade:all, in contrast of logging into the Install Tool and manually clicking your way thorugh the UI.

Never run into timeouts, even on large TYPO3 installations.

It is possible to make code changes on a local development system, test the upgrade and finally deploy to live. This works smoothly because typo3cms upgrade:all not only can be easily added to your deploy chain, it also is idempotent. It does nothing, if upgrade has been performed already.

Thanks

The require property MUST be provided and it MUST contain the requirement to TYPO3 with a version constraint.

It should be very easy to understand, that TYPO3 extensions only work with TYPO3 and thus TYPO3 needs to be added as dependency to the composer.json of extensions. But let me elaborate what I recommend as concrete version constraint.

If your extension is compatible with the two latest LTS releases, the version constraint is pretty straightforward: ^6.2 || ^7.6

If your extension relies on a certain bugfix of a certain bugfix version, you can specify that minor release easily: ^6.2.13 || ^7.6.11

Things get a little bit more tricky, if your extension supports current TYPO3 sprint releases. It is important to know that minor versions bumps of TYPO3 sprint releases may contain breaking changes. If your extension is compatible with version 8.0.0, it does not necessarily mean it is compatible with 8.1.0 or 8.2.0.

If you now specify a version constraint of ^8.0.0 or < 8.99.99 at the time when version 8.0.0 is released, then you declare that this exact same version, this exact same code base, will be compatible with all future sprint releases, which in fact you can not, unless you can forsee the future.

If you nevertheless use such a loose version constraint, you are making a bet. If you are lucky, nothing bad happens and your code magically works with a future sprint release. If you are not so lucky, than your incompatible extension version might be installed with the latest TYPO3 sprint release and might be the reason, why a complete TYPO3 installation is broken.

I suggest not to bet, but specify exactly with which sprint release version your extension is compatible: ^7.6 || >=8.3.0 <8.6 would be a perfect version constraint, when 8.5.0 has been released and your extension version is compatible with that version, but needs at least 8.3.0. If 8.6.0 is released and your code still works, just bump the version constraint with one commit and release a new version that is compatible with the latest TYPO3 sprint release. If your code does not work, you made sure, broken code is not installed.

It SHOULD contain at least a requirement to typo3/cms-core. It SHOULD NOT contain a requirement to typo3/cms directly to avoid confusions as extensions cannot be used as root package with TYPO3.

Now what does this mean? The package typo3/cms-core isn't available on Packagist, so why would I want to add a dependency to it? Well, here is the deal: typo3/cms is available on Packagist and declares to be a replacement for typo3/cms-core and all other system extensions of the current TYPO3 core (typo3/cms-backend, typo3/cms-info, …). Currently if you specify typo3/cms-core, your extension composer.json file cannot act as a root package in the sense that you cannot go to the extension directory and simply type composer install. If you do so, you'll get an error message. This is important, because TYPO3 cannot deal with extensions that are not in the typo3conf/ext directory anway.

You may want to install and test multiple TYPO3 version in your CI setup. But even then I find it useful to not specify typo3/cms directly in the extension composer.json and do composer install during CI, because then composer will always only install the latest TYPO3 version your extension is compatible with.

So you would end up doing composer require typo3/cms $TYPO3_VERSION anyway, which would overwrite any previously specified dependency to typo3/cms, but will add a depedency to it alongside letting composer check whether the newly required TYPO3 version is actually compatible with what the extension specifies. Believe me or, this setup caught quite some errors for me and some other people I know. It is not a strict requirement to use typo3/cms-core instead of typo3/cms, but it gives enough benefits to recommend it.

It COULD contain requirements to other individual TYPO3 system extensions, to properly specify which TYPO3 extensions this extension depends on. This is useful especially in combination with TYPO3 Console, but also provides information, which will be useful in the future.

What I mean with that is, that individual core extensions might be split out of the big TYPO3 repository, making it possible to install only these TYPO3 system extensions, that are strictly required for TYPO3 or your project. For this to work properly, extension (or composer package) authors would need to start specifying which exact TYPO3 system extensions their code depends on. I did this for TYPO3 console for example. Doing so now comes with no downside and no side effect. It only enables better compatiblity for the future and allows TYPO3 Console to automatically detect which TYPO3 system extensions need to be active in a project.

Agreement

There is one thing I agree with and I want to mention it first. I agree that Surf isn't a particulary good product name, not only because SEO for such a word would be incredible hard. The full name TYPO3 Surf isn't much better either. OK, that's it.

Disagreement

The name

And here the disagreement starts. I wouldn't say TYPO3 is a particulary good product name as well. But here we are with it being a solid CMS, which generates a lot of revenue for a lot of people (mainly in central Europe). And I like it. And I like Surf as well. The name does not make either a bad product.

NIH

The author claims that Surf is a "NIH-Product", but admits that it was invented before other PHP deployment tools existed. That is a contradiction. If it was inveted before any other likewise products, then NIH cannot be the problem here. It is just used as a pejorative rhetorical figure.

I also don't quite get how the product name has something to do with NIH, but it is listed as one reason why NIH is harful.

And how some technical state of the past (Surf needs Flow) is related to that topic also is not immediately clear to me. To state false facts (Surf could only deploy Flow applications) back then, does not help here.

But maybe Surf is "too late to be an option for the whole PHP world"? Well, which project (except composer) is? Is this a requirement for a project to not count as NIH? If that would be the case, a lot of useful projects wouldn't exist by now. In fact when starting TYPO3 Console, I was asked many times why anybody would need such a thing, as the extension coreapi already existed. Back then I mainly had my personal reasons, now it enables TYPO3 to be deployed in the cloud and thus helps to leverage the complete TYPO3 ecosystem to a new level.

I'm not trying to say that reeinventing the wheel is something to strive for in general. But I'm sure that giving up on something because it is potentially NIH, harms innovation and leaves our world poorer than it could be.

Stability and reliability

The author suggests Surf being unstable and unreliable because there are only a few people (he claims it is only me) are maintianing it. Especailly because

Deployment software is not just something you reinvent or which you want to allow to fail occasionally. You want to have a fast and reliable tool, that updates the live version of your website with the least impact for the user.

thus suggests to use Capistrano.

The thing is, a deployment software itself is fairly simplistic, as long as the concept is mature enough, because in the end a deployment is just a list of tasks that need to be executed. That is why any tool that can run a list of (depending) tasks is able to deploy an application to a server. The complexity kicks in with the tasks themselves. What is needed when? What needs to be executed in which order? What is the state of my application if certain steps fail?

This complexity isn't tackled by Capistrano (or any other deployment tool). It does not support you with it. It just runs the tasks you tell it to run.

It is partly solved for TYPO3 CMS with dkddeploy, which uses capistrano, and is partly solved with TYPO3 Surf for TYPO3, as it also comes with pre-configured tasks to deploy this CMS in a reliable way.

Last but not least. The concept for executing tasks (bundled in a workflow and set up in an application) is a solid one. It is taken from the matured Capistrano project and implemented into Surf. This code hasn't been changed for long, not because it is not maintained, but because it is mature.

TYPO3 Surf a posterchild of failure?

The author puts Surf in one line with Neos, Flow and Fluid. Like Neos and Flow, Surf is a failure because it was developed by people who wanted to change the world to the better. Clearly they failed because 2016 happend. Right? I don't know. Even if I would agree with the statement that Neos and Flow failed (which I clearly do not), why on earth should this affect the adoption of a completely separate tool with a completely different scope? I mean, why should Fluid have a chance to succeed, since it is most appearent to be NIH back then. Even now, with all the other templating engines have incredibly much higher adoption rates within the PHP community. Taking these arguments, we would need to switch to Twig and drop Fluid immediately. It is not dropped because people still think it can make a difference at some point in the future.

Can Surf make a difference? It might, it might not. I don't know. I told you anway to not trust me. But don't trust others who claim to know what the future will bring and use what you are comfortable with.

My Conclusion

If you broaden your view from not just looking at some tool, that runs tasks in some order, but at the mission to reliably deploy your website, then suddenly other things start to matter.

The Surf deployment definition (basically a description what tasks need to be executed in which order) for TYPO3 CMS was the first one publicly available. It was the first turnkey solution for automated TYPO3 CMS deployments which was open sourced (by me), I'm aware of. Back then Surf was based on Flow and maintained by others, yet it was the only tool that was appearent to me.

I chose Surf because it did the job and I knew even back then, that the important part is the deployment definition, not the tool. Surf was basically useless for me before I started experimenting with automated deployments for TYPO3 CMS. It ran some tasks that left the application half ready. I knew that bigger agencies had their solutions, but they didn't share anything back then. So yes, I created deployment definitions myself knowing other solution existed (hello other good reason for NIH). Some people I asked thankfully helped me a bit in the beginning. Every single bit I learned and created, I shared.

I shared, so that other people could feel the relief I felt when my first automated deployment worked and I still feel a bit every time I now do a deployment. And it worked. I made people happy with my knowledge and this tool.

And this is what matters. The message I wanted to and still want to spread:

Burn your ftp clients

Shutdown the extension manager

Don't have any manual step in your deployment chain

The TYPO3 community evolved and I meanwhile see people using all kind of tools, like Ansible, Capistrano, GiltabCI, Jenkins, what have you. But still very few are sharing their deployment solutions.

Surf had its share in pushing this evolution, both for me personally but also for others because it targeted TYPO3 products only regarding the delivered deployment definitions. It is (currently) better suited to deploy TYPO3 CMS than any other publicly available tool (except dkddeploy), because it delivers a solution for the complicated parts.

Before considering writing posts that conclude in Surf being a "dead horse", please consider to publish your depolyment definitions (regardless which tool you use) instead. This is what helps to drive the community forward.

How to read this specification

I have created an example with a minimal setup for a TYPO3 extension. In the following I will describe each property in the composer.json using the modifiers MUST, SHOULD, COULD. MUST means that it will break hard without specifying that configuration property or if specifying a wrong value. SHOULD means that not providing that configuration property might lead to faulty or inconsistent and confusing behaviour. COULD means that this is completely optional, but a nice addition and therefore recommended.

The following rules apply in case the extensions are released publicly. For private extensions the requirements are less strict, but they SHOULD be followed nevertheless.

Configuration properties

"name"

The name property MUST be provided and it MUST have a vendor name and a package name (separated by a slash /). Vendor name and package name MUST only contain lowercase alphanumerical characters, dashes or underscores (regex of the complete name: [a-z0-9]([_.-]?[a-z0-9]+)*/[a-z0-9]([_.-]?[a-z0-9]+)*).

The vendor name MUST not already be registered on Packagist. If it is registered, you MUST be owner of this vendor name on Packagist. If it is not registered you MUST register at least one package with this vendor name on Packagist.

"description"

The description property is one line describing the extrension and MUST be provided.

"keywords"

The keywords property COULD be added to tag your extension with some keywords.

"homepage"

The homepage property COULD be added to provide an URL to the homepage of the extension.

"authors"

The authors property SHOULD be added to provide information who created the extension.

"license"

The license property SHOULD be added to specify the license under which the extension is published

"type"

The type property MUST be provided and it MUST exactly have the value typo3-cms-extension

"version"

The version property SHOULD NOT be provided, if you are using TYPO3 version 7.6 and above. However for TYPO3 version 6.2 it SHOULD be provided to avoid issues during extension activation in TYPO3 extension manager.

"require"

The require property MUST be provided and it MUST contain the requirement to TYPO3 with a version constraint. It SHOULD contain at least a requirement to typo3/cms-core. It SHOULD NOT contain a requirement to typo3/cms directly to avoid confusions as extensions cannot be used as root package with TYPO3. It COULD contain requirements to other individual TYPO3 system extensions, to properly specify which TYPO3 extensions this extension depends on. This is useful especially in combination with TYPO3 Console, but also provides information, which will be useful in the future.

For TYPO3 version 6.2, the require section MUST NOT contain any other dependencies than the TYPO3 version.

For TYPO3 version 7.6, the require section SHOULD NOT contain any dependencies than the TYPO3 version. If composer packages are added, then non composer usage of the extension SHOULD be considered. If TYPO3 extensions are added, then the composer name from Packagist SHOULD be used. If the dependent extension is only available in TER, then the TER composer name SHOULD be used.

"replace"

The replace property MUST be provided if the composer package name part does not match the extension name and is highly recommended thus SHOULD be provided in any case. If it is provided, the first replace value MUST match exactly the extension key and the second replace value SHOULD match the TER package key (extension key prefixed with "typo3-ter/" and with underscores (_) replaced with dashes (-). The version constraint MUST exactly be self.version

"autoload"

The autoload property MUST be provided if the extension provides PHP classes. The autoload definition MUST be psr-4 or classmap and MUST NOT contain some other autoload type. The folder name for psr-4 is recommended to be Classes/, but can be any other folder as well. The PHP namespace prefix for psr-4 MUST only have two parts, where the first MUST be your vendor name in upper camel case and the second MUST be your extension name in upper camel case. While the latter sounds very restrictive, it is essential especially when using Extbase, as the framework very often makes assumptions on the extension key derived from the namespace prefix.

Tools support

Currently there is no tool support, so you have to check the above specification manually. I would find it useful to have two things. One tool, that supports you with creating a composer.json for your TYPO3 extension. A second tool could validate existing composer.json files against the specification above. If you are willing to contribute such functionality, get in touch with me and I will mentor this effort. Thanks in advance.

Spread the word!

Please check your own extensions and fix the composer.json files accordingly and help others to fix their composer.json files. It'll help to improve the TYPO3 extension ecosystem.

The years B.C.

In the years B.C. (before composer), PHP code was handled differently. It was randomly copied from forums, publicly available code repositories or at best taken from PEAR and then burried into an extension and most of the time kept there untouched for ages. Updates or bugfixes in libraries weren't handled most of the time and sometimes the library code was changed arbitrarily for the needs of the extension, making updates of the library code close to impossible. But it worked well (enough). It worked well enough because nobody knew a better way to handle it.

The age of composer

Then some crazy people decided to change the PHP world and created a set of tools and kickstarted an ecosystem which since then thrives. You need a PHP library in your composer managed project? Go find it on Packagist, type composer require name-of-vendor/name-of-package in your project root and be done. Using the library code just works. Updating it works simply by typing composer update name-of-vendor/name-of-package. If the library needs another library, or anything needs a special PHP version, or another library needs another version of a library you intend to use, composer will resolve everything for you just nicely (most of the times).

The TYPO3 Extension Repository (TER)

The TER exists for years and it an integral part of the ecosystem that made TYPO3 thrive. However technically it is a bit aged. The TER and the TYPO3 Extension manager can handle updates of TYPO3 versions and even extensions. It can also handle dependencies to other extensions. However resolving these dependencies is less stable and capable than what composer does. Most importantly it can only handle dependencies from the TYPO3 ecosystem, not from the composer ecosystem. This means, if you want to publish an extension to TER, that needs code from a composer library, you need to take care and include it into your TYPO3 extension, like in the old B.C. days.

Bundling libraries in phar files

There is a project, which you can use to bundle any composer based library, or even a set of libraries as single phar file. I find it much nicer to ship one file with dependent third party code, instead of hundreds of files.

Benefits

More secure: The complete phar might be exposed but not single files.

Cleaner: It is much more clear which is a library and which is your code.

More reliable: Code scans or code formatters or linters in your CI environment will ignore phar files by default

Smaller: The phar files are smaller in size than the sum of all files

And probably more, but these are enough to convince me.

Howto

Install the tool globally and make sure $HOME/.composer/vendor/bin is set in your path:

composer global require clue/phar-composer

Create a directory in your extension and create the phar archive (symfony/process for example):

mkdir Libraries phar-composer build symfony/process Libraries/symfony-process.phar

In your extension code, before you want to use the libraries in question, include the autoloader file of your phar:

use TYPO3\CMS\Core\Utility\ExtensionManagementUtility @include 'phar://' . ExtensionManagementUtility::extPath('ext-key') . 'Libraries/symfony-process.phar/vendor/autoload.php';

Optimization

There are several things I have done to optimize this in my TYPO3 Console package:

I excluded the phar from the packages delivered by composer. In that case the phar is not needed.

I recreate the lib with the current version on every travis build, which means the TER upload will always include the current version

I set classmap-authoritative to true and prepend-autoloader to false for the phar class loader. That way I can just include the phar if present and the autoloader of the phar is only used when the class is not loadable.

Summary

These files do not replace a proper dependency management. It still is a relic from the old times to bundle third party code and especially when this becomes more public, we all will run into conflicts when one extension requires another incompatible version of the same library. But I consider this OK, to support legacy systems at least for a while.

But if you can, you should start setting up your TYPO3 projects with composer. Mid term the TER will also be replaced with something that speaks composer but not "Extension Manager".

Let's look into each of these properties and how (or if at all) you can influence them.

Cookie

You can influence the domain name for which cookies are set, by changing [SYS][cookieDomain] (or [BE][cookieDomain] or [FE][cookieDomain]). This can lead to a situation where the cookie is not set in a way, that the browser does not send the cookie back to TYPO3, or it it interferes with older cookies that have previously been set.

As a general recommendation I suggest to not change these values, as the default (not set) is result in the compatible and most secure.

If you suspect trouble with this setting: clear all cookies, set the value to default, try again.

Session Lifetime

For what period of time a TYPO3 backend session is valid is not only controlled by the lifetime of the cookie, but by a configuration option. The cookie itself always is a session cookie, which means it only expires after closing the browser window. The configuration option is [BE][sessionTimeout] and is set to one hour by default. On development systems i set this to one year, for production systems I leave the default. Frontend sessions have more complex configuration options, which I leave to be discribed in a different article.

IP Address

TYPO3 sessions are also bound to the IP Address of the client, which established the session. By default for backend session all IPV4 segments are respected and need to be the same in subsequent request, while for frontend sessions only the first two segments are respected. You can change the default values with [BE][lockIP] and [FE][lockIP] configuration options. A value of 0 means, that client IP address is completely ignored. It is also possible to disable the IP restrictions for individual backend users, by ticking the "Disable IP lock for user" checkbox in the user record.

Btw, there is no handling for IPV6 Addresses at the moment, which needs to be changed at some point.

User Agent String

That TYPO3 also evaluates the user agent for sessions is probably not widely know. Additionally the configuration options to change the behavior are far from being intuitive. The options are [BE][lockHashKeyWords] for backend and [FE][lockHashKeyWords] for frontend sessions. lockHashKeyWords only accepts two values: empty string and useragent. The latter is the default and means that if the user agent changes, TYPO3 invalidates the session.

The user agent string changes more often than you might think, as some browser plugins influence it and also proxies sometimes touch this HTTP request header.

Summary

If you or your customers experience random or often "logouts" from TYPO3 sessions, most of the time changing IP addresses or changing user agents are the reasons for that. I would suggest to first try to set lockHashKeyWords to an empty value, as it is the weakest protection for sessions and if this does not help, set lockIP to 0 (or disable it for individual users). Since both disable some additional security measurement, I would suggest to look for the reason of changing user agent or IP addresses and fix that, but I would not consider it completely unreasonable to have them disabled completely.

Proper tagging

What you still need to do however is to create a tag for every release. But you do this anyway already, don’t you? There is one additional thing you need to respect to get a better experience and that is that you better create an annotated tag instead of a simple one. This is nicer, because you can use the annotation (which is nothing more than a multi line text like a commit message) to write release notes for this release, which are independent from the commit message from the last commit of this tag.

Creating an annotated tag is easy and git GUIs like SourceTree have built in support for this git feature. On the command line you can just do:

git tag -a 1.4.2 -m "My release notes for version 1.4.2"

This is a great addition to the meta data of your releases and we'll use this annotation lateron when uploading the extension to TER as upload message.

To read the annotation you can use the following git command:

git show 1.4.2

This gives you all the information to the tag including the annotation.

Now that you know how you do releases to TER in the future, let's look at the required travis set up.

Travis configuration explained

It adds an after_script: section to the .travis.yml. Commands in this section are executed after your main CI scripts have run and no matter whether these failed or not. It is also important to know, that these commands are (of course) run for every build in your matrix. This is not exactly what we want. We want the TER upload script to run only when all builds succeed and it needs to be run only once!

Travis CI has an open feature request to implement what we need, but for the time being, we'll need to use an external tool to add the wanted bahavior to Travis. The tool is called travis-after-all and it uses the Travis API to ensure that the upload of the extension code to TER is only executed once and only if all builds in your matrix passed.

Using the tool is as easy as adding one line to your .travis.yml and checking whether the return value is zero (all builds finished successfully).

For the upload to TER, we of course also need user credentials. These are passed as encrypted environment variables, which can be added through the settings of your repository on the travis website:

Although I trust Travis enough to pass the credential to them (which they must be able to decrpyt to get the script running correctly), I created a new typo3.org account for the automated upload only and passed the ownership of the extension key to this account. By doing so, I make sure only an account with the least privileges to get the job done is at risc and not my regular one. I find this risc acceptable, but of course this is up to you to decide.

Summary

Setting this up is as easy as:

Adding the travis configuration

Adding your typo3.org credentials as protected environment variables

Introduction

What I'm going to describe here is not a complete composer setup, which would also be perfectly possible with TYPO3 6.2, but the few steps you need to take to take advantage from composer class loading in your existing non composer TYPO3 6.2. projects. If you already use composer to setup your TYPO3 projects, you may as well want to have a peak into this article to make sure you get the most out of the possibilities.

Setup

A prerequisite for this to work is, that you put your TYPO3 sources in a folder, not a link, and this folder resides on the same level as your index.php like this (I skipped fileadmin, typo3conf, … directories here for better overview):

lrwxr-xr-x 1 19 Sep 28 10:29 index.php -> typo3_src/index.php lrwxr-xr-x 1 15 Sep 28 10:29 typo3 -> typo3_src/typo3 drwxr-xr-x+ 17 578 Sep 28 10:28 typo3_src

Step 1: Add composer.json

Add this composer.json file on the same directory level:

Step: 2: Adapt class loading registration in composer.json

The example composer.json file from the gist comes with class loading information for every TYPO3 core extension, including the class aliases for 4.x compatiblity. What you now need to do is add class loading information for every single extension in your installation. I prepared examples for typo3_console, news and tt_news, which should be self-explanatory. You basically need to add all paths in which the extensions in your installation have classes. Make sure not to forget to specify the relative path to the directory (seen from the composer.json file).

You can even try to only specify "classmap": ["typo3conf/ext"] as a single path, which saves you quite some time and works out in most cases, but can fail in some edge cases.

Step 3: Perform a composer install in this directory

Download composer if not done already, navigate to the directory with the composer.json file and perform a composer install. The result should look like this:

lrwxr-xr-x 1 19 Sep 28 10:29 index.php -> typo3_src/index.php drwxr-xr-x+ 3 102 Sep 28 10:22 Packages -rw-r--r--+ 1 8194 Sep 29 12:52 composer.json -rw-r--r--+ 1 2334 Sep 28 13:16 composer.lock drwxrwsr-x+ 4 136 Sep 28 13:22 fileadmin lrwxr-xr-x 1 19 Sep 28 10:29 index.php -> typo3_src/index.php lrwxr-xr-x 1 15 Sep 28 10:29 typo3 -> typo3_src/typo3 drwxr-xr-x+ 17 578 Sep 28 10:28 typo3_src -rwxr-xr-x+ 1 1279 Sep 28 10:32 typo3cms drwxrwsr-x+ 8 272 Oct 10 17:45 typo3conf drwxrwsr-x+ 24 816 Sep 29 12:48 typo3temp drwxrwsr-x+ 8 272 Sep 28 13:10 uploads

It is mandatory that the Packages folder, which is created during the composer install run is on the same level as the typo3_src folder.

Step 4: Set the TYPO3_COMPOSER_AUTOLOAD environment variable

Add the following to your apache vhost configuration:

SetEnv TYPO3_COMPOSER_AUTOLOAD "On"

or the following in your .htaccess file:

RewriteRule .? - [E= TYPO3_COMPOSER_AUTOLOAD:On]

For nginx, you can set a fcgi param like that:

fastcgi_param TYPO3_COMPOSER_AUTOLOAD On;

Verify that it worked

Remove all class loader cache files in typo3temp/Cache/Data/cache_classes and load the backend and frontend of your website. The website should show up and no cache files should be added to this directory. Do not delete the complete typo3temp/Cache because then, a few class cache entries are written anyway and you cannot confirm or deny whether you did everything correctly.

Optimize

If you do not have every single TYPO3 core extension installed, you should consider removing the lines from the composer.json file which point to core extensions you do not use in the project.

If you do not want to expose the TYPO3 sources directory and the Packages directory and your composer.json file, as they contain quite some information about your installation, you can create a web folder and link the necessary folders and files into it, e.g. like that:

$ ls -l web/ lrwxr-xr-x 1 12 Oct 10 17:43 fileadmin -> ../fileadmin lrwxr-xr-x 1 12 Oct 10 17:44 index.php -> ../index.php lrwxr-xr-x 1 8 Oct 10 17:44 typo3 -> ../typo3 lrwxr-xr-x 1 12 Oct 10 17:45 typo3conf -> ../typo3conf lrwxr-xr-x 1 12 Oct 10 17:44 typo3temp -> ../typo3temp lrwxr-xr-x 1 10 Oct 10 17:44 uploads -> ../uploads

Minimum setup

In this section I describe what you can do to get better class loading suport in TYPO3 7LTS. Everthing you find here, you can safely do without loosing compatibility with older TYPO3 versions!

Compatiblity mode

If you don't want to, you do not have to do anything! Regarding class loading your extension will work with TYPO3 7LTS anyway in 99,9% of the cases. TYPO3 scans the extension directory for class or interface files and registers every single one automatically to ensure backwards compatibility.

Important: If you add classes during development they will not be found by the class loader unless you update your class information files. If you have classes located e.g. in a tests folder which should not be availble in production, you should consider one of the following ways to configure class loading.

Extensions having one folder with classes

Considering you have an Extbase extension (or an extension where all classes and interfaces reside in a Classes folder) you can simply add the following to your ext_emconf.php file:

'autoload' => array( 'classmap' => array('Classes') ),

The folder Classes will be scanned and every class or interface found will automatically be registered, no matter if the classes use PHP namspaces or not.

Important: If you add classes during development they will not be found by the class loader unless you update your class information

Extensions using namespaces (possible since TYPO3 6.0)

TYPO3 introduced a way to load classes using namespaces, which was later standardized by the PHP framework interoperability group as PSR-4. If your extension has namespaced classes, then you can add the following to your ext_emconf.php file:

'autoload' => array( 'psr-4' => array('Vendor\\ExtName\\' => 'Classes') ),

Here you do not only specify a folder, but also a namespace prefix which all classes in this folder must have. The prefix must end with a backslash!

Note: If you do so, newly added classes during development can be resolved automatically, whithout any further action! There is no need to update your class information files!

Legacy extension which previously used an ext_autoload.php file

As outlined before, it is not necessary any more to specify the name and location of every single class. You can specify folders and TYPO3 scans for the classes in them. So for example to provide class information for tt_news, Rupi could add the following to the ext_emconf.php:

'autoload' => array( 'classmap' => array( 'pi', 'mod1', 'lib', 'class.tx_ttnews_compatibility.php', ) ),

Note: As you can see, it is perfectly fine to add multiple folders or even single files to the class map. However, just to stress it again, if you add classes during development they will not be found by the class loader unless you update your class information files manually.

Advanced setup

This section is a collection of further things you can do to tweak your development experience regarding class loading in TYPO3 7LTS.

Class loading information in composer.json

If your extension has a composer.json file already with a psr-4 or classmap autoload section, you are done. No further action is required. The section in the composer.json file can look like this:

"autoload": { "psr-4": { "Vendor\\ExtName\\": "Classes" }, "classmap": ["Resources/Private/PHP/SomeLib"] },

As you can see, you can combine both classmap and psr-4 sections (which is also possible of course in an ext_emconf.php file.

Fixture classes

The old TYPO3 class loader could automatically resolve classes inside a Tests folder, in case the namespace reflected the folder structure below. In TYPO3 7LTS this is not the case any more, which really is a good thing! If you do not want to use composer to execute your extension tests individually like I described in a previous article, but execute them in the context of your TYPO3 project installation, you can add an autoload-dev section in your ext_emconf.php file:

'autoload-dev' => array( 'psr-4' => array('Vendor\\ExtName\\Tests\\' => 'Tests') ),

Note: Just like composer, TYPO3 ignores an autoload-dev section in a composer.json file. So unlike the regular autoload section, the autoload-dev section is only evaluated if present in the ext_emconf.php file.

General notes

Update class loading information

The class loading information is updated every time you activate or deactivate an extension via extension manager. In the rare case you want to update it manually, you can run the following command from the command line:

typo3/cli_dispatch.phpsh extbase \ extension:dumpclassloadinginformation

Composer mode

It is possible to manage the source code for your TYPO3 project exclusively with composer (which I can highly recommend). In that case, all class loading information provided by extensions in the ext_emconf.php files is completely ignored! However as best practice and for the time being it is recommended to nevertheless provide this information in your extensions, so that it can be used in projects which are not set up using composer.

The challenge

We invested a lot of time to improve the situation and introduced some locking, but still a few main conceptual problems remained:

Besides having a registry with class name to file mapping, TYPO3 tried to resolve class files using several (partly conflicting) strategies.

To avoid doing these expensive operations again and again, the information is cached in files, but these files contain absolute paths, which makes the cache unusable if CLI access has different paths than web access. Warming up caches on CLI during deployment is impossible in such situations.

The file location of classes never change in an installation and could be known upfront, but TYPO3 tries to resolve the location of a class dynamically in a regular request. At the same time, it can never be determined whether the information is complete or not, because different code paths, could need different PHP classes that are then looked up dynamically.

In a production system the class information does not change unless an extension is installed. However this information was written into volatile caches, which are often flushed. This lead to repeatedly unnecessary re-calculation of these caches.

The solution

Finding a solution for these issues was brain twisting for me, until I realized that composer and standardization of class autoloading solved all problems for us at once. We only needed to implement composer more deeply into our system. Here is why composer class loading solves all the issues we had:

There is no magic involved in the composer class loader. How classes are loaded must be specified for each package (or in TYPO3 terms: each extension).

By specifying the class loading information, the class names in your extensions, must no longer be related to the extension name and are not bound to a Classes folder.

With this class loading configuration the composer class loader knows how to load every single class during build time. No assumptions, no magic, no additional class information lookup at runtime.

The class information is static. In a production (TYPO3) system, it is just there and never changes unless a package (extension) is added or removed. No failures due to broken caches during deployment, as there are no caches any more.

Class information configuration is simple. Just a few lines instead of long ext_autoload.php files.

Class information is stored with relative paths. So it can be generated on one system and deployed to another one without issues.

The implementation

We completely removed all TYPO3 class loading code in TYPO3 7 and now rely on the composer class loader to load all classes of TYPO3 and every installed extension. If you set up your TYPO3 installation (including extensions) with composer (not downloading a package and symlinking the source like in the old days), TYPO3 code does nothing regarding class loading, composer takes over that task completely.

Non composer mode

In case you want to set up your TYPO3 project traditionally without using composer, TYPO3 ships with the composer class loader and all necessary information to load all core classes, but additionally executes a very slim layer of code to detect and register classes of active extensions, so that the composer class loader can load them as well. If an extension does not provide class loading information, the complete extension directory is scanned for class files and every class found, will automatically be registered for the class loader. The ext_autoload.php file is completely ignored in TYPO3 7.

There might be some cases where a full scan of all directories is not desired, e.g. when third party libraries are shipped or test classes are provided with the extension. Because of that, extension authors can provide class loading information which will be evaluated by TYPO3. As of TYPO3 7.5 this information can be provided in an autoload section of the ext_emconf.php file, or in a composer.json file.

In contrast to composer, TYPO3 only evaluates the psr-4 and classmap sections. But we think this is more than enough and even easier to handle than before.

Benefits

We do not have a classes cache any more. So this cache cannot be corrupt, will never be flushed (and thus being unavailable) and must not be build during runtime. Besides that, we have a lot less code. The results are: 

Better performance

Higher reliability

Higher flexibility

Less maintenance costs

Implications

We do not generate class loading information during runtime any more, but change the class loading information every time an extension is added or removed from the installation. If you use composer, it takes over that task and updates the class loading information on every composer install or composer update. If you do not use composer, TYPO3 will update the class loading information every time you add or remove an extension with the extension manager or via command line (typo3/cli_dispatch.phpsh extbase extension:install realurl) or if you explicitly trigger TYPO3 to re-build the information from the command line: typo3/cli_dispatch.phpsh extbase extension:dumpclassloadinginformation

If you develop extensions and add classes, it is therefore highly recommended to follow the PSR-4 specification and add a psr-4 section to your ext_emconf.php or composer.json of your extension.

Combined TYPO3 6.2 and TYPO3 7 compatiblity

Since ext_autoload.php files are ignored in TYPO3 7 and an autoload section in ext_emconf.php is ignored in TYPO3 6.2, it is pretty easy to provide compatibility for both TYPO3 versions with your extension. Just add both, and your extension will work in both TYPO3 versions. If you follow the PSR-4 standard, you of course only need to add the according section in ext_emconf.php. Of course your classes and interfaces must reside in a Classes folder, as TYPO3 6.2 only looks up classes in that folder. My scaffold extension for example is compatible with both TYPO3 versions in that regard.

Conclusion

The listed benefits should be enough to justify this change. With this step we also step closer to the broader PHP community by using composer. It is something new to learn for TYPO3 community members, but we are convinced this is for the benefit of our product and both the TYPO3 and PHP community.

During the TYPO3 Developer days this year, I talked to Thomas Löffler about his vision regarding the future of the TER. He had the idea that extension authors should not need upload a package for each extension version, but register a repository URL which is then evaluated regarding its tags. So pushing a new tag would mean releasing a new extension version. This is nothing new for everybody who knows packagist and composer, but for our grown infrastructure this could be a challenge.

A few days later this topic popped up for me very often and eventually I got in touch with Alexander Schnitzler. Together with him (and in cooperation with Thomas) we figured out a first step into the future of TER.

We agreed on the following:

Extension authors can set a repository URL to their version control repository when editing the extension meta data on typo3.org.

This looks like this:

If this is done, we scan this repository and publish it on composer.typo3.org automatically.

Before you do so now for your extension, please read the requirements that need to be fulfilled for that to work:

Requirements

The repository mus contain tags names which are version numbers, so that they can be identified as release

Tags and branches must contain a composer.json file. Having an ext_emconf.php file only is currently not enough!

If these requirements are not fulfilled, or better, if none of your tags or branches have a composer.json, your extension will not get an additional listing from this repository URL. It is perfectly fine however, if at least one branch or tag contains a composer.json file. If the requirements are fulfilled, the extension shows up on composer.typo3.org with the composer name you defined in your composer.json. This means that e.g. my typo3_console extension is now available as helhum/typo3-console as well as typo3-ter/typo3-console. The latter stays for compatiblity, the first should be used for more control and better composer support (e.g. more precise class loading information).

Important! This is not the complete!

Adding the URL as described above does not (yet) upload your extension to the TER! It only provides it for composer users. If you want it to be available through the TYPO3 extension manager, you must still additionally upload the package for each version. Tagging alone is not enough. But we have ideas already how we can close this gap to a brighter future of our TYPO3 extension repository (TER).

One more thing

Since we use Satis to scan the repositories and build a final composer repository, there is now the possibility to list all registered packages and even search for them.

Benefits over registering on packagist

There are not much benefits you get if you register your extension repo on typo3.org. One is, that we poll the repositories, so there is no hook required in your repository to update th versions on composer.typo3.org. You trade this in however for the need to specify the additional http://composer.typo3.org repository. In the end it is up to you what you prefer as extension author.

Credits

Alexander Schnitzler did most of the programming (typo3.org and composer.typo3.org), Thomas Löffler helped with reviews for the typo3.org stuff and I helped conceptually, with reviews, a little code and the final deployment. Cheers as well to the server team for their help!

We hope you like it. If so, let us know!

I'd like to add one to this list, which is my helhum/t3satis package. Satis is a composer package repository generator, which you can use from the command line. Basically you create a satis.json file with all your (private) repositories listed in there and use the command line tool to build a little static website, which can act as composer repository like packagist.

However repositories added to the satis.json must contain a composer.json in the tags or branches. This very often is not the case when you have legacy TYPO3 extensions in repositories. To be able to work around that limitation, I created t3satis.

If you create your own composer repository for your legacy extension with t3satiss, there is no need any more for a huge list of package repositories in your root composer.json file. Just add one composer repo with the URL to your t3satis website and you are done.

If you know satis, this should work immediately for you. Just look at the example that is provided in the readme