tl;dr: composer.typo3.org remains for the time being, but extension developers are encouraged to publish their extensions on Packagist
Introduction
To get things straight first: composer.typo3.org is a service to support legacy TYPO3 extensions, which are not composer enabled (e.g. not having a proper composer.json file). The short term goal here is, that this custom composer repository becomes obsolete and all extensions published to TER are also available on Packagist.
To achieve this goal, we highly encourage extension maintainers to add a proper composer.json file to their extensions and publish their extension on Packagist as well. Only a few setup steps are required, which need to be done once. Unlike TER upload, Packagist will automatically detect new releases by analyzing git tags that look like a version number.
Improvements
In the past many extension developers already published their extensions to Packagist. If that is the case it is recommended to require these packages over the ones offered via composer.typo3.org (e.g. georgringer/news instead of typo3-ter/news).
It is recommended, because as a legacy service, composer.typo3.org does not expose all features extension developlers can use. Most notably, requirements to third party composer packages are ignored. Also class aliases are ignored.
However, to check whether an extension is also available on Packagist, involved quite some boring steps and an additional lookup - until now.
Tell users your extension is available on Packagist
To improve this situation, extension owners that have their extension published on Packagist, can now conveniently communicate this to their users, by just confirming a checkbox and adding the composer name of their extension directly on extensions.typo3.org.
Just log in and navigate to your extensions:
Then open settings for the extension by pressing “Edit”, tick the checkbox to confirm the extension is published, enter the composer name and press save.
If you do so, this information is automatically propagated to composer.typo3.org, where a nice red warning box is show, that requiring this extension through the legacy service is deprecated:
Additionally a warning message appears as well on command line, when requiring this extension through our legacy service:
Last but not least, this information will be presented directly on extensions.typo3.org (this feature is not ready yet, but will arrive soon).
Further improvements
Satis has been upgraded to the latest stable version
Abandoned extensions are shown as abandoned (in the same way, but without the notice which composer name to use instead)
All extensions have a link to their detail view on extensions.typo3.org included
Credits
Thanks a lot Thomas Löffler for your constant work on improving extensions.typo3.org and your help with reviewing, merging and deploying this feature.
Thanks a bunch mehrwert for sponsoring this feature.
Anya is live and ready to show you everything. Watch her strip, dance, and perform exclusive shows just for you. Interact in real-time and make your fantasies come true.
✓ Live Streaming✓ Interactive Chat✓ Private Shows✓ HD Quality
Anya is LIVE right now
FREE
Free to watch • No registration required • HD streaming
tl;dr: Doing TYPO3 Upgrades from command line is more reliable, faster and automatable.
Introduction
With TYPO3 Console it has become possible to automate a lot of TYPO3 related tasks. It is the only way to automate a TYPO3 installation and is essential for any deployment stategy, including deploying to "the cloud".
What was still missing though, was to automatate a TYPO3 upgrade (e.g from 6.2 to 7.6). To accomplish such a task one would still have to go to the install tool and execute every single upgrade step manually. Doing so not only is time consuming. It also is error prone, especially with large TYPO3 installations with large data sets, where it can happen that some upgrade wizards run into the configured PHP request timeout.
Challenge
While I always wanted to have such a feature and I knew others were interested in it as well or even created some code already, I didn't have the time to work on a solution that was good enough to be implemented into TYPO3 Console.
The existing code by Christin Zenker worked for older TYPO3 installations, starting with TYPO3 7.6 the install tool code changed in a way, that it was even more coupled with a web request. This means the old code stopped working and (re-) implementing it was far from trivial.
Not only that the current install tool code expects $_GET to be filled, all user interaction was streamlined for the web, upgrade wizard IDs were changed to the fully qualified (namspaced) class names, which is a bit vebose and clunky to type on the command line and and most output from the wizards contain HTML, which makes it ugly or unreadable on cli.
Last but not least the install tool upgrade tool got more features like being able to re-run upgrade wizards, although they have been marked as executed already.
Two things were clear to me.
Having a good user experience from command line for this task and implementing a complete feature set, would require quite some conceptual work besides the required work for the implementation.
It was nothing I could accomplish im my (limited) free time.
Challenge accepted
A few months ago I was contacted by Mittwald. They asked if I could implement this feature, if they sponsor it. Only a few weeks later I was at a TYPO3 code sprint at Systime, where I talked to Kasper Ligaard about this topic. Systime also was very interested in this feature, because they were about to upgrade literally hunderds of TYPO3 installations from 7.6 to 8.7 and therefore offered to sponsor my work on that as well.
With such generous sponsoring I was able to prioritize this task, postpone other work and start with working on it a few weeks later.
Result
I quickly had some code ready to be able to execute a single upgrade wizard from the command line, but was a very small portion of what needed to be done.
It took me another three weeks and two TYPO3 Console releases (4.5 and 4.6) to get the whole feature set:
List all upgrade wizards with a short identifier
List wizards that need execution
Execute a single wizard using its short identifier
Execute multiple wizards by using their short identifier
Force execution of a single wizard, so that it is executed even when marked as done
Pass arguments to a single wizard, so that wizards that expect user interaction can be executed
Pass arguments for multiple wizards
Execute all wizards that are scheduled for execution, making it possible to perform a complete TYPO3 migration without user interaction
Check compatiblity of all third party extensions with a given TYPO3 version
Automatically disable third party extensions, that have broken (incompatible) code in ext_tables.php or ext_localconf.php
A progress bar when executing all wizards, so that progress for long running upgrades is made visible
Every action is covered with functional tests
Profit
It will change the way TYPO3 upgrades are done in the future.
With these features now present in TYPO3 Console, I personally will never do TYPO3 upgrades using the install tool any more. Given the success of TYPO3 Console, I predict that others will jump on that wagon, too. This feature will change the way TYPO3 upgrades are done in the future. Here is why:
Upgrades are done by executing a single command typo3cms upgrade:all, in contrast of logging into the Install Tool and manually clicking your way thorugh the UI.
Never run into timeouts, even on large TYPO3 installations.
It is possible to make code changes on a local development system, test the upgrade and finally deploy to live. This works smoothly because typo3cms upgrade:all not only can be easily added to your deploy chain, it also is idempotent. It does nothing, if upgrade has been performed already.
Thanks
Thank you very much Mittwald and Systime for sponsoring this great feature. It helps to make TYPO3 (with TYPO3 Console) a better product.
tl;dr: Don't try to forsee the future, but add TYPO3 composer dependencies wisely
In my last article "composer.json specification for TYPO3 Extensions" I already specified in a few words, how the require section for extensions should look like.
Let's dissect some of the statements a bit further and add some more explanation and context to it.
The require property MUST be provided and it MUST contain the requirement to TYPO3 with a version constraint.
It should be very easy to understand, that TYPO3 extensions only work with TYPO3 and thus TYPO3 needs to be added as dependency to the composer.json of extensions. But let me elaborate what I recommend as concrete version constraint.
If your extension is compatible with the two latest LTS releases, the version constraint is pretty straightforward: ^6.2 || ^7.6
If your extension relies on a certain bugfix of a certain bugfix version, you can specify that minor release easily: ^6.2.13 || ^7.6.11
Things get a little bit more tricky, if your extension supports current TYPO3 sprint releases. It is important to know that minor versions bumps of TYPO3 sprint releases may contain breaking changes. If your extension is compatible with version 8.0.0, it does not necessarily mean it is compatible with 8.1.0 or 8.2.0.
If you now specify a version constraint of ^8.0.0 or < 8.99.99 at the time when version 8.0.0 is released, then you declare that this exact same version, this exact same code base, will be compatible with all future sprint releases, which in fact you can not, unless you can forsee the future.
If you nevertheless use such a loose version constraint, you are making a bet. If you are lucky, nothing bad happens and your code magically works with a future sprint release. If you are not so lucky, than your incompatible extension version might be installed with the latest TYPO3 sprint release and might be the reason, why a complete TYPO3 installation is broken.
I suggest not to bet, but specify exactly with which sprint release version your extension is compatible: ^7.6 || >=8.3.0 <8.6 would be a perfect version constraint, when 8.5.0 has been released and your extension version is compatible with that version, but needs at least 8.3.0. If 8.6.0 is released and your code still works, just bump the version constraint with one commit and release a new version that is compatible with the latest TYPO3 sprint release. If your code does not work, you made sure, broken code is not installed.
It SHOULD contain at least a requirement to typo3/cms-core. It SHOULD NOT contain a requirement to typo3/cms directly to avoid confusions as extensions cannot be used as root package with TYPO3.
Now what does this mean? The package typo3/cms-core isn't available on Packagist, so why would I want to add a dependency to it? Well, here is the deal: typo3/cms is available on Packagist and declares to be a replacement for typo3/cms-core and all other system extensions of the current TYPO3 core (typo3/cms-backend, typo3/cms-info, …). Currently if you specify typo3/cms-core, your extension composer.json file cannot act as a root package in the sense that you cannot go to the extension directory and simply type composer install. If you do so, you'll get an error message. This is important, because TYPO3 cannot deal with extensions that are not in the typo3conf/ext directory anway.
You may want to install and test multiple TYPO3 version in your CI setup. But even then I find it useful to not specify typo3/cms directly in the extension composer.json and do composer install during CI, because then composer will always only install the latest TYPO3 version your extension is compatible with.
So you would end up doing composer require typo3/cms $TYPO3_VERSION anyway, which would overwrite any previously specified dependency to typo3/cms, but will add a depedency to it alongside letting composer check whether the newly required TYPO3 version is actually compatible with what the extension specifies. Believe me or, this setup caught quite some errors for me and some other people I know. It is not a strict requirement to use typo3/cms-core instead of typo3/cms, but it gives enough benefits to recommend it.
It COULD contain requirements to other individual TYPO3 system extensions, to properly specify which TYPO3 extensions this extension depends on. This is useful especially in combination with TYPO3 Console, but also provides information, which will be useful in the future.
What I mean with that is, that individual core extensions might be split out of the big TYPO3 repository, making it possible to install only these TYPO3 system extensions, that are strictly required for TYPO3 or your project. For this to work properly, extension (or composer package) authors would need to start specifying which exact TYPO3 system extensions their code depends on. I did this for TYPO3 console for example. Doing so now comes with no downside and no side effect. It only enables better compatiblity for the future and allows TYPO3 Console to automatically detect which TYPO3 system extensions need to be active in a project.
Hope this clears some things up. Happy composer.json writing!
Today I read a statement on twitter, which I don't agree with. A more detailed article followed shortly after that.
In another post I will explain what I found to be an inappropriate tone, but first I need to nullify the arguments given in Alex' article.
Please note, that I don't want to sell Surf to you. Neither do I ask to trust me. I just ask you to look carefully whether the reasons given for this or that are logical and make sense. I let you decide whether this is the case in my or the other article.
Disclaimer
trigger warning: The tone of this post can be preceived to be pretty harsh. It can also contain traces of sarcasm and is partly at least a bit polemic. However I tried my best to argue as logically as possible why I disagree. Most likely it also contains a lot of typos, because I wrote this in kind of a haste.
Agreement
There is one thing I agree with and I want to mention it first. I agree that Surf isn't a particulary good product name, not only because SEO for such a word would be incredible hard. The full name TYPO3 Surf isn't much better either. OK, that's it.
Disagreement
The name
And here the disagreement starts. I wouldn't say TYPO3 is a particulary good product name as well. But here we are with it being a solid CMS, which generates a lot of revenue for a lot of people (mainly in central Europe). And I like it. And I like Surf as well. The name does not make either a bad product.
NIH
The author claims that Surf is a "NIH-Product", but admits that it was invented before other PHP deployment tools existed. That is a contradiction. If it was inveted before any other likewise products, then NIH cannot be the problem here. It is just used as a pejorative rhetorical figure.
I also don't quite get how the product name has something to do with NIH, but it is listed as one reason why NIH is harful.
And how some technical state of the past (Surf needs Flow) is related to that topic also is not immediately clear to me. To state false facts (Surf could only deploy Flow applications) back then, does not help here.
But maybe Surf is "too late to be an option for the whole PHP world"? Well, which project (except composer) is? Is this a requirement for a project to not count as NIH? If that would be the case, a lot of useful projects wouldn't exist by now. In fact when starting TYPO3 Console, I was asked many times why anybody would need such a thing, as the extension coreapi already existed. Back then I mainly had my personal reasons, now it enables TYPO3 to be deployed in the cloud and thus helps to leverage the complete TYPO3 ecosystem to a new level.
I'm not trying to say that reeinventing the wheel is something to strive for in general. But I'm sure that giving up on something because it is potentially NIH, harms innovation and leaves our world poorer than it could be.
Stability and reliability
The author suggests Surf being unstable and unreliable because there are only a few people (he claims it is only me) are maintianing it. Especailly because
Deployment software is not just something you reinvent or which you want to allow to fail occasionally. You want to have a fast and reliable tool, that updates the live version of your website with the least impact for the user.
thus suggests to use Capistrano.
The thing is, a deployment software itself is fairly simplistic, as long as the concept is mature enough, because in the end a deployment is just a list of tasks that need to be executed. That is why any tool that can run a list of (depending) tasks is able to deploy an application to a server. The complexity kicks in with the tasks themselves. What is needed when? What needs to be executed in which order? What is the state of my application if certain steps fail?
This complexity isn't tackled by Capistrano (or any other deployment tool). It does not support you with it. It just runs the tasks you tell it to run.
It is partly solved for TYPO3 CMS with dkddeploy, which uses capistrano, and is partly solved with TYPO3 Surf for TYPO3, as it also comes with pre-configured tasks to deploy this CMS in a reliable way.
Last but not least. The concept for executing tasks (bundled in a workflow and set up in an application) is a solid one. It is taken from the matured Capistrano project and implemented into Surf. This code hasn't been changed for long, not because it is not maintained, but because it is mature.
TYPO3 Surf a posterchild of failure?
The author puts Surf in one line with Neos, Flow and Fluid. Like Neos and Flow, Surf is a failure because it was developed by people who wanted to change the world to the better. Clearly they failed because 2016 happend. Right? I don't know. Even if I would agree with the statement that Neos and Flow failed (which I clearly do not), why on earth should this affect the adoption of a completely separate tool with a completely different scope? I mean, why should Fluid have a chance to succeed, since it is most appearent to be NIH back then. Even now, with all the other templating engines have incredibly much higher adoption rates within the PHP community. Taking these arguments, we would need to switch to Twig and drop Fluid immediately. It is not dropped because people still think it can make a difference at some point in the future.
Can Surf make a difference? It might, it might not. I don't know. I told you anway to not trust me. But don't trust others who claim to know what the future will bring and use what you are comfortable with.
My Conclusion
If you broaden your view from not just looking at some tool, that runs tasks in some order, but at the mission to reliably deploy your website, then suddenly other things start to matter.
The Surf deployment definition (basically a description what tasks need to be executed in which order) for TYPO3 CMS was the first one publicly available. It was the first turnkey solution for automated TYPO3 CMS deployments which was open sourced (by me), I'm aware of. Back then Surf was based on Flow and maintained by others, yet it was the only tool that was appearent to me.
I chose Surf because it did the job and I knew even back then, that the important part is the deployment definition, not the tool. Surf was basically useless for me before I started experimenting with automated deployments for TYPO3 CMS. It ran some tasks that left the application half ready. I knew that bigger agencies had their solutions, but they didn't share anything back then. So yes, I created deployment definitions myself knowing other solution existed (hello other good reason for NIH). Some people I asked thankfully helped me a bit in the beginning. Every single bit I learned and created, I shared.
I shared, so that other people could feel the relief I felt when my first automated deployment worked and I still feel a bit every time I now do a deployment. And it worked. I made people happy with my knowledge and this tool.
And this is what matters. The message I wanted to and still want to spread:
Burn your ftp clients
Shutdown the extension manager
Don't have any manual step in your deployment chain
The TYPO3 community evolved and I meanwhile see people using all kind of tools, like Ansible, Capistrano, GiltabCI, Jenkins, what have you. But still very few are sharing their deployment solutions.
Surf had its share in pushing this evolution, both for me personally but also for others because it targeted TYPO3 products only regarding the delivered deployment definitions. It is (currently) better suited to deploy TYPO3 CMS than any other publicly available tool (except dkddeploy), because it delivers a solution for the complicated parts.
Before considering writing posts that conclude in Surf being a "dead horse", please consider to publish your depolyment definitions (regardless which tool you use) instead. This is what helps to drive the community forward.
Oh, and I like Surf and think it is great and can reliably deploy your (TYPO3) website(s) if you let it.
EDIT (13.08.2016, 22:30h): Added hints about "version" property and amended the "require" property description.
tl;dr: On top of general requirements for composer packages, TYPO3 extensions need some additional information in a composer.json file to function properly.
How to read this specification
I have created an example with a minimal setup for a TYPO3 extension. In the following I will describe each property in the composer.json using the modifiers MUST, SHOULD, COULD. MUST means that it will break hard without specifying that configuration property or if specifying a wrong value. SHOULD means that not providing that configuration property might lead to faulty or inconsistent and confusing behaviour. COULD means that this is completely optional, but a nice addition and therefore recommended.
The following rules apply in case the extensions are released publicly. For private extensions the requirements are less strict, but they SHOULD be followed nevertheless.
Configuration properties
"name"
The name property MUST be provided and it MUST have a vendor name and a package name (separated by a slash /). Vendor name and package name MUST only contain lowercase alphanumerical characters, dashes or underscores (regex of the complete name: [a-z0-9]([_.-]?[a-z0-9]+)*/[a-z0-9]([_.-]?[a-z0-9]+)*).
The vendor name MUST not already be registered on Packagist. If it is registered, you MUST be owner of this vendor name on Packagist. If it is not registered you MUST register at least one package with this vendor name on Packagist.
"description"
The description property is one line describing the extrension and MUST be provided.
"keywords"
The keywords property COULD be added to tag your extension with some keywords.
"homepage"
The homepage property COULD be added to provide an URL to the homepage of the extension.
"authors"
The authors property SHOULD be added to provide information who created the extension.
"license"
The license property SHOULD be added to specify the license under which the extension is published
"type"
The type property MUST be provided and it MUST exactly have the value typo3-cms-extension
"version"
The version property SHOULD NOT be provided, if you are using TYPO3 version 7.6 and above. However for TYPO3 version 6.2 it SHOULD be provided to avoid issues during extension activation in TYPO3 extension manager.
"require"
The require property MUST be provided and it MUST contain the requirement to TYPO3 with a version constraint. It SHOULD contain at least a requirement to typo3/cms-core. It SHOULD NOT contain a requirement to typo3/cms directly to avoid confusions as extensions cannot be used as root package with TYPO3. It COULD contain requirements to other individual TYPO3 system extensions, to properly specify which TYPO3 extensions this extension depends on. This is useful especially in combination with TYPO3 Console, but also provides information, which will be useful in the future.
For TYPO3 version 6.2, the require section MUST NOT contain any other dependencies than the TYPO3 version.
For TYPO3 version 7.6, the require section SHOULD NOT contain any dependencies than the TYPO3 version. If composer packages are added, then non composer usage of the extension SHOULD be considered. If TYPO3 extensions are added, then the composer name from Packagist SHOULD be used. If the dependent extension is only available in TER, then the TER composer name SHOULD be used.
"replace"
The replace property MUST be provided if the composer package name part does not match the extension name and is highly recommended thus SHOULD be provided in any case. If it is provided, the first replace value MUST match exactly the extension key and the second replace value SHOULD match the TER package key (extension key prefixed with "typo3-ter/" and with underscores (_) replaced with dashes (-). The version constraint MUST exactly be self.version
"autoload"
The autoload property MUST be provided if the extension provides PHP classes. The autoload definition MUST be psr-4 or classmap and MUST NOT contain some other autoload type. The folder name for psr-4 is recommended to be Classes/, but can be any other folder as well. The PHP namespace prefix for psr-4 MUST only have two parts, where the first MUST be your vendor name in upper camel case and the second MUST be your extension name in upper camel case. While the latter sounds very restrictive, it is essential especially when using Extbase, as the framework very often makes assumptions on the extension key derived from the namespace prefix.
Tools support
Currently there is no tool support, so you have to check the above specification manually. I would find it useful to have two things. One tool, that supports you with creating a composer.json for your TYPO3 extension. A second tool could validate existing composer.json files against the specification above. If you are willing to contribute such functionality, get in touch with me and I will mentor this effort. Thanks in advance.
Spread the word!
Please check your own extensions and fix the composer.json files accordingly and help others to fix their composer.json files. It'll help to improve the TYPO3 extension ecosystem.
Anya is live and ready to show you everything. Watch her strip, dance, and perform exclusive shows just for you. Interact in real-time and make your fantasies come true.
✓ Live Streaming✓ Interactive Chat✓ Private Shows✓ HD Quality
Anya is LIVE right now
FREE
Free to watch • No registration required • HD streaming
While some solutions for autocompletion of TYPO3 console commands existed, I never could make them work on my shell (bash).
Now that I learned from the great package bamarni/symfony-console-autocomplete, I adopted that code to make it generate autocompletion scripts for TYPO3 console for both bash and zsh shell.
From TYPO3 console 3.3.0 and above, you can generate such script with the command: ./typo3cms help:autocomplete bash. This script must be installed in the fitting location. In my case (I use bash and shell completion from mac ports): ./typo3cms help:autocomplete bash tt | sudo tee /opt/local/etc/bash_completion.d/typo3cms
tl;dr: We would be better off fully embracing composer in the TYPO3 ecosystem. You should do so for your own projects.
If you are not interested in a little bit of history, you can safely skip the following chapter.
History
This is actually a pretty good question, but also a pretty old one. Interestingly this is nowadays asked much more frequently than a few years ago. And it is always asked by mentioning composer in some way.
Like: "I know how to include this PHP library if I would use composer, but how do get this to work for my TER extension?"
Or: "I would like to use this PHP library in my extension, but it must also work for people not using composer?"
I can only speculate, but I assume the reason that this question pops up so often now is, that composer has already transformed the way people in the PHP world think about code.
The years B.C.
In the years B.C. (before composer), PHP code was handled differently. It was randomly copied from forums, publicly available code repositories or at best taken from PEAR and then burried into an extension and most of the time kept there untouched for ages. Updates or bugfixes in libraries weren't handled most of the time and sometimes the library code was changed arbitrarily for the needs of the extension, making updates of the library code close to impossible. But it worked well (enough). It worked well enough because nobody knew a better way to handle it.
The age of composer
Then some crazy people decided to change the PHP world and created a set of tools and kickstarted an ecosystem which since then thrives. You need a PHP library in your composer managed project? Go find it on Packagist, type composer require name-of-vendor/name-of-package in your project root and be done. Using the library code just works. Updating it works simply by typing composer update name-of-vendor/name-of-package. If the library needs another library, or anything needs a special PHP version, or another library needs another version of a library you intend to use, composer will resolve everything for you just nicely (most of the times).
The TYPO3 Extension Repository (TER)
The TER exists for years and it an integral part of the ecosystem that made TYPO3 thrive. However technically it is a bit aged. The TER and the TYPO3 Extension manager can handle updates of TYPO3 versions and even extensions. It can also handle dependencies to other extensions. However resolving these dependencies is less stable and capable than what composer does. Most importantly it can only handle dependencies from the TYPO3 ecosystem, not from the composer ecosystem. This means, if you want to publish an extension to TER, that needs code from a composer library, you need to take care and include it into your TYPO3 extension, like in the old B.C. days.
Bundling libraries in phar files
There is a project, which you can use to bundle any composer based library, or even a set of libraries as single phar file. I find it much nicer to ship one file with dependent third party code, instead of hundreds of files.
Benefits
More secure: The complete phar might be exposed but not single files.
Cleaner: It is much more clear which is a library and which is your code.
More reliable: Code scans or code formatters or linters in your CI environment will ignore phar files by default
Smaller: The phar files are smaller in size than the sum of all files
And probably more, but these are enough to convince me.
Howto
Install the tool globally and make sure $HOME/.composer/vendor/bin is set in your path:
composer global require clue/phar-composer
Create a directory in your extension and create the phar archive (symfony/process for example):
In your extension code, before you want to use the libraries in question, include the autoloader file of your phar:
use TYPO3\CMS\Core\Utility\ExtensionManagementUtility @include 'phar://' . ExtensionManagementUtility::extPath('ext-key') . 'Libraries/symfony-process.phar/vendor/autoload.php';
Optimization
There are several things I have done to optimize this in my TYPO3 Console package:
I excluded the phar from the packages delivered by composer. In that case the phar is not needed.
I recreate the lib with the current version on every travis build, which means the TER upload will always include the current version
I set classmap-authoritative to true and prepend-autoloader to false for the phar class loader. That way I can just include the phar if present and the autoloader of the phar is only used when the class is not loadable.
Summary
These files do not replace a proper dependency management. It still is a relic from the old times to bundle third party code and especially when this becomes more public, we all will run into conflicts when one extension requires another incompatible version of the same library. But I consider this OK, to support legacy systems at least for a while.
But if you can, you should start setting up your TYPO3 projects with composer. Mid term the TER will also be replaced with something that speaks composer but not "Extension Manager".
TYPO3 evaluates and validates four client side values to re-constitute a session:
be_typo_user (or fe_typo_user for frontend) Cookie
Time since last request of a TYPO3 page
IP Address
Browser user agent string
This means, if any of these values change, the session will become invalid. As a result, users are "logged out", or sometimes cannot log in at all.
Let's look into each of these properties and how (or if at all) you can influence them.
Cookie
You can influence the domain name for which cookies are set, by changing [SYS][cookieDomain] (or [BE][cookieDomain] or [FE][cookieDomain]). This can lead to a situation where the cookie is not set in a way, that the browser does not send the cookie back to TYPO3, or it it interferes with older cookies that have previously been set.
As a general recommendation I suggest to not change these values, as the default (not set) is result in the compatible and most secure.
If you suspect trouble with this setting: clear all cookies, set the value to default, try again.
Session Lifetime
For what period of time a TYPO3 backend session is valid is not only controlled by the lifetime of the cookie, but by a configuration option. The cookie itself always is a session cookie, which means it only expires after closing the browser window. The configuration option is [BE][sessionTimeout] and is set to one hour by default. On development systems i set this to one year, for production systems I leave the default. Frontend sessions have more complex configuration options, which I leave to be discribed in a different article.
IP Address
TYPO3 sessions are also bound to the IP Address of the client, which established the session. By default for backend session all IPV4 segments are respected and need to be the same in subsequent request, while for frontend sessions only the first two segments are respected. You can change the default values with [BE][lockIP] and [FE][lockIP] configuration options. A value of 0 means, that client IP address is completely ignored. It is also possible to disable the IP restrictions for individual backend users, by ticking the "Disable IP lock for user" checkbox in the user record.
Btw, there is no handling for IPV6 Addresses at the moment, which needs to be changed at some point.
User Agent String
That TYPO3 also evaluates the user agent for sessions is probably not widely know. Additionally the configuration options to change the behavior are far from being intuitive. The options are [BE][lockHashKeyWords] for backend and [FE][lockHashKeyWords] for frontend sessions. lockHashKeyWords only accepts two values: empty string and useragent. The latter is the default and means that if the user agent changes, TYPO3 invalidates the session.
The user agent string changes more often than you might think, as some browser plugins influence it and also proxies sometimes touch this HTTP request header.
Summary
If you or your customers experience random or often "logouts" from TYPO3 sessions, most of the time changing IP addresses or changing user agents are the reasons for that. I would suggest to first try to set lockHashKeyWords to an empty value, as it is the weakest protection for sessions and if this does not help, set lockIP to 0 (or disable it for individual users). Since both disable some additional security measurement, I would suggest to look for the reason of changing user agent or IP addresses and fix that, but I would not consider it completely unreasonable to have them disabled completely.
What you need to know about PHP opcode cache clearing
tl;dr: You need to execute the code that clears caches in the context you want to have them cleared.
I recently got a pull request for typo3 console that added a command to clear opcode caches. This command called the API TYPO3 provides in the command line context. Unfortunately clearing those caches is not that easy.
Calling that API only clears the caches of the currently running PHP process. When calling the clear caches code from command line, then only the opcode cache for the PHP process spawned from the command line for that particular user is flushed, NOT the cache for the PHP process that delivers your website.
Conclusion
Clearing PHP opcode caches heavily depends on your (hosting) environment and your workflow. We can recommend tools, to mitigate the issue during deployment, but we can never provide a generic fix for this specific infrastructure issue in our applications on runtime.
My advice would be: Upgrade to a fast enough PHP version (7.x) and your website will still be fast enough for small to medium range websites, or invest the time to fine tune your OPCache configuration and an (automated) deployment that takes care to properly clear the caches for larger scale websites.
In any case disable opcode caches during development!
Automatically upload TYPO3 extensions to TER with Travis CI
tl;dr: TER releases as easy as creating a git tag. Thanks to Mathias Brodala for helping me to get this going.
In a previous article I described how to setup continuous integration services with Travis CI for your TYPO3 extensions that are hosted on Github.
If you’ve come so far and have an extension, that should also be released in the TYPO3 Extension Repository (TER), this tutorial explains how to set up everything, so you never have to do the upload to TER manually again.
Proper tagging
What you still need to do however is to create a tag for every release. But you do this anyway already, don’t you? There is one additional thing you need to respect to get a better experience and that is that you better create an annotated tag instead of a simple one. This is nicer, because you can use the annotation (which is nothing more than a multi line text like a commit message) to write release notes for this release, which are independent from the commit message from the last commit of this tag.
Creating an annotated tag is easy and git GUIs like SourceTree have built in support for this git feature. On the command line you can just do:
git tag -a 1.4.2 -m "My release notes for version 1.4.2"
This is a great addition to the meta data of your releases and we'll use this annotation lateron when uploading the extension to TER as upload message.
To read the annotation you can use the following git command:
git show 1.4.2
This gives you all the information to the tag including the annotation.
Now that you know how you do releases to TER in the future, let's look at the required travis set up.
Travis configuration explained
It adds an after_script: section to the .travis.yml. Commands in this section are executed after your main CI scripts have run and no matter whether these failed or not. It is also important to know, that these commands are (of course) run for every build in your matrix. This is not exactly what we want. We want the TER upload script to run only when all builds succeed and it needs to be run only once!
Travis CI has an open feature request to implement what we need, but for the time being, we'll need to use an external tool to add the wanted bahavior to Travis. The tool is called travis-after-all and it uses the Travis API to ensure that the upload of the extension code to TER is only executed once and only if all builds in your matrix passed.
Using the tool is as easy as adding one line to your .travis.yml and checking whether the return value is zero (all builds finished successfully).
For the upload to TER, we of course also need user credentials. These are passed as encrypted environment variables, which can be added through the settings of your repository on the travis website:
Although I trust Travis enough to pass the credential to them (which they must be able to decrpyt to get the script running correctly), I created a new typo3.org account for the automated upload only and passed the ownership of the extension key to this account. By doing so, I make sure only an account with the least privileges to get the job done is at risc and not my regular one. I find this risc acceptable, but of course this is up to you to decide.
Summary
Setting this up is as easy as:
Adding the travis configuration
Adding your typo3.org credentials as protected environment variables
Anya is live and ready to show you everything. Watch her strip, dance, and perform exclusive shows just for you. Interact in real-time and make your fantasies come true.
✓ Live Streaming✓ Interactive Chat✓ Private Shows✓ HD Quality
Anya is LIVE right now
FREE
Free to watch • No registration required • HD streaming
tl;dr: Using the benefits from using composer class loading to stabilize TYPO3 6.2 projects
The benefits we saw when implementing composer class loading into TYPO3 7LTS were so tremendous, that we took the additional effort to backport some of these changes to TYPO3 6.2.10. By default TYPO3 since 6.2.10 uses the composer class loader to load all classes for required TYPO3 core extensions, but you can use it to load every class in your TYPO3 6.2 project. To do so, you have a little extra effort, but you can say goodby to any class loading related troubles during deployment or after clearing caches.
Introduction
What I'm going to describe here is not a complete composer setup, which would also be perfectly possible with TYPO3 6.2, but the few steps you need to take to take advantage from composer class loading in your existing non composer TYPO3 6.2. projects. If you already use composer to setup your TYPO3 projects, you may as well want to have a peak into this article to make sure you get the most out of the possibilities.
Setup
A prerequisite for this to work is, that you put your TYPO3 sources in a folder, not a link, and this folder resides on the same level as your index.php like this (I skipped fileadmin, typo3conf, … directories here for better overview):
Add this composer.json file on the same directory level:
Step: 2: Adapt class loading registration in composer.json
The example composer.json file from the gist comes with class loading information for every TYPO3 core extension, including the class aliases for 4.x compatiblity. What you now need to do is add class loading information for every single extension in your installation. I prepared examples for typo3_console, news and tt_news, which should be self-explanatory. You basically need to add all paths in which the extensions in your installation have classes. Make sure not to forget to specify the relative path to the directory (seen from the composer.json file).
You can even try to only specify "classmap": ["typo3conf/ext"] as a single path, which saves you quite some time and works out in most cases, but can fail in some edge cases.
Step 3: Perform a composer install in this directory
Download composer if not done already, navigate to the directory with the composer.json file and perform a composer install. The result should look like this:
It is mandatory that the Packages folder, which is created during the composer install run is on the same level as the typo3_src folder.
Step 4: Set the TYPO3_COMPOSER_AUTOLOAD environment variable
Add the following to your apache vhost configuration:
SetEnv TYPO3_COMPOSER_AUTOLOAD "On"
or the following in your .htaccess file:
RewriteRule .? - [E= TYPO3_COMPOSER_AUTOLOAD:On]
For nginx, you can set a fcgi param like that:
fastcgi_param TYPO3_COMPOSER_AUTOLOAD On;
Verify that it worked
Remove all class loader cache files in typo3temp/Cache/Data/cache_classes and load the backend and frontend of your website. The website should show up and no cache files should be added to this directory. Do not delete the complete typo3temp/Cache because then, a few class cache entries are written anyway and you cannot confirm or deny whether you did everything correctly.
Optimize
If you do not have every single TYPO3 core extension installed, you should consider removing the lines from the composer.json file which point to core extensions you do not use in the project.
If you do not want to expose the TYPO3 sources directory and the Packages directory and your composer.json file, as they contain quite some information about your installation, you can create a web folder and link the necessary folders and files into it, e.g. like that:
$ ls -l web/ lrwxr-xr-x 1 12 Oct 10 17:43 fileadmin -> ../fileadmin lrwxr-xr-x 1 12 Oct 10 17:44 index.php -> ../index.php lrwxr-xr-x 1 8 Oct 10 17:44 typo3 -> ../typo3 lrwxr-xr-x 1 12 Oct 10 17:45 typo3conf -> ../typo3conf lrwxr-xr-x 1 12 Oct 10 17:44 typo3temp -> ../typo3temp lrwxr-xr-x 1 10 Oct 10 17:44 uploads -> ../uploads
Last but not least: Enjoy the gained stability and speed!
How to configure class loading for extensions in TYPO3 7LTS
tl;dr: In 99,9% of the cases, you do not have to do anything. Every configuration option below is just for clarity and convenience during development or to cover the 0,1% edge case. It is however considered best practice to add any of the following configuration options.
In my last article I gave a lot of background information why we discarded our class loader in TYPO3 7LTS and switched to use the one provided by composer. This post is more a tutorial with a lot more examples. It has a focus on how rather than why and the target group is TYPO3 extension developers.
Minimum setup
In this section I describe what you can do to get better class loading suport in TYPO3 7LTS. Everthing you find here, you can safely do without loosing compatibility with older TYPO3 versions!
Compatiblity mode
If you don't want to, you do not have to do anything! Regarding class loading your extension will work with TYPO3 7LTS anyway in 99,9% of the cases. TYPO3 scans the extension directory for class or interface files and registers every single one automatically to ensure backwards compatibility.
Important: If you add classes during development they will not be found by the class loader unless you update your class information files. If you have classes located e.g. in a tests folder which should not be availble in production, you should consider one of the following ways to configure class loading.
Extensions having one folder with classes
Considering you have an Extbase extension (or an extension where all classes and interfaces reside in a Classes folder) you can simply add the following to your ext_emconf.php file:
The folder Classes will be scanned and every class or interface found will automatically be registered, no matter if the classes use PHP namspaces or not.
Important: If you add classes during development they will not be found by the class loader unless you update your class information
Extensions using namespaces (possible since TYPO3 6.0)
TYPO3 introduced a way to load classes using namespaces, which was later standardized by the PHP framework interoperability group as PSR-4. If your extension has namespaced classes, then you can add the following to your ext_emconf.php file:
Here you do not only specify a folder, but also a namespace prefix which all classes in this folder must have. The prefix must end with a backslash!
Note: If you do so, newly added classes during development can be resolved automatically, whithout any further action! There is no need to update your class information files!
Legacy extension which previously used an ext_autoload.php file
As outlined before, it is not necessary any more to specify the name and location of every single class. You can specify folders and TYPO3 scans for the classes in them. So for example to provide class information for tt_news, Rupi could add the following to the ext_emconf.php:
Note: As you can see, it is perfectly fine to add multiple folders or even single files to the class map. However, just to stress it again, if you add classes during development they will not be found by the class loader unless you update your class information files manually.
Advanced setup
This section is a collection of further things you can do to tweak your development experience regarding class loading in TYPO3 7LTS.
Class loading information in composer.json
If your extension has a composer.json file already with a psr-4 or classmap autoload section, you are done. No further action is required. The section in the composer.json file can look like this:
As you can see, you can combine both classmap and psr-4 sections (which is also possible of course in an ext_emconf.php file.
Fixture classes
The old TYPO3 class loader could automatically resolve classes inside a Tests folder, in case the namespace reflected the folder structure below. In TYPO3 7LTS this is not the case any more, which really is a good thing! If you do not want to use composer to execute your extension tests individually like I described in a previous article, but execute them in the context of your TYPO3 project installation, you can add an autoload-dev section in your ext_emconf.php file:
Note: Just like composer, TYPO3 ignores an autoload-dev section in a composer.json file. So unlike the regular autoload section, the autoload-dev section is only evaluated if present in the ext_emconf.php file.
General notes
Update class loading information
The class loading information is updated every time you activate or deactivate an extension via extension manager. In the rare case you want to update it manually, you can run the following command from the command line:
It is possible to manage the source code for your TYPO3 project exclusively with composer (which I can highly recommend). In that case, all class loading information provided by extensions in the ext_emconf.php files is completely ignored! However as best practice and for the time being it is recommended to nevertheless provide this information in your extensions, so that it can be used in projects which are not set up using composer.
Enjoy developing with TYPO3 7LTS. It is an awesome release!
[UPDATE]: If you are less interested in background information and more interested in a "how to" tutorial, you should consider reading this article I posted lately.
tl;dr: This is a long post decribing why composer class loading is awesome and how you benefit from it in TYPO3 extension development
Little bit of history
Class auto loading shipped the first time with TYPO3 4.3.0. We improved it over time, added caching and made it possible to load namespaced classes since TYPO3 6.0. Lastly we changed it to work with a back port of the Flow package management in TYPO3 6.2. The caching layer for the class loader introduced in that TYPO3 version was powerful and fragile at the same time. Clearing caches under heavy load, can lead to a situation where the complete TYPO3 page is broken.
The challenge
We invested a lot of time to improve the situation and introduced some locking, but still a few main conceptual problems remained:
Besides having a registry with class name to file mapping, TYPO3 tried to resolve class files using several (partly conflicting) strategies.
To avoid doing these expensive operations again and again, the information is cached in files, but these files contain absolute paths, which makes the cache unusable if CLI access has different paths than web access. Warming up caches on CLI during deployment is impossible in such situations.
The file location of classes never change in an installation and could be known upfront, but TYPO3 tries to resolve the location of a class dynamically in a regular request. At the same time, it can never be determined whether the information is complete or not, because different code paths, could need different PHP classes that are then looked up dynamically.
In a production system the class information does not change unless an extension is installed. However this information was written into volatile caches, which are often flushed. This lead to repeatedly unnecessary re-calculation of these caches.
The solution
Finding a solution for these issues was brain twisting for me, until I realized that composer and standardization of class autoloading solved all problems for us at once. We only needed to implement composer more deeply into our system. Here is why composer class loading solves all the issues we had:
There is no magic involved in the composer class loader. How classes are loaded must be specified for each package (or in TYPO3 terms: each extension).
By specifying the class loading information, the class names in your extensions, must no longer be related to the extension name and are not bound to a Classes folder.
With this class loading configuration the composer class loader knows how to load every single class during build time. No assumptions, no magic, no additional class information lookup at runtime.
The class information is static. In a production (TYPO3) system, it is just there and never changes unless a package (extension) is added or removed. No failures due to broken caches during deployment, as there are no caches any more.
Class information configuration is simple. Just a few lines instead of long ext_autoload.php files.
Class information is stored with relative paths. So it can be generated on one system and deployed to another one without issues.
The implementation
We completely removed all TYPO3 class loading code in TYPO3 7 and now rely on the composer class loader to load all classes of TYPO3 and every installed extension. If you set up your TYPO3 installation (including extensions) with composer (not downloading a package and symlinking the source like in the old days), TYPO3 code does nothing regarding class loading, composer takes over that task completely.
Non composer mode
In case you want to set up your TYPO3 project traditionally without using composer, TYPO3 ships with the composer class loader and all necessary information to load all core classes, but additionally executes a very slim layer of code to detect and register classes of active extensions, so that the composer class loader can load them as well. If an extension does not provide class loading information, the complete extension directory is scanned for class files and every class found, will automatically be registered for the class loader. The ext_autoload.php file is completely ignored in TYPO3 7.
There might be some cases where a full scan of all directories is not desired, e.g. when third party libraries are shipped or test classes are provided with the extension. Because of that, extension authors can provide class loading information which will be evaluated by TYPO3. As of TYPO3 7.5 this information can be provided in an autoload section of the ext_emconf.php file, or in a composer.json file.
In contrast to composer, TYPO3 only evaluates the psr-4 and classmap sections. But we think this is more than enough and even easier to handle than before.
Benefits
We do not have a classes cache any more. So this cache cannot be corrupt, will never be flushed (and thus being unavailable) and must not be build during runtime. Besides that, we have a lot less code. The results are:
Better performance
Higher reliability
Higher flexibility
Less maintenance costs
Implications
We do not generate class loading information during runtime any more, but change the class loading information every time an extension is added or removed from the installation. If you use composer, it takes over that task and updates the class loading information on every composer install or composer update. If you do not use composer, TYPO3 will update the class loading information every time you add or remove an extension with the extension manager or via command line (typo3/cli_dispatch.phpsh extbase extension:install realurl) or if you explicitly trigger TYPO3 to re-build the information from the command line: typo3/cli_dispatch.phpsh extbase extension:dumpclassloadinginformation
If you develop extensions and add classes, it is therefore highly recommended to follow the PSR-4 specification and add a psr-4 section to your ext_emconf.php or composer.json of your extension.
Combined TYPO3 6.2 and TYPO3 7 compatiblity
Since ext_autoload.php files are ignored in TYPO3 7 and an autoload section in ext_emconf.php is ignored in TYPO3 6.2, it is pretty easy to provide compatibility for both TYPO3 versions with your extension. Just add both, and your extension will work in both TYPO3 versions. If you follow the PSR-4 standard, you of course only need to add the according section in ext_emconf.php. Of course your classes and interfaces must reside in a Classes folder, as TYPO3 6.2 only looks up classes in that folder. My scaffold extension for example is compatible with both TYPO3 versions in that regard.
Conclusion
The listed benefits should be enough to justify this change. With this step we also step closer to the broader PHP community by using composer. It is something new to learn for TYPO3 community members, but we are convinced this is for the benefit of our product and both the TYPO3 and PHP community.
[EDIT]: Now xsd.helhum.io is available via https:// let's encrypt ftw!
For my post describing how to make your IDE auto complete Fluid view helper tags, I created http://xsd.helmut-hummel.de as a place to download and reference Fluid xsd schemas. There was a small issue with that however. It missed a way to support different versions of TYPO3 or Fluid.
To fix that, I now added different XSDs for different TYPO3 versions to this service.
While doing so, I decided on a different URL pattern which now is as follows:
It is a generic URL where parts of it are replaced with the composer name and another part with the version number (but only major and minor or "master").
This means that the following (new) schema files URLs are now available:
A lot of things changed in and for TYPO3 during the past months. Most notably composer support improved a lot, but also class loading is now based on composer under the hood. As a result, also the test setup improved, but also radically changed compared to older TYPO3 versions.
To help developers benefit and adapt to the changes, I created a skeleton extension, which has everything set up to easily execute tests, and benefit from composer support.
As additional gem, I added a .travis.yml, which is a ready to go set up to run your test builds using the great service Travis CI
Check out the README, which should also explain why everything is set up that way.
Let me know if some things remained unclear, so that it can be improved, as everything will eventually end up in the official documentation for the TYPO3 7LTS release.
Anya is live and ready to show you everything. Watch her strip, dance, and perform exclusive shows just for you. Interact in real-time and make your fantasies come true.
✓ Live Streaming✓ Interactive Chat✓ Private Shows✓ HD Quality
Anya is LIVE right now
FREE
Free to watch • No registration required • HD streaming
[EDIT] Added the packagist comparison section and clarified the requirements section a bit.
During the TYPO3 Developer days this year, I talked to Thomas Löffler about his vision regarding the future of the TER. He had the idea that extension authors should not need upload a package for each extension version, but register a repository URL which is then evaluated regarding its tags. So pushing a new tag would mean releasing a new extension version. This is nothing new for everybody who knows packagist and composer, but for our grown infrastructure this could be a challenge.
A few days later this topic popped up for me very often and eventually I got in touch with Alexander Schnitzler. Together with him (and in cooperation with Thomas) we figured out a first step into the future of TER.
We agreed on the following:
Extension authors can set a repository URL to their version control repository when editing the extension meta data on typo3.org.
This looks like this:
If this is done, we scan this repository and publish it on composer.typo3.org automatically.
Before you do so now for your extension, please read the requirements that need to be fulfilled for that to work:
Requirements
The repository mus contain tags names which are version numbers, so that they can be identified as release
Tags and branches must contain a composer.json file. Having an ext_emconf.php file only is currently not enough!
If these requirements are not fulfilled, or better, if none of your tags or branches have a composer.json, your extension will not get an additional listing from this repository URL. It is perfectly fine however, if at least one branch or tag contains a composer.json file. If the requirements are fulfilled, the extension shows up on composer.typo3.org with the composer name you defined in your composer.json. This means that e.g. my typo3_console extension is now available as helhum/typo3-console as well as typo3-ter/typo3-console. The latter stays for compatiblity, the first should be used for more control and better composer support (e.g. more precise class loading information).
Important! This is not the complete!
Adding the URL as described above does not (yet) upload your extension to the TER! It only provides it for composer users. If you want it to be available through the TYPO3 extension manager, you must still additionally upload the package for each version. Tagging alone is not enough. But we have ideas already how we can close this gap to a brighter future of our TYPO3 extension repository (TER).
One more thing
Since we use Satis to scan the repositories and build a final composer repository, there is now the possibility to list all registered packages and even search for them.
Benefits over registering on packagist
There are not much benefits you get if you register your extension repo on typo3.org. One is, that we poll the repositories, so there is no hook required in your repository to update th versions on composer.typo3.org. You trade this in however for the need to specify the additional http://composer.typo3.org repository. In the end it is up to you what you prefer as extension author.
Credits
Alexander Schnitzler did most of the programming (typo3.org and composer.typo3.org), Thomas Löffler helped with reviews for the typo3.org stuff and I helped conceptually, with reviews, a little code and the final deployment. Cheers as well to the server team for their help!
Making private legacy TYPO3 extension git repositories composer aware
tl;dr: The t3satis package helps to make your legacy TYPO3 extensions that do not have a composer.json file yet composer aware.
I'd like to add one to this list, which is my helhum/t3satis package. Satis is a composer package repository generator, which you can use from the command line. Basically you create a satis.json file with all your (private) repositories listed in there and use the command line tool to build a little static website, which can act as composer repository like packagist.
However repositories added to the satis.json must contain a composer.json in the tags or branches. This very often is not the case when you have legacy TYPO3 extensions in repositories. To be able to work around that limitation, I created t3satis.
If you create your own composer repository for your legacy extension with t3satiss, there is no need any more for a huge list of package repositories in your root composer.json file. Just add one composer repo with the URL to your t3satis website and you are done.
If you know satis, this should work immediately for you. Just look at the example that is provided in the readme