Over the recent weeks and months, our team members have been involved in a project aimed at updating, enhancing, and resolving issues within the official TYPO3 documentation. While we didn't directly modify the content, we collaborated closely with the documentation team to refine the codebase of the application that indexes and searches through documentation files. Today, I'd like to share some details about our work with you.
Updating the Codebase of the Indexing Application
Updating to the Latest LTS Version of Symfony 6.4
The TYPO3 documentation indexing application, which utilizes Symfony components, was developed several years ago and its Symfony version has since become somewhat outdated. Last summer saw the first update to the code. However, due to technical constraints on the production servers, the updated application was not released.
Toward the end of 2023, we undertook another upgrade. This time, the application was updated to Symfony 6.4, the Long-Term Support (LTS) version. We also updated all other dependencies, fixed and expanded the tests, and finally transitioned to PHP 8.2. The release of this version was planned for the beginning of 2024.
Adapting Code for New PHP Documentation Rendering Support and Bug Fixing
After completing the transition to Symfony 6.4 and PHP 8.2, we began updating the internals of our indexing application to accommodate documentation files generated by both the old rendering process (which utilized Sphinx, a tool written in Python available at https://www.sphinx-doc.org/) and the new one based on PHP.
The documentation team opted to gradually migrate away from Sphinx due to challenges in maintaining the container used for generating documentation files. This was partly because of a shortage of individuals with Python expertise and the slow rendering times in certain cases (for instance, the old changelog took 60-90 minutes to render). Shifting to a PHP-based solution ensured a broader pool of people with the necessary skills to maintain the new container. Importantly, this switch has also significantly accelerated documentation rendering speeds (now, the changelog renders in about 5 minutes).
We have updated the indexing application code to support documentation files rendered using either Sphinx or PHP containers. Additionally, we introduced further improvements after consulting with the TYPO3 documentation team.
In the meantime, we have resolved several issues that were reported in the past months:
- Made each version label rendered on the search results list link to the corresponding version file
- Prevented rendering of duplicated version labels on the search results list
- Boosted search result items that are part of an extension manual if the search word matched the extension name
- By default, link each search result entry to the latest documentation version
- Fixed indexing and searching of TYPO3 changelogs
- Added a Changelog facet to the Document Type filters
- Indexed several documentation files and folders that were previously omitted
Additionally, we made some minor improvements that, while often not visible on the frontend, enhance the internal workings of the app.
Deploying on Production Servers
In mid-February 2024, in collaboration with members of TYPO3 GmbH, the updated version of the indexing application was deployed to the production server. Since then, it has been utilized to index and search all types of documentation files, including TYPO3 core manuals, extension documentation, etc., regardless of whether they were generated using the old approach or the new one based on PHP.
Searching Documentation Globally or Within a Single Manual Scope
Following the recent updates and migration of the documentation rendering process, modifications were also made to the page layout itself. A valuable feature that was previously available to users — the ability to search exclusively within the current project, rather than all available manuals — has been affected. There used to be a dedicated input field in the left sidebar for this purpose. However, this field has now been removed from pages generated using the new PHP rendering.
Old rendering - search inside project vs New rendering - no search inside project
Since this functionality enabled users to narrow their search scope and receive results specific to the project of interest, the TYPO3 documentation team has decided to reinstate it. We at Macopedia were also involved in this effort.
After consulting with the documentation team, we decided to restore this feature in a slightly modified form. Now, when you visit a documentation page generated with the new rendering, you'll notice an addition to the main search input field: a select list that allows you to choose whether you want to search within the general scope or the scope of the current project whose documentation you are reading.
By default, the "Search all" option is selected. If you switch it to "Search current" and then enter a search phrase, the search result list will display entries related only to the project you were viewing previously.
Main search form extended with the scope selector
While implementing this change, we also standardized the assets (CSS and JavaScript files) used on both the manual pages and the search results page. Previously, as these pages served different purposes, they imported various files and versions from a CDN. Now, they utilize the same, most recent version of these files, ensuring a consistent appearance across both pages.
Future TYPO3 Documentation Search Improvements
While addressing the previously mentioned issues and features, I also conducted a brief investigation into how the search form could be enhanced. The TYPO3 documentation team suggested it could evolve into a form similar to GitHub’s search, which allows easy searching within a single repository or an organization, among other options. So i spend some time on it and came up with the following mockup:
Improved TYPO3 documentation search form mockup
Our team at Macopedia decided to propose this development and realization as part of the TYPO3 Community Budget Ideas for Q2 2024 (you can read the full proposal here). The voting phase has now concluded, and unfortunately, our idea was not among the four that received funding. However, we plan to propose it again for Q3, hopeful that the community will recognize it as a valuable and desired improvement. :)