Joomla! Documentation: Future Directions

Introduction

Much of the Joomla Documentation is handled by a MediaWiki installation. It does a good job of delivering the Help pages invoked  with a Toolbar button on almost all Administrator pages. However, it has shortcomings that have led many in the community to look for an alternative. Some of the problems are mentioned at the end of this article.

In the search for an alternative to Mediawiki, some effort has been put into delivering Programmer documentation with Docusaurus. I think this is not well suited to the needs of Joomla! Again I give some reasons at the end of this article.

It is my belief that we should look no further than Joomla itself for a delivery and management system for Joomla documentation. This article is a description of how I would do it. I am actually doing it! My work may result in a free Joomla extension suitable for managing any documentation. Perhaps re-inventing the wheel. Perhaps not.

This article is about Future Directions. So only documentation for Joomla 4 and later. Everything else will stay in Mediawiki as an archive.

Source Storage

It seems that most of those involved in Joomla documentation agree that source text should be stored in a Git repository in Markdown format. I agree too! It also seems that Github Flavoured Markdown, which is a subset of CommonMark (a Markdown pseudo-standard), is to be preferred to Mediwiki markdown.

Markdown Conversion

I have written a php script to convert MediWiki markdown format to Github markdown format. To fetch the MediaWiki pages I made an index list of Headings and Files. It takes about 10 minutes to fetch all of the English Joomla 4 Help screens, convert them and store them in a local git repo. Other languages take less time because not all Help pages have been translated. For this demonstration I have converted 1757 files in 8 languages. Here is a sample of the terminal output progress:

php manualp-convert.php -l="en" -m="help" -p="yes"

help-screens
.
admin-modules
......................
articles
......
banners
........
cache
..
chunks
.
component
....
components
................
contacts
.....

The words are the headings and the dots are individual files.

Git Repo

I am using a local git repo for development work There is no particular necessity to use GitHub to store documentation! In fact it could be a disadvantage. The storage structure within the repo looks like this:

manuals/help/de/heading/filename.md
manuals/help/de/heading/next-filename.md
manuals/help/de/next-heading/filename.md
...
manuals/help/en/heading/filename.md
manuals/help/en/heading/next-filename.md

There are three manuals: User, Help and Developer,  and perhaps Documenter and others to come.

Data Storage

Although the sources of individual documentation pages are markdown files some infrastructure is required to deliver the pages to end users. Amongst other things, there needs to be a list of contents and a method to convert markdown to html. For that purpose, the MediaWiki to GFM script mentioned above builds an sql query to create and populate a table containing information on all of the markdown pages. Joomla can handle that in the same way it does for any other component.

Pages List

The English version of the source markdown pages is always used as a basis for listing. If another language is selected an extra column shows whether that document has been translated into that language, as in the following screenshot:

In this list, apart from Help, the Toolbar buttons are not available to Editors. The user can select any one of the available manuals: Help, User and Developer and any one of several languages, not related to installed language packs.

The Index Key is the text used in Joomla to fetch pages from MediaWiki. It is the only thing that ties the translations to the original English. Click a Stash button to view or update the page in the selected language.

In the Toolbar, the Update HTML button is used to generate the static HTML delivered to end users. It is only used on site setup. It fetches the Markdown text from the repository, converts it to html and stores it in a field in the table.

Making changes

To edit a page a user must first open a copy of the page. When saved, a stash copy of the page is created in a separate table in the database. The user can work on that stash copy for as long as necessary. When content, the user can make a pull request to have the changes committed to the repository file. Whoever handles pull requests can examine the changes, ask for revisions or commit them.

The commit process involves writing out the changed page to the git repository, staging and committing the change and then deleting the stash copy from the database. All in one operation! That gives us an updated repository with a commit history.

My Stashes List

The second tab in the list page contains a list of all of the stashes I have saved and not committed.

Pull Requests List

This list is only seen by users with Committer privileges. Links open the same edit form used for a stash. Setting up a system with Translators and Committers is covered in the installation README.md file.

Edit a Page

The GFM Edit page has a number of tabs. The following illustrations show tabs for translation into German.

Details Tab

Most of the data in the Details tab comes from the initial installation so the only item that can be changed is the Display Title.

Stash Tab

When editing a translation, this page shows the original English alongside the translation area. When editing the original English the translation area occupies the full width of the tab. In this particular case, whoever created the translation only translated the page Display Title. Unfortunately it is very common to find only partially translated pages.

English diff

This tab shows the diff for the current English page and the previous edition. Translators need this to see at a glance what changed and what may need attention in the update. The change illustrated is just a note that this page really needs revision.

GFM Source Tab

This tab shows the current source of the page in Github Flavour Markdown format in the git repository.

Diff Tab

This tab shows the difference between the repo file and the stash version. For the following illustration I used Google to translate an English source into German for a document that had not been translated. The original is on the left and the stash version on the right.

Preview Tab

This shows a HTML rendered version of the saved stash.

Comments Tab

This tab has a field for a commit message to be filled out by the Editor or Committer. There is also a text area box to provide additional information to the committer or feedback from the committer prior to commit.

Delivery Storage

When a markdown file is imported or updated in the repo a HTML version is saved in the Joomla database for delivery to end users.

Appendices

WikiMedia Problems

• It is alleged to be insufficiently friendly to encourage new user contributions.
• It is difficult to create documents in the style of modern software documentation with an index column to the left and page content to the right.
• It is all to easy to create lists of documents for different groups of users that soon lead back to the starting point.
• The source pages in English have become cluttered with translation markup, sometimes excessively so.
• Translation is sometimes difficult because translators are presented with hundreds of isolated fragments of text that give little idea of the context. For example, language codes such as en may not indicate whether they belong to an image or a link and whether a translated version actually exists.
• The search function has insufficient filters. Searches for anything my return results cluttered with articles intended for earlier Joomla versions.
• The Category system returns lists of articles that include all language variants so are ten times longer than they need to be.

Why not Docusaurus

• It is another technology that someone need to keep up with.
• Provision for translation is vague at best and possibly non-existent or not usable.
• The contribution procedure is complicated: make a branch, change the content, make a pull request, commit and then rebuild the public content.
• It is not clear where images will be stored. They could bloat a repo to the Gb level.
• The layout of the delivered content is defined by the storage file structure. Bad idea! Difficult to change later.
• ...