Concerned about vendor costs when it comes to producing technical documentation for the web? You might want to take a look at our recent article in EE Times reprinted below. (See the original article at https://www.embedded.com/free-tools-build-multi-format-documentation-systems/.)

Using open source tools, it’s possible to create a documentation system that can present the same information on large and small displays.

In a recent column on Embedded.com, Max Maxfield blasted the slapdash guide that accompanied a module he’d purchased and asked why so many manufacturers neglect documentation until the last minute (see Basic documentation—is it too much to ask for?). Was a budget or staff ever too big to blame for procrastination? Yet even a conscientious development team can’t supply useful help without a well-organized system in place to record work in progress, edit the information, and publish documentation in whatever formats are required for easy access. There are many ways to build such a system based on commercial software, which can be expensive, or by using free and open-source tools.

My firm has generated documentation for more than 30 years for semiconductor fabrication equipment, graphics processors, test instruments, network gear, CAE tools, and so forth. We’ve rescued clients on the brink of product rollout and worked on multiyear projects from the inception. The best path for any manufacturer, whether a startup or an established enterprise, is—before product development begins—to put in place internally a documentation system tailored to deliver information in ways most convenient to customers. Let’s consider a straightforward, low-cost approach one company used to get up and running quickly.

A long-established manufacturer that produces biomedical instruments and related assays for disease screening sought a documentation system that could output product literature in both PDF and in HTML format for Web presentation from the same source text. The company had built an extensive library of instrument manuals and detailed guides for its many assays, all of which had been produced using the Adobe page-layout tool, FrameMaker, and needed a flexible platform for generating searchable new documents for electronic display that would be virtually identical to their print counterparts. The content, as well as the structure of any of the documents in either format, requires FDA approval subject to stringent federal review.

The project was fast-tracked by a battery-backed portable instrument that was in development. This instrument tests biological samples in the field to quickly determine whether patients have HIV, certain influenza strains, or other infectious diseases. It is intended for use at remote sites and in neighborhood clinics, especially in developing nations, where there may be limited electrical power and economy is paramount. The user interface is a cell phone controlled by a dedicated application, unlike other instruments made by the company, which communicate with a laptop or desktop computer.

Instructions for operating the instrument, as well as the guides for the assays it runs, reside in the cell phone, except for brief startup steps on paper to turn on the phone and open the app. Assay guides run 30 to 40 pages in PDF, posing a challenge how to present the material legibly on a display that is less than six inches long by three inches wide in a structure that could be easily searched.

An open-source standard, DITA (short for Darwin Information Typing Architecture), was chosen as the foundation for the documentation system. DITA, which was originated by IBM, defines an XML architecture for publishing information in multiple formats for print, Web display, and retrieval on mobile devices. Document outputs in the various formats are implemented using the DITA Open Toolkit, a collection of open-source software programs. The upshot of DITA is that content is distinct and independent of how it will be presented, with reordering and reuse in mind.

Some existing assay guides had to be translated from PDF for use with the portable instrument. The content was extracted in essentially a cut-and-paste operation and then tagged in a DITA XML markup. Formatting templates were created so the documentation system would strictly adhere to the FDA-approved style for the company’s literature in PDF, and then the toolkit was used to output files in HTML5. The toolkit can render DITA XML files for output in several formats, including XHTML, HTML5, PDF, and others. Although FrameMaker, the page-layout program, also can export files in HTML, it would actually complicate building a Web portal for documentation: you can’t make inter-document links, for example, or readily create a hierarchy for building a documentation site.

Arranging the HTML5 output from the documentation system for display on the cell-phone screen involved Bootstrap, a framework for automatically scaling websites for viewing on phones, to tablets, to desktop computers. Bootstrap, which is a collection of cascading style sheets and JavaScript, employs a grid for defining how information should appear within different screen dimensions. For a large 4k display, for example, content could be presented in multiple columns, if desired; for smaller screens, how elements shrink, are rearranged, or remain visible can be defined.

In the case of the project we are discussing, content is displayed in a two-column makeup on a widescreen, and in one column on the cell phone with a collapsible table of contents at the top. For the phone, each document section amounts to a Web page. Information is displayed one section at a time. There is always a table of contents that can be expanded for quick and easy navigation through the document. If the table of contents is collapsed, the material for one section is displayed and—when read to the bottom—there is a link to go to the next or previous sections.

During the development of this system, a technical detail tied to regulatory acceptance arose that had to be resolved. When documents are authored in FrameMaker, tables that continue from one PDF page to the next repeat the table title. However, the PDF outputs yielded by the DITA process don’t repeat table titles from one page to the next—just the table headers.

Another tricky issue, general in nature, was how figures are numbered in the HTML5 output from the DITA toolkit. The PDF output is fundamentally a continuous scroll, but the HTML5 output is broken up into sections and the toolkit does not number figures in succession but starts again from 1 in each section. The fix involves adapting a bit of code from the PDF process. Basically, the PDF process produces a file that merges topics mapped in DITA, but HTML documents are collected assemblies of the topics, each of which remain in a separate file. The PDF process merges everything from the DITA map, everything from the document hierarchy, rolled into one big file that is used to count such things as figures. The code that was appropriated from the PDF process modifies the HTML process to maintain the consecutive numbering of figures.

Any plaintext editor can be used at the front end of such a system to create content, though a modest investment for Oxygen, XMetaL, or other commercial XML editor is worthwhile. Those programs are much less expensive than FrameMaker, which is the conventional workhorse for document creation.

Startups, especially, who want to build a flexible documentation system quickly and inexpensively that can publish material in multiple formats from the same content can benefit from this approach, which is based on free and open-source software tools. The only problem remaining, therefore, is procrastination.

A client who provides a web proxy service to shield enterprise customers from malware recently hired us to build a documentation system we had proposed to quickly produce content for web presentation. Draft text is shared, edited, coupled to graphics, and automatically tailored for export straight to the web in a smooth, secure process based on free and open-source tools.

The company as a proxy executes customers’ web sessions, removing Flash and other active elements that might contain threats, and then relays the sanitized results to the customers, all without perceptible delay. Their customers need clear instructions for installing the service but when the company approached us, the documentation at hand was a mix of PDFs, Word texts, and a few procedures online, all in need of a thorough technical edit and consistent style and branding. We converted the existing documentation so it would be clearly legible on the company website and structured a system for creating new documents that could be easily uploaded to the website and displayed in any screen format with no need for conversion. We had in mind an infrastructure that would be easy to maintain.

First, we converted the existing documents to plain text and then we used Markdown, a simple mark-up language that provides a syntax for plain-text formatting, to generate the typographic conventions for headings, italics, bold type, numbered lists, unordered lists, and so forth. Moving forward, we developed a consistent style in which documents would appear on the website.

Easy to implement

We based the documentation system on a static-site generator—Jekyll–which takes the plain-text files in Markdown, organizes those according to templates we created for text and style, and creates web pages and a navigation hierarchy. Simply run a command and the system produces a directory of files for upload to the web server and the process is complete.

Jekyll is an open-source static-site generator that was intended for blogs as an alternative to WordPress, which was becoming complicated, cumbersome, and involved managing a back-end database. It is extremely easy to use. Features that make it helpful for blogs also make it a very convenient engine for a documentation system. For example, it can automatically create lists of all the blog posts from an author in reverse chronological order and create tags so posts about a particular subject can be collected on one page. We leveraged Jekyll because it thereby enabled building an organizational framework for the documentation system, not only style templates. For fast website deployment, security, and limiting infrastructure maintenance, we decided a static-site generator would best serve the application.

Streamlined workflow 

Once we converted the existing documents to plain text, we put them into a private repository on the website Github.com to share and collaborate with the client’s staff. The Github website is based on Git; a version-control system that can pull from many servers. Everyone given company access has their own version of the entire private repository, enabling the staff to work on the files and see each other’s changes.

The client’s chief of publications produced raw content adding to the work we had done, and then we did a technical edit of the new material and submitted changes to him, in an informal ongoing process. However, based on Github, the structure is in place to implement a formal process for merging changes from authors, who can make technical edits and then submit a pull request: I’ve made this change; do you want to accept this into the main branch?

Beyond version control, Github gives a preview of how documents will look. Simply upload a Markdown file with accompanying images and Github melds into order the text and graphics free from compositing symbols. The publication chief can confirm how pages would be organized, set the layout with the Jekyll templates, and commit the results for upload to the company website.

The benefit of using plain text for generating documents, instead of such programs as Framemaker or Word, is that you can tap into the decades of development founded on plain text; namely, all the sophisticated tools that can analyze two very different versions of a text, see where they differ, automatically merge changes, and identify where a conflict must be resolved.

There is some downside of working in plain text, at least for engineers who are documenting product designs in progress. Composing in plain text is not as familiar as writing in Word. However, making the transition from Word to composing in plain text is not especially onerous.

Given the simplicity, convenience, and low cost of using this combination of free and open-source software to build a flexible and robust online documentation system, the approach can benefit many ventures, especially startups who have limited IT resources, small budgets, and pressing calendars.

Building a New Publishing Process While Developing Content for a Future Product Release

We recently completed the first phase in a significant new project. It required designing, building and implementing a new documentation process, while at the same time developing content for three new products.

Our client builds mobile application software development tools. They were looking for a tech writing company that also could make their documentation process more agile and productive.

The process they were using relied on structured FrameMaker which then was converted to HTML and PDF files. But they found this slow, hard to manage, and outdated. Our role was to design a new content development and workflow process and to assist in its implementation.

At the same time our tech writers would be working with their software developers, who are located at different regional centers. We would have to develop a working relationship with remote experts and collect, develop, update, and publish content in a timely manner.

The group assigned to the document publishing process set about exploring different publishing processes. The group assigned to developing content immediately started working with the subject matter experts.

The Publishing Process

To start, we examined several options for publishing content to the web. These included Python, Sphinx, DocBook, Slate, and others. We built demo sites to show their look and feel. And we evaluated their ease of use,  their output templates, their suitability for the client’s resources and processes, and their scalability.

As a software development company, the client was already using Atlassian’s enterprise software tools for issue tracking and team collaboration. The Developers, our subject matter experts, were comfortable working within Confluence, the Atlassian wiki, to document the products. And Confluence uses a REST API, so the wiki could be queried and content pulled from it.

We realized that if we could continue using that wiki, the Developers would be much more responsive in delivering the content we needed. We also could use JIRA, the Atlassian issue tracking tool, to monitor issues blocking content development.

The goal then became to build a process that used the wiki and integrated seamlessly into their workflow. The result would allow the client to be in complete control of the process, without having to learn any new technologies or tools.

However, important issues remained to be solved.

Because they had in-house NodeJS resources, we proposed using DocPad as the interface to pull the content from the wiki. To get this to work, we built a special DocPad plug-in to automate the query and pull the content from the wiki. The content was rendered to templates built for customer-facing content and then deployed to their AWS servers.

The following screenshots show the content as developed in the Confluence wiki and the output as it appears on the company’s web site. Changes made on the wiki can be scheduled to appear on the web site whenever you want.

Confluence wiki page	Public-facing content

So what does this mean for you? First, we listened to what the client wanted. We worked with them to determine the best approach to the problem, and we made maximum use of their in-house resources. The result was a process that they completely control. At any time, anyone in the company can view the current status of the content …. There’s no hiding, so you’ll always know if the content will be ready with the product at release time. And when it’s ready for release, it can be pushed to the web site and packaged with the product build as part of the SDK. Fast …. Easy!

By the way, if you’re interested in the DocPad plug-in, you can get it at NPM (Node Package Manager):

https://www.npmjs.com/package/docpad-plugin-conflux

GitHub:

https://github.com/phoenixtechpubs/docpad-plugin-conflux

Developing Document Content

Of course, what good is a process if you don’t have content to publish? The first issue the tech writers had to resolve was developing a working relationship with subject matter experts located in three geographical regions. Site visits could be made to the local office, but video conferencing tools were needed for remote sites. Two that we found useful were Google+ Hangouts and join.me by LogMeIn, Inc.

Working with the subject matter experts, the tech writer created wiki pages for the product content. Links to those pages then were sent to the subject matter experts so that they could write, review, and correct content. The “Watch” feature in Confluence alerted participants to any changes; the “Talk” plug-in was used for contextual Q&A exchanges. The tech writers ensured that the content was written with a consistent editorial style, that it fit within the overall organization of the material, and that there was a smooth narrative throughout.

During the development process, the wiki content was published to an internal review site so that anyone with authorization could see what it would look like when published to the web site. This was important because the template for the public-facing content is branded differently from that displayed in the wiki.

Document control was handled using Comala, the Atlassian workflow product. This ensured that the content went through the company approval process before being released to the web site.

So how did this work out for the client? They were never in doubt as to the availability of the documentation when it came to product release. And those last minute changes that came in were no problem at all.