Docuamoury is now Live - What is it, Why I built it and what problems occurred along the way

05-11-2024 18:34

If you’ve been following Twitter account you would have seen that I’ve building a new project for the last few months called Docuarmoury.

Its taken a bit longer than I expected, but its finally available. If you don’t know what it is, then read on as I’m going to go into what Docuarmoury is, why I built it, and what problems I faced during the development.

What is Docuarmoury?

Docuarmoury is a central location for your products and services as well as your internal business documentation. Its a simple documentation service where Markdown can be used to create documents, either by writing Markdown directly, or using the built in WYSIYG (What You See Is What You Get) Editor.

You can host the documentation via your own custom domain, just need to point a DNS CNAME record for your domain (or subdomain) and point it to Docuarmoury.

Docuarmoury supports both public and private documentation. For example, your customer facing products or services are publicly accessible by anyone, or your internal app and processes can be hidden behind user authentication so only people you provide access to can access Docuarmoury.

When creating a documentation page, Docuarmoury handles the SEO for you, it uses Natural Language Processing (NLP) to take the document you’ve written and auto extract keywords and description and assign to Meta tags that are used to help search engines.

Why did I Build Docuarmoury?

There was a couple of reasons I built Docuarmoury. I create a fair few number of projects and each time, invariably, I need some sort of documentation to be able to provide to users. This meant I was building something new each time for every project which is time consuming.

Secondly, other tools I found, were either expensive, and as a solo developer wasn’t worth spending money for something that currently is making me money, or needed to be redeployed for every documentation site, so, didn’t really resolve the issue of having to create something new each time, although this set up was probably quicker than writing a new documentation site from scratch.

Because of that, I decided to build my own, yes building my own takes time, and as they say time is money, but my time doesn’t come out my bank account :)

Therefore I built Docuarmoury as a simple markdown documentation site, where I don’t need to design or host something new every time, I can just sign in to Docuarmoury, go straight into creating my documentation, then point my projects documentation URL to Docuarmoury for instant hosting.

What Issues Did I Run Into?

The main development process wasn’t too bad. There were a couple of big issues but were relatively easy to overcome, once I figured out what was going on.

First off, earlier I mentioned how a document can auto generate the SEO meta tags such as meta keywords and meta description using Natural Language Processing (NLP).

I do all of my work locally on a Mac so I built this initially using Python where the main Docuarmoury app would send an API request to a Python script that included the document content, and then the Python script would run a local language model to process the content and extract the keywords and meta description.

The meta keyword extraction was relatively straight forward as this doesn’t use NLP, this is done algorithmically by determining how often a word appears in the text. The description however did use full NLP processing so was using a language model to do the processing. This worked fine on my local Mac PC I use for development.

When I then deployed it to my hosting server (a VPS (Virtual Private Server) in Digital Ocean) the API request from Docuarmoruy to the Python script would fail.

Initially didn’t understand why but I eventually found out the API route from Docuarmoury would time out, and after the timeout the Python script would return the keywords and description but it was too late as the API route had already timed out the request.

I debugged the python script to determine it was purely the NLP side of the script that was the problem, the keyword extraction worked fine. Eventually, probably a bit longer than I would like to admit, the penny dropped as to what was the problem. My Mac dev machine, has a built in GPU, the VPS within Digital Ocean doesn’t, and AI tools such as Natural Language Processing use GPU for better performance compared to CPU, so because the VPS doesn’t have a GPU, it tried to use the CPU, and the CPU was low powered as it was only a POC server.

I didn’t really want to upgrade the VPS to a high end CPU just to do a bit of NLP, as this would be expensive, even if I had lots of customers, it would probably cost more than I would make, especially at the early stages of the project. Secondly, I thought about a VPS that does have a GPU, at the time Digital Ocean didn’t offer this (I believe they do now as an early release) but again I would have the same cost problem.

After a bit of research I found an alternative. I can still send an API request from Docuarmoruy to the Python script to do the keyword extraction, but instead of me doing the NLP locally, I found that that Cloudflare offers AI API Routes that uses the same model I was already using in the Python script. So I modified the Python script to send an API request to the Cloudflare AI API route and now the extraction was happening within a couple of seconds.

The next issue was towards the end of the development and on the testing phase. I completed my basic developer testing, so then deployed to a staging server as an official release and went through a more formal testing plan to confirm that everything works as expected end to end.

This is where I ran into an issue where viewing a certain page would just throw an internal server error and crash. The same worked on the dev environment so was very confused as to why the production build would fail in this way.

Eventually I figured out it was to do with using an older 1.x version of headlessui for React for tabs and using Sentry, where Sentry needs to push props to a component, but headless ui was using a fragment so sentry wasn’t able to push its props to the fragment as fragments don’t support props.

If you aren’t aware, Sentry is a crash and error reporting tool so I can receive these reports automatically should something unexpected happen within the App, I can get a report and then work on a fix and re-deploy without having to rely on customer reports.

Initially I was still confused as to why this didn’t happen in dev, but then realised I set up Sentry to only work in production releases not dev builds as its kind of annoying to keep getting crash and error reports for a project you actively developing.

I then realised that headless ui had a new version which meant it didn’t use Fragments any more and resolved the issue, however it was fairly drastic change to how their components were implemented so I then had to re-write some of my components that made use of headlessui to work with the new version.

Once this was done, I was able to redeploy a production release to my staging environment and re-do my tests which finally all tests completed and I was able to release Docuarmoury.

If you want to sign up, there’s a 14 day free trial at https://docuarmoury.com. If you have any feedback, good or bad then please let me know.