@CJ_Clippy
patreon
October 2023 Update
🕑 Added 2023-10-15 22:24:53 +0000 UTCNew features
- Tags & Timestamps have returned
- IPFS Content IDs get a copy button
The plan for second half of October, first half of November
Bugfixes and system reliability improvements.
Futureporn has become quite a complex system. There is a frontend powered by next.js, and a backend powered by Strapi. There's also a pair of IPFS provider nodes powered by kubo, with pinset orchestration powered by ipfs-cluster. Then there's the realtime events system and VOD recording apparatus comprised of futureporn-realtime, futureporn-scout, and futureporn-capture. Then there's the process for long running tasks which I call futureporn-qa. QA being short for, "Quality Assurance." futureporn-qa is the process that generates video thumbnails for vods that are missing them.
Lately, I've been carrying out a much needed refactor of futureporn-qa. It was written quickly and it works, but it wasn't the super robust and high quality software that I wanted it to be. The way it was coded was that there were several discreet modules which each defined a task they would perform. For example, one of them was generateThumbnail. Another was identifyVodIssues. This modular approach is great for code readability and maintainability, but there was one problem. The interface of each of these modules was completely unique.
Instead of having a common way to run each module, each module had a unique way of running. This was confusing to work with as more modules entered the fray, since I'd have to remember or refer to documentation on every unique way to run the module.
In the ongoing refactor, I've created a standardized way of running each module. Going forward, every module will adhere to the standard of having a runTask() function, which makes each module predictable and easy to use from a developer perspective.
futureporn-qa is becoming a more reliable component in Futureporn's system, and that is the goal for the second half of October and the first half of November.
Dashboard
The first step towards improved reliability is monitoring. I need the insight to see when systems are down or misbehaving, and track that behavior over time. I think I've wrote about Grafana previously, but I'll write about it again because it's such a useful tool. I've dusted off an old monitor and added it to my workstation as a dedicated dashboard.
This second monitor stays on 24/7 and gives me a quick view of system status. Green means good on key metrics like website availability, SSL certificate validity, DNS lookup speed, ipfs service state, repo size, and disk usage. In the few days that I've added this dashboard, I've already caught errors that might have gone unnoticed otherwise.
status.futureporn.net
Previously, I was using Upptime.js, an HTTP status monitor powered by GitHub's continuous integration backends. It's kind of a clever hack of using Github's systems in a way they weren't designed, but it definitely works in the right scenario. Lately, a problem emerged where I was getting many false alarms. Dozens a day. I was experiencing alert fatigue after getting the e-mail describing the problem, going to investigate, and finding nothing wrong.
Granted, the upgrade from the old Alpine.js site to the new Next.js site has not been smooth sailing. There were outages, lost features, broken functionality and revision after revision to get the site back to stability. Some of the new changes may have triggered errors via Upptime's service checks, I'm not sure. Regardless, It's something I've been wanting to do which is to get off the hackish Upptime and onto a monitoring service that's laser focused, purpose built and dedicated to the task of monitoring HTTP systems.
I still love Upptime and I recommend it for small projects. However for Futureporn, I've discontinued using it and replaced it with status.futureporn.net powered by UptimeRobot. This might eventually get replaced by Prometheus/Grafana which I'm already using to track metrics, but for now UptimeRobot is doing a great job at creating a simple status page.
Costly mistakes
I had a major slip-up this month where I incurred an accidental $560 bill via Mux's video transcoding service. The problem arose after I refactored futureporn-qa's allocateMuxAsset.js module. In Mux's dev environment, I didn't think to implement rate limiting in my code which adds VODs to Mux. What happened was that all 270+ vods were instantly added, incurring hefty Mux transcoding usage fees.
Futureporn's video source CDN1 is powered by Mux. This is a great service which offers video transcoding, storage, and CDN, with a great API and lots of video player integration options. They take a load off the developer by creating a video platform as a service. This lets the developer spend more time on implementing other parts of their software stack. This is great for me, because I can spend my time savings on improving Futureporn UI or backend. I have been loving everything about Mux except for one thing-- their billing. Many PaaS companies offering services have a feature for setting hard caps on monthly usage. If this cap is met, the service gets cut off and any further requests are responded to with something like 509 Bandwidth Limit Exceeded.
Mux does not have anything like that. Instead, the developer is expected to pay for every usage by every visitor, even if it's over budget. So far, I've mitigated any potential overbudget issues by making CDN1 accessible to patrons only. The bill for this usage has been entirely within my means (thanks to patrons!), but I'm always afraid of a possible black swan event that runs up the bill so high that I quite literally go bankrupt.
This is unlikely, but it's possible. And as Murphy's law states, If it can happen, it will happen.
Vertical integration
The more I learn about software stacks, microservices and platforms as a service (PaaS), the more I realize that Futureporn's long-term success depends on taking on more responsibilities. Mux has been a giant on whose shoulders I can stand, but eventually I'll have to get off those shoulders and become a giant myself. This means running my own transcoding machines, storage, and content delivery network in a way that is performant and competitive. It's a ways off, but it's on my radar at this point.
Erroneous dates
An issue exists where one or more VOD timestamps are incorrect. This issue was caused when I thought I was supposed to enter the timestamp in UTC, but I entered the timestamp in my local timezone. It wasn't until later when I discovered my mistake. Because there are so many vods, it's not immediately clear which VODs are affected.
My plan for fixing these errors is a multi-step process. I think of this process like completing a big quest in an RPG, one with several intermediate steps. Each of those steps is a mini-quest in it's own right which will take time and dedication. It's going to take awhile, but it's going to feel really good when the overall quest has been finished.
The first step is implementing Stream History, a feature I've written about previously. This will give us good data consisting of accurate timestamps for past livestreams. With this data, I can determine all the VODs with incorrect dates.
The second step is implementing useful 404 pages which redirect visitors who clicked on links with incorrect dates
The third step is to correct the dates in the affected VODs.
In conclusion
The next 30 days are dedicated to bugfixes and improving site reliability. If there are any annoying bugs that you want fixed, describe them in the comments below and I'll add them to my issue tracker. I always appreciate negative feedback because it helps me focus on things that need improvement. As always, thank you for your support!
