This past summer in-between semesters at school I set out to work on a project I’ve wanted to work on for a while. Some time ago I extracted the hard drive from my childhood TiVo before sending the actual unit off to an e-waste facility during a round of spring cleaning. My thought at the time was that hard drives take up way less space than an aging TiVo unit, and one day I may want to get the data off the drive. So with the free time of a summer break, I set out to work on this project.
However, what I couldn’t predict going into this project was the chaos of life that would occur around this project. I set out to work on this project while moving between apartments, Apple recalling my MacBook, and the everyday stressors of a college student going into their fourth year at university.
Although I didn’t accomplish my goal of extracting the video streams from the drive, I did experiment with creating a write-up while working on the project. Below, is that writeup:
Growing up our family used a Series One TiVo (Sony SAT T-60) to watch and record TV. We used it for probably ten years before upgrading to a non-TiVo HD DirecTV DVR, and eventually just cutting the cord all together a few years later. A few years after upgrading we decided to throw out the old TiVo box during a round of Spring cleaning.
However, being the data nerd I am at heart I made sure to remove and stash the hard drive from the TiVo before throwing the unit away. At the time I figured a hard drive is a hard drive, and one day I may want to try and retrieve the contents from the drive. Well, that day has come!
Below is a log of my attempt to extract the recordings off of the hard drive of my childhood TiVo:
Before we begin, let’s do some prep work.
Our SAT T-60 TiVo came with a 40 Gigabyte Maxtor IDE Drive, this is the same hard drive I removed. TiVopedia and Wikipedia both have high-level write-ups about TiVo hard drives and the Media File System (MFS) on them.
Luckily for me, the TiVo modification community has existed for a long time and has a lot of open source code still available within it. Most notably, MFS Tools should be helpful for learning how to interact with the file system.
Before I even start attempting to extract the recordings, I want to create a clone of the drive. It is an 18-year-old mechanical hard drive that has been sitting idle for several years now, and I don’t want to put too much stress on the drive. Interacting with a disc image of the drive will be more convenient and safer.
After booting my MacBook into Manjaro with the drive attached using an IDE-to-USB converter cable we’re good to go. Gnome Disks shows the 40 Gigabyte drive accessible at
/dev/sdd with no mountable partitions.
dd to create an image of the drive on a 1 Terabyte External USB Drive:
sudo dd if=/dev/sdd of=/mnt/External/tivo_hdd.iso status=progress
After some time I have a 40 GB disk image and can power off the drive and put it away.
Before powering down the drive and putting it away, I tried using a statically linked version of
mfs_tools available on their SourceForge Project Page but wasn’t able to yield anything immediately helpful. Running
mfs_tools backup on the drive only yields a 1 GB binary file, and running
mfs_tools backup -a to capture all streams on the drive fails.
Will try working with the ISO tomorrow.
Since MFS Tools won’t be able to extract the MPEG streams I want for me, I’ll just have to code some tools myself.
This seems like a good opportunity to learn some Rust. Rust seems like a good language to write the tools I’ll need for this in 2019, given it’s a low-level language known for its memory management.
rustup update after installation, and reading some of the preloaded language book using
rustup doc --book I’m feeling good to start coding with Rust.
On June 20, 2019 Apple recalled the battery in my MacBook Pro. (Apple Support Article on the Recall) Because of this, I’m going to have to spend the day backing up my laptop and getting the data I need off of it and usable on my Surface Go. That includes this project and this write up.
After backing up my MacBook with Time Machine, I committed the Rust codebase I created for this project to a GitHub repo and this write up to a draft document on my website’s CMS. Going to be coding on Windows until I get my laptop back. When it comes to coding, I am more comfortable with the macOS and Linux environments on my MacBook so this might be an added challenge with this project moving forward.
Left my MacBook with Apple after meeting with a Genius Bar Technician today. Hopefully should have my primary computer back in 1-2 weeks with a brand new battery.
Admittedly, not much work on this project is going to get done this weekend. In between my primary machine getting recalled, needing to configure a coding environment on my Surface Go, and moving apartments haven’t had a ton of time to do much on this aside from doing some research in my free time.
The last thing I implemented in this project was functionality to byte swap the ISO file I made of the hard drive. While this was a fun and very helpful exercise for getting more comfortable with Rust and the data I’m working with, It’s not very practical for the long term. Going forward I think I’ll just let the
byteorder crate handle the BigEndian-ness of the TiVo’s hard drive data.
Looking through the
mfstools source code has been very helpful in figuring out how to interact with the TiVo drive data. Using it and a hex viewer I’ve been able to see around the general binary of the ISO file I created. The first helpful thing I’ve found was using the first byte of the drive to determine if it is a valid TiVo hard drive, and if the drive is byte swapped (Little Endian or Big Endian.)
mfstools code base helped show that TiVo was using the Apple Partition Map for its hard drives. I was confused why the
mfstools author kept referring to “Mac partitions” on the TiVo, until I Googled the Hex value in the
MAC_PARTITION_MAGIC constant and found a book chapter on the forensic analysis of Macintosh hard drives explaining that
0x504d was used as a signature byte on drives using Apple Partition Map.
I speculate that TiVo chose to use the Apple Partition Map on its drives because they were using PowerPC processors and Apple had already created a partition map for their PowerPC machines known to work well with media. Because the Linux kernel (which the TiVo OS was based on) supported Apple partitions, this would mean less work for TiVo and they could spend more time perfecting the experience of the DVR instead of worrying about the low-level technical implementation of it.
Loading up the byte swapped ISO image I created into a Hex Viewer yields the following:
Here we can see
0x1492 in the first two bytes of the image, so we know this should be a valid TiVo disk image. As well, we can see
0x504d in the 200 & 201st bytes indicating an Apple Partition Map Partition Entry.
Thanks to the Wikipedia entry on APM, we know that the drive is likely divided into logical blocks of 512 bytes. Setting the hex viewer to display bytes in rows of 512 bytes yields the following:
Cool. Now we can more easily see the 13 partition entries on the drive. We can see the partition that contains the partition map we’re currently interacting with. As well, as eight partitions related to the Linux install on the drive. The two
Ext2 partitions on there could be cool to poke around one day. We can see two MFS application partitions, and most importantly two MFS media regions. Awesome. Being able to visualize the disk like this has been incredibly helpful, and being able to reference the partition map entry layout from Wikipedia is going to be very helpful as we charge forward.
Note from Future Kepler: Because they were just lists of commits done in the oViT repo that day and not explaining what was worked on or learned those days, days 7 through 17 have been removed from this post. (October 12, 2019)
It’s been a bit since I last did anything with this project. I’ve spent some time working on my website and have gotten cognitive distance from what I’m trying to do here. Earlier today I got to watch a really great talk from Carol Nichols called Rust: A Language for the Next 40 Years. It was a great talk with a heavy dose of computational history. As someone that really values the intersection of technology and society, the comparison of systems languages to American railroads was amazing – especially watching it a day after the U.S. House Committee on Financial Services discussed the Rust language in respect to Facebook’s Libra.
While watching Carol’s talk it occurred to me that while writing all of oViT by hand in Rust definitely has its strengths, it is slower and more painful process. If I want a proper tool sooner than later, I should treat
mfs-utils as legacy code libraries and use the already written functions that I haven’t implemented yet.
Going to use the
cc crate to build the existing C code into oViT. Because it is very likely I’m going to need to patch and modify the C code for these projects, I’m not going to use git submodules to clone the codebases in. I’ve cloned
mfs-utils from the elitak/mfs-utils repo on GitHub, and
mfs-tools from their SourceForge page. This should ensure I am working against the most recent copies of both codebases; with
mfs-utils’s last commit in 2009 and
mfs-tools’s last commit in 2015.
Three Hours Later
Nope. Giving up on the FFI & Legacy Code idea. Good idea in concept, maybe not for this project right now. Got stuck with the code getting compiled, but not linked. Going to move all of today’s commits into a separate branch for later.
Sadly, I never wrote anything more on this write-up after the 18th. I did continue to work on oViT a bit more that month, but had to move on and work on other things as school started back just a few weeks later in August. The oViT source code can be found in its GitHub repo. One day I hope to get back to this project, but I don’t foresee that happening for a while.
This project was often very frustrating to work on. Going into the project I knew that much of the material I was going to be referring to wasn’t going to be from first-hand sources. But as I continued to research and work on this drive I kept encountering the issue of link rot in the increasingly aging pages I was looking for. While Tivopedia and The TiVo Community Forum remain online after all of these years, many of the hyperlinks in the 14-18-year-old posts I was reading had simply gone dead. People hosting information, software, and source code on personal websites that they have long stopped paying for hosting, domain registration, and otherwise abandoned. While looking for information from others who reverse-engineered the first-generation TiVo before me, I often related to Randell Monroe’s xkcd #979:
There is a bright side to this experience though. Before starting work on oViT, I was incredibly burnt out of software development. Working on oViT allowed me to work on and research an incredibly interesting problem, allowed me to learn a new programming language in Rust, and reignited my interest in website development. Writing this write-up gave me a reason to build my website up, and get back into development.
Burnout sucks. Link rot sucks. Both are incredibly important to overcome. Burn out is inevitable when you’re constantly spinning your wheels on everything in life (academic and professional.) And sadly, link rot is inevitable with the current state of the web and the often fleeting nature of information. We must do everything in our power to preserve our personal energies and the important work we do with that energy.