Troubleshooting backup problems with Idera R1soft under Centos (part 1)

Photo by GotCredit

Everyone knows it’s a good idea to keep backups, but it can be difficult to get them up and running, and even harder to keep them staying that way. On top of that, the backup process can make your server slow to a crawl if you have typical hard drives instead of SSDs. R1Soft is a popular backup solution in the web hosting industry which addresses the performance concern by continuously monitoring the OS for file system changes, so that it only has to back up the data that has changed.

Unfortunately, R1Soft does not address the larger concern of backup reliability, as R1Soft backups are prone to failure and the entire system needs to be coddled along under a variety of normal conditions. The most common problems you’ll encounter are either that the CDP agent is no longer active (usually after a kernel update), or, that at some point the file system or a disk safe had run out of space. I am going to address the issue of the CDP agent failure in this blog article, and we’ll discuss the problems that can happen due to lack of disk space in a future blog article.

In this article we are discussing specifically R1Soft 3.0, but much of the information applies to both older and newer versions. In R1Soft, the CDP Client Agent software is necessary in order to keep track of all changes on the file system. If the agent is not running at any given time, it will not be able to track these changes, which can cause serious problems with backups. After updating your kernel, the CDP Agent will be deactivated and needs to be manually reloaded. In version 3.0, what happens next is that the CDP server will realize it “missed” some file updates, and will perform a full block scan to get back up to date. Although this process is slow, it does ensure that your backups are not corrupted. In 2.0 and older versions, the CDP Agent was not smart enough to know whether or not it needed to do a full block scan, and so would silently corrupt disk safes. For that reason we strongly urge anyone to upgrade to at least 3.0 if they are using a 2.x or older version.

So, after making sure you are using at least 3.0, you’ve upgraded your kernel and CDP agent is no longer running. It’s time to get that reinstalled. For the purposes of this article, we assume that you’re running Centos on the servers that you are backing up.

The process of reinstalling the CDP agent is generally quite a bit easier than installing R1Soft the first time, but it still helps to know exactly what to do. From an SSH prompt, you will need to first make sure the kernel headers libraries are available by running this command:

yum install kernel-devel kernel-headers

Reply “yes” when prompted, and if everything goes well you can continue. If your server is running the Xen kernel, you will require the kernel-xen-devel library as well, so also run this command:

yum install kernel-xen-devel

Next, you need to rebuild the kernel module. To force a custom build using your kernel headers and source files:

r1soft-setup --get-module --no-binary

If it still doesn’t work, then the headers downloaded might be for a different kernel version than the version running. You may need to reboot into the updated kernel before you can properly install the r1soft module.

After you have successfully acquired a CDP Agent kernel driver you will need to restart the agent to load the module, and configure the agent to load at bootup. From SSH run the following commands:

/etc/init.d/cdp-agent restart
/etc/init.d/cdp-agent start
chkconfig --level 2345 cdp-agent on

Then you can check that the module is loaded properly with the lsmod command in Linux. Run the following command and note the output:

lsmod | grep hcpdriver

If everything looks good, you should be able to initiate a backup, which will then perform a full block scan to ensure the consistency of your backups. This will take some time and use a lot of disk i/o, so it is best to run this during off-peak hours for your server.

If you have any questions about this information, or want to learn about ioflood.com dedicated servers, email us at sales [at] ioflood.com