Supporting extFAT and rsync to an exFAT partition in Linux

Ok this was a fairly annoying. exFat isn’t supported in Linux by default. But no fear it can be supported by installing exfat-utils and exfat-fuse packages from the official repos:

http://askubuntu.com/questions/370398/how-to-get-a-drive-formatted-with-exfat-working

Sweet, ok so now rsync should surely work right? Nope, not well anyway. Here’s why:

http://blog.marcelotmelo.com/linux/ubuntu/rsync-to-an-exfat-partition/

The above post explains it perfectly but the main problem is that the Linux exFAT does not cope well with the switches, particularly archive -a and spits out loads of chmod permission errors, but you can fix this by running rsync as follows:

[code]rsync -rltDv [SRC] [DESTINATION][/code]

Epic!

Useful Rsync BackUp Command

Rsync is a utility software and network protocol for Unix-like systems that synchronizes files and directories from one location to another while minimizing data transfer by using delta encoding when appropriate.

It is a very useful too for backing up your data which is an absolute must.

Here is my command for backing up a users system on a Mac.

[shell]
rsync -auv –delete –ignore-errors –force –exclude ‘Movies’ –log-file=backup.log /Users/UserName/ /Volumes/UserName/BACKUP/UserName/
[/shell]

Lets explain some of these options I have used.

  • a : archive mode
  • u : update, skip files that are newer on the receiver
  • v : verbose, increase verbosity
  • –delete : This tells rsync to delete extraneous files from the receiving side (ones that aren’t on the sending side), but only for the directories that are being synchronized.
  • –ignore-errors: Delete even if there are I/O errors (this is needed if running on a live system)
  • –force : This tells rsync to force deletion of dirs even if not empty.
  • –exclude : Exclude a folder, use –exclude-from then file name with a list of directories if there are multiple
  • –log-file : Create a log file

Run this once a day and you should be all good, obviously first run takes a while but after that only new files will be copied across to your backup location and run time will be dramatically reduced.

Mac OS X launchd Automount Network Shares

As Apple seem to like to dictate what they should do with the world, they kindly got rid of the robust amazing cron feature of linux, oh and they also seem to of got rid of /etc/fstab … nice so basically you now have to make shell scripts and load them with launchd to do anything automated.

Thanks apple for making life harder as ALWAYS.

Well as this is quite mind boggling upon first look it needs to be written down as my brain just can’t fit all in.

Basically you need to do three things.

  • Create a shell script containing your commands you want to run.
  • Create .plist file with a load of jazz in it which can seem pretty confusing but not to apple, but more specifically contains the path to your shell script and the frequency you want it to run.
  • Copy the .plist file to /Library/LaunchDaemons/
  • Load the .plist file via launchctl load

The shell script:

[shell]
#!/bin/bash

local_admin_user="xxx" # an admin user name on the computer running this script
local_admin_pswd="xxx" # the password for the above user
mount_point="/Volumes/Myshare" # the name the mounted share point will be known as on this computer

host_name="my-host.local" # the domain name or IP of the system hosting the share point
share_point="MyShare" # the name of the remote share point
share_owner="userX" # the user name of the owner of the remote share point
share_pswd="xxx" # the password for the above user

# IMPORTANT:
# If the share_pswd contains any symbol characters, they need to be escaped.
# So, instead of: share_pswd="abc&123#"
# Do this: share_pswd="abc\&123\#"

#IMPORTANT
# Because I am running this at startup, the mount kept failing because the network wasn’t ready yet
# Adding this delay fixes that issue, only took 3 hours of debugging…. doh
sleep 15

# let’s clean up just in case the mount was dropped
echo $local_admin_pswd | sudo -S -u $local_admin_user umount $mount_point
echo $local_admin_pswd | sudo -S -u $local_admin_user rmdir $mount_point

# basic steps to mount the private share point
echo $local_admin_pswd | sudo -S -u $local_admin_user mkdir $mount_point
echo $local_admin_pswd | sudo -S -u $local_admin_user chown $local_admin_user $mount_point
echo $local_admin_pswd | sudo -S -u $local_admin_user chmod +rwx $mount_point
echo $local_admin_pswd | sudo -S -u $local_admin_user mount_afp afp://$share_owner:[email protected]$host_name/$share_point/ $mount_point
[/shell]

The .plist file

[html]
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.apple.automount.readynas</string>
<key>ProgramArguments</key>
<array>
<string>/Scripts/automount_readynas.sh</string>
</array>
<key>UserName</key>
<string>root</string>
<key>UserGroup</key>
<string>wheel</string>
<key>RunAtLoad</key>
<true/>
</dict>
</plist>
[/html]

The copy command

[shell]
sudo cp /Scripts/com.apple.automount.readynas.plist /Library/LaunchDaemons/
[/shell]

Now don’t forget to load the .plist file

[shell]
sudo launchctl load /library/LaunchDaemons/com.apple.automount.readynas.plist
[/shell]

So with all that done, restart your mac and see if it works!

Thanks to these useful sources:

http://answer123.com/misc/launchd_notes.html
http://www.gregwillits.ws/articles/using_launchd_to_mount_afp_share_point_upon_startup

Leverage Browser Caching

Here is a very useful .htaccess snippet to leverage browser caching and help improve website speed.

[code]
<IfModule mod_expires.c>

# Enable expirations
ExpiresActive On

# Default directive
ExpiresDefault "access plus 1 month"

# My favicon
ExpiresByType image/x-icon "access plus 1 year”

# Images
ExpiresByType image/gif "access plus 1 month"
ExpiresByType image/png "access plus 1 month"
ExpiresByType image/jpg "access plus 1 month"
ExpiresByType image/jpeg "access plus 1 month"

# CSS
ExpiresByType text/css "access 1 month”

# Javascript
ExpiresByType application/javascript "access plus 1 year"

</IfModule>
[/code]

Source: http://fortheloveofseo.com/blog/performance/leverage-browser-caching-how-to-add-expires-headers/

Linux File Server Problems

Ok I just spent the last few days trying to configure a simple Linux file sharing server via SAMBA. This is usually a trivial task installing CentOs but for some reason this time round it was a joke. These are the problems I faced and overcome eventually.

  • First the on board NIC card just wasn’t working after install, issues with drivers. So tried installing the latest version of CentOs 6.0 but still it didn’t work. (Fixed it by having to admit defeat and buy a new NIC PCIe Card)
  • Then I noticed once trying to install a second hard drive that the system was treating them as IDE with /dev/hda /dev/hdb etc. The drives are SATA so should be referred to as /dev/sda /dev/sdb. The transfer rate was ridiculously slow so due to this being set wrong on install. Finally I found out this was down to a BIOS setting. The SATA controller was set as IDE and the only other option I had to try was ADHCI. I chose this option reinstalled again and finally the drives were being read as /dev/sda /dev/sdb and the install completed in a bout 15mins. About a million times quicker.
  • Then when setting up SAMBA I could see the shares on a windows machine but couldn’t access them. Very annoying, eventually this was down to SELinux which is an additional security protocol that was preventing other users from accessing the share. I disabled this completely and it fixed the problem.

The above problems seem small but the time it took with trail and error debugging was ridiculous, plus I think I installed CentOs about 6 times before finally having success. I have never had this issue before with other Linux file sharing box’s I have made. I guess this motherboard was just not that Linux loving. Hopefully now I will have the answer if these issues ever arise again.

Useful Crontab Commands

I always end up searching to double check how to run programs at specific times using the Cron tab, so here are my useful ones.

First here’s a key to how crontab works.

*     *     *   *    *        command to be executed
-     -     -   -    -
|     |     |   |    |
|     |     |   |    +----- day of week (0 - 6) (Sunday=0)
|     |     |   +------- month (1 - 12)
|     |     +--------- day of the month (1 - 31)
|     +----------- hour (0 - 23)
+------------- min (0 - 59)

Here are some commands to edit the crontab, note knowledge of using vi is pretty important here 🙂

Edit your crontab file, or create one if it doesn’t already exist.
[sourcecode language=”sh”]# crontab -e[/sourcecode]

Display your crontab file.
[sourcecode language=”sh”]# crontab -l[/sourcecode]

Remove your crontab file.
[sourcecode language=”sh”]# crontab -r[/sourcecode]

Here are some practical examples.

Daily at 4am.

[sourcecode language=”sh”]

0 4 * * *  path_to_program_to_run

[/sourcecode]

Weekly at 4am on Monday.

[sourcecode language=”sh”]

0 4 * * 1  path_to_program_to_run

[/sourcecode]

Monthly at 4am on the 1st of the month.

[sourcecode language=”sh”]

0 4 1 * *  path_to_program_to_run

[/sourcecode]

Monthly at 4am on the 1st and 15th of the month.

[sourcecode language=”sh”]

0 4 1,15 * *  path_to_program_to_run

[/sourcecode]

Public (requires no auth) and Private Shares with Samba

I just found out how to do this and thought I better blog it straight away as I couldn’t believe how easy it was.

I always just ignored the fact that Samba always asked for a user name and password from a windows machine and just ensured users were added to Samba with the same credentials they use to log onto their windows machine.

This is pretty good practise anyway as it allows users to have a private share that they only see when they log onto Samba.

But its also good to have just a public share that doesn’t require background authentication and acts just like when you share a folder on windows.

So enough of me beating around the bush, the key is adding just one line to your ‘/etc/samba/smb.conf‘ file.

Just add this line to your [Global] section.

[sourcecode language=”sh”]

map to guest = bad user

[/sourcecode]

Then if you haven’t already ensure your public share is the following:

[sourcecode language=”sh”]

[Public Dump]
path = /public_dump
writeable = yes
browseable = yes
guest ok = yes

[/sourcecode]

Finally just restart Samba to make the changes take effect.

[sourcecode language=”sh”]

# service smb restart

[/sourcecode]

Thinking about it this makes perfect sense as windows always sends your user name and password in the background when you access a share, so when you try to access the Samba share without being in the Samba user table, Samba will map you as a guest user leaving you with just being able to view the public shares you have set up.

How to Install and Format a new Linux Hard Drive

Anything in Linux is awalys slightly more taxing but is not that hard once you know how.

So first up you need to know whether you are working with a IDE and PATA (formerly just ATA) drive or a SCSI, SATA and USB drive.

Linux refers to IDE and PATA drives with “hdx” and SCSI, SATA and USB drives with “sdx“. Usually, a number is also put at the end of “hdx” or “sdx” to denote different partitions on the same physical drive, but for the purpose of formatting, you only need to know which letter the drive you want to format is.

To see all the drives attached to your system use the following commands.

[sourcecode language=”sh”]ls /dev/hd*[/sourcecode]

or

[sourcecode language=”sh”]ls /dev/sd*[/sourcecode]

In my case I have installed a new SATA hard drive so I am working with ‘/dev/sdb

First we will use ‘fdisk‘ to erase any old partitions on the drive, if you have a fresh new hard drive you wont need to do this.

[sourcecode language=”sh”]fdisk /dev/sdb[/sourcecode]

Now we can type ‘p‘ to see a partition table of the drive. The first line of output from the “p” command will also tell you the size of the drive. This is a good way to double-check that you are working with the correct drive.

To delete any existing partitions, press “d” and then “Enter.” It will ask you which partition number you wish to delete. The number of the partition is the number that follows ‘sdb‘. You can always view the partition table again with the “p” command.

Type “n” and hit “Enter.” Then press “p” to create a primary partition. It asks you for a partition number; enter “1” Now you are asked which cylinder the partition should start at. The beginning of the drive is the default, so just hit “Enter.” Then, you are asked for the last cylinder. The end of the drive is the default, so you can just press “Enter” again.

Now you are back at fdisk’s command prompt. Use the “p” command to check the partition table. You should now see your new partition at the bottom of the output. In the example, it lists as “/dev/sdb1

You now need to set the filesystem type for your new partition with the “t” command. You are asked for the Hex code of the file system you wish to use. We will use the standard Linux ext2 file system, which is “83

Now just issue the “w” command to write your new partition table and exit ‘fdisk

Finally we need to create the file system on the drive. We use the ‘mkfs‘ command.

[sourcecode language=”sh”]mkfs -t ext2 /dev/sdb1[/sourcecode]

Its good practise to now run a file system check on the new drive, we do this with the following command.

[sourcecode language=”sh”]fsck -f -y /dev/sdb1[/sourcecode]

Last but not least we need to mount the drive at boot. We do this by adding an entry into the ‘/etc/fstab‘ file.

[sourcecode language=”sh”]vi /etc/fstab[/sourcecode]

In my case the I want to mount the drive at ‘/meat‘ so we enter the follwing line in the ‘fstab‘ file.

[sourcecode language=”sh”]/dev/sdb1          /meat          ext2          defaults          0 0[/sourcecode]

Thanks to http://www.ehow.com/how_1000631_hard-drive-linux.html

Editing files with vi Commands

If you administrate servers you will almost definitely need to use vi command from the terminal as a GUI is not always possible, and vi is generally quicker to use I find.

These are my useful commands, ‘esc’ = the escape button.

Open a file:
[sourcecode language=”sh”]
vi filename
[/sourcecode]

Edit the file:
[sourcecode language=”sh”]
esc i
[/sourcecode]

Close the file:
[sourcecode language=”sh”]
esc :q
[/sourcecode]

Close and save the file:
[sourcecode language=”sh”]
esc :wq
[/sourcecode]