S3FS to Mount S3 Buckets (NFS)

This article shows the use of https://mydomain.au/content rather than a subdomain https://content.mydomain.au. There is no need for a subdomain when using S3 files.

URL’s such as https://photographybyshaw.au/aws/content/brisbane-river-northshore/DSD_4626.jpg with my own files, are far better for search engines than a fully qualified S3 Bucket URL such as https://s3.ap-southeast-2.amazonaws.com/myS3bucketname/brisbane-river-northshore/DSD_4626.jpg.

The URL is able to keep protection so that it is only viewable from the website, rather than directly accessing the true bucket URL.

The information below shows how to install the s3fs software, the shell script required to mount it, the crontab error checking for infrequent situations where the “nfs” mount point is corrupt, and examples of how we may use the mount point to access documents or images.

As an aside, all of my archived websites place the wp-content/uploads files onto S3 buckets without any need for scripting.

That is, if we create a bucket, say s3mydomain.au, and create a mount point called /var/content that points to this bucket, we can create a soft link:

/var/www/html/some_archive/wp-content/uploads -> /var/content/some_archive/wp-content/uploads

For calling documents or photographs, we could use:

/var/www/html/content -> /var/content, and from there access subdirectories with images, such as /var/content/brisbane and so on, or
/var/www/html/content/mydocuments -> /var/content/mydocuments.

This gives a sense of what we are doing. There is no configuration of CDN in this case, or use of problematic WordPress plugins that access buckets.

Something to keep in mind, there is a limit on the size of a .tar backup file in an S3 bucket, so if you have thousands of photographs, you create several tar files. S3 bucket speed it great, and practical for use compared to other cloud systems I have tested.

Install the software…

dnf -y install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel
cd /home/ec2-user
git clone https://github.com/s3fs-fuse/s3fs-fuse.git
cd s3fs-fuse
./autogen.sh 
./configure --prefix=/usr
make
make install

[let's create a mount point...]

cd /var
mkdir content
chmod 2775 content
cd /var/www/html
ln -s /var/content content
ls -la

Now create an IAM role , with any name such as s3admin.

Give it these permission policies:

AmazonS3FullAccess
AdministratorAccess
CloudFrontFullAccess

I add Cloudfront in my own role as I use that elsewhere.

Now add this role to the EC2 instance under the EC2 console, Instances (then click on the instance) > Security > Modify IAM Role

Using the above examples, this is how you mount the bucket:
(You would use your own region, bucket name, and so on.)

s3fs -o iam_role="s3admin" -o use_path_request_style -o url="https://s3-ap-southeast-2.amazonaws.com" -o endpoint=ap-southeast-2 -o dbglevel=info -o curldbg -o allow_other -o use_cache="" s3mydomain.au /var/content

You may notice caching is set to null. Otherwise copies of files are made to your instance under /tmp, which is not good!

Then issue the Unix command to view the mount point.
df

You may add a test file to your bucket from the Amazon S3 console, and try viewing it with:

ls /var/content/*

If the mount point gets “twisted up” during your installation, you have to restart the instance from the instance console.

When you add files to the mount point, simply use Unix commands. For example:

cd /var/content

mkdir documents

chmod 2775 documents

cd /home/ec2-user

mv mydocument /var/content/documents

cd /var/content/documents

chmod 744s myfile

You can experiment with permissions.

Here is an example of crontab, to keep a check on the mount point:

15 1 * * 1 s3fs -o iam_role="s3admin" -o use_path_request_style -o url="https://s3-ap-southeast-2.amazonaws.com" -o endpoint=ap-southeast-2 -o dbglevel=info -o curldbg -o allow_other -o use_cache="" s3mydomain.au /var/content >/dev/null 2>&1

Here is a shell script you can manually use, /home/ec2-user/remount.sh with file permissions of 775.

#!/bin/sh
s3fs -o iam_role="s3admin" -o use_path_request_style -o url="https://s3-ap-southeast-2.amazonaws.com" -o endpoint=ap-southeast-2 -o dbglevel=info -o curldbg -o allow_other -o use_cache="" s3mydomain.au /var/content 
exit

Here is an example of a crontab entry to check the mount point is not corrupted, and if so to remount it and send you a courtest e-mail via postfix…

5 * * * * /home/ec2-user/mount_error.sh

Now create the shell scripts below. Let’s assume you have a general log file called /home/ec2-user/info.log that your shell scripts can use as required.

cd /home/ec2-user
touch t1.dat
vi /home/ec2-user/mount_error.sh

#!/bin/sh
:>/home/ec2-user/t1.dat
a="ok"

s3fs -o iam_role="s3admin" -o use_path_request_style -o url="https://s3-ap-southeast-2.amazonaws.com" -o endpoint=ap-southeast-2 -o dbglevel=info -o curldbg -o allow_other -o use_cache="" s3mydomain.au /var/content >/home/ec2-user/t1.dat 2>&1
a=`grep "Transport endpoint is not connected" /home/ec2-user/t1.dat|awk -F: {'print $3'}|awk  '{print $1}'`
if [ "$a" = "Transport" ] ;
then
        fusermount -u /var/content
    s3fs -o iam_role="s3admin" -o use_path_request_style -o url="https://s3-ap-southeast-2.amazonaws.com" -o endpoint=ap-southeast-2 -o dbglevel=info -o curldbg -o allow_other -o use_cache="" s3mydomain.au /var/content >/dev/null 2>&1
    d2=`date`
        echo "/var/content error" $d2 >> /home/ec2-user/info.log
sudo /usr/sbin/postfix start
sleep 2
sudo /usr/sbin/postfix reload
sleep 2
sudo /usr/sbin/sendmail -f admin@mydomain.au admin@mydomain.au < /home/ec2-user/mount_error.txt
sleep 5
sudo /usr/sbin/postfix stop
else
    :
fi

exit

[save and exit the editor]

vi mount_error.txt
From: admin <admin@mydomain.au>
Subject: mydomain.au s3fs was down
mydomain.au s3fs was down and is now remounted
.

[You must have a single blank line below the fullstop above. Save and exit the editor]
[You would of course use your own domain name and email address, and already have postfix running correctly.]

chown root mount_error.sh mount_error.txt t1.dat
chgrp ec2-user mount_error.sh mount_error.txt t1.dat
chmod 775 mount_error.sh
chmod 777 mount_error.txt t1.dat

[You may test the above script...]

sh -x ./mount_error.sh

We now come to the point of showing examples to use these mount points. This can be quite detailed and depends on your own programming skills.

Example 1 – list documents.

[Use your won font family, domain name, and icon images. I will supply the images as well for you.
We will call this script from PHP, so we use single quotes in many places ratehr than double quotes.]

cd /var/www/html
vi s3ls.sh

#!/bin/sh
IFS=$'\n'
font="Open Sans"
cd /var/content/$1
for a in `ls -l --time-style=+"%d %m %Y"`
do
p="                      "
b=`echo $a|awk '{print $5}'`
bt=`echo bytes: $b$p|cut -c 1-17`
bz=`echo $bt|awk '{print $2}'`
if [ "$bz" == "0" ] || [ "$bz" == "" ]
then
:
else
file=`echo $a|awk '{print $9" "$10" "$11" "$12" "$13" "$14" "$15}'|sed 's/^ *//;s/ *$//;s/  */ /;'`
e=""
e=`echo $file|awk -F. '{print $NF}'`

case $e in
pdf)
ext="<img src=https://mydomain.au/wp-content/uploads/adobe-reader.jpg width=20 height=20 alt=mydomain.au />"
;;
docx|doc)
ext="<img src=https://mydomain.au/wp-content/uploads/msword.jpg width=20 height=20 alt=mydomain.au />"
;;
xlsx|xls)
ext="<img src=https://mydomain.au/wp-content/uploads/excel-icon.jpg width=20 height=20 alt=mydomain.au />"
;;
pptx|ppt)
ext="<img src=https://mydomain.au/wp-content/uploads/ppt-icon.jpg width=20 height=20 alt=mydomain.au />"
;;
png|jpg)
ext="<img src=https://mydomain.au/wp-content/uploads/media-icon.jpg width=20 height=20 alt=mydomain.au />"
;;
mp3|aac|wav|m4a|mpeg|ogg)
ext="<img src=https://mydomain.au/wp-content/uploads/audio-icon.jpg width=20 height=20 alt=mydomain.au />"
;;
mov|avi|mp4|mpeg4|3gp|3gpp|3gpp2|3gp2|quicktime)
ext="<img src=https://mydomain.au/wp-content/uploads/movie-icon.jpg width=20 height=20 alt=mydomain.au />"
;;
*)
ext="<img src=https://mydomain.au/wp-content/uploads/file-icon.jpg width=20 height=20 alt=mydomain.au />"
;;
esac

pre="<img src=https://photographybyshaw.au/aws/wp-content/uploads/null-icon.jpg width=0 height=0 alt=axoncpn />"
d1=`echo $a|awk  '{print $6}'`
d2=`echo $a|awk  '{print $7}'`
d3=`echo $a|awk  '{print $8}'`
dt="date: $d1 $d2 $d3"
# echo $dt $a
echo "$dt $bt $ext file: <a href='https://mydomain.au/content/$1/$file' target=_blank><span style=text-decoration:underline;color:#444;font-family:$font;>$pre$file</span></a>"
fi
done
exit

I create a PHP shortcode with the following code. I use the plugin called ‘Add Shortcodes Actions And Filters’ from the author Michael Simpson. There are other plugins that do the same thing. Let’s call it ‘s3listfiles’.
This plugin will need you to tick the boxes, Activated, Shortcode, Code echoes output.

$place = "$atts[opt]";
$command = "/var/www/html/s3ls.sh $place 2>/dev/null";
$output = shell_exec($command);
echo "$output";
PHP_EOL;

You will notice how there is only one echo output.
Once this is done, you call the shortcode on a WordPress page.
For example:
[s3listfiles opt=”mydocuments”] where you place these square brackets around this code.
We have only programmed for one subdirectory below the bucket’s root directory. So, you could have directories like, personal, public, projectX, projectY and so on, but not /var/content/public/projectX. Rather, you would have /var/content/projectX.

I format the shortcode into a table format. For example:

<div style="background-color: #d6d6d6; padding-left: 20px; padding-right: 20px;">
<table style="text-align: left; background-color: #d6d6d6;">
<thead>
<tr>
<th style="color: #333; text-align: left; padding-top: 20px;" align="left">My Documents</th>
</tr>
</thead>
<tbody>
<tr style="text-align: left;">
<td style="color: #444; text-align: left; font-family: Courier, Times,sans-serif;" align="left">
<pre>[s3listfiles opt="mydocuments"]</pre>
</td>
</tr>
</tbody>
</table>
</div>