{"id":10696,"date":"2021-12-06T10:29:35","date_gmt":"2021-12-06T09:29:35","guid":{"rendered":"https:\/\/www.origo.io\/info\/?page_id=10696"},"modified":"2022-11-28T11:49:48","modified_gmt":"2022-11-28T10:49:48","slug":"minio-object-storage-service","status":"publish","type":"page","link":"https:\/\/origo.systems\/info\/stabiledocs\/guides\/minio-object-storage-service\/","title":{"rendered":"Installing MinIO Object Storage Service"},"content":{"rendered":"\n

In this guide we will install an S3 object storage service using MinIO<\/a> to a Kubernetes cluster in Origo OS<\/a>. We will use the Kubernetes stack to quickly fire up a Kubernetes cluster, prepare the virtual disks that will be used by MinIO and then install MinIO to the cluster.<\/p>\n\n\n\n

First install a multi-node Kubernetes cluster<\/strong><\/h2>\n\n\n\n
\"\"<\/a><\/figure><\/div>\n\n\n\n

Install a Kubernetes cluster by following this guide<\/a>.
IMPORTANT: Make sure you install at least 4 nodes instead of the default 2 (as shown in the image above). Also make sure you name the cluster “MinIO”, as this naming is assumed for the commands below (specifically regarding the formatting of drives).
<\/p>\n\n\n\n

After the Kubernetes cluster is up and running, you must set a password for the stabile user in the “security” tab, and allow ssh access from your current IP address.
Once you have done this, head straight to your favourite ssh terminal and ssh to the stack administration server (running the control-plane), to execute all the commands in the steps below.<\/p>\n\n\n\n

\"\"<\/a><\/figure><\/div>\n\n\n\n

The administration server running the control plane is the one with a name ending in “.0” – find its IP address in the dashboard (as shown in the image above).<\/p>\n\n\n\n

Prepare the data disks on all nodes<\/strong><\/h2>\n\n\n\n

To use the data disks attached to each of the nodes in the Kubernetes cluster, we must resize, reset the and disable mouting of the disks (because going forward they will be managed by MinIO). We will do this from the ssh terminal as the user “stabile”, and we will use the utility “stabile-helper”, which allows us to execute the same command in parallel on all servers in a Stack:<\/p>\n\n\n\n

First resize the data disks of all the nodes to the desired capacity – we use 100GB in this example:<\/p>\n\n\n\n

stabile-helper runcommand \"stabile-helper resizestorage 100GB vdb1\"<\/code><\/pre>\n\n\n\n
Change fstab on all nodes to prevent trying to mount the data disks, as they no longer will contain a regular file system:<\/p>\n\n\n\n
stabile-helper runcommand \"sed -i 's\/\\\/dev\\\/vdb1.*\/\/' \/etc\/fstab\"<\/code><\/pre>\n\n\n\n
Then unmount and wipe the data disks on all nodes, so they are ready for MinIO:<\/p>\n\n\n\n
stabile-helper runcommand \"umount \/mnt\/local\"<\/code><\/pre>\n\n\n\n
stabile-helper runcommand \"wipefs -a \/dev\/vdb\"<\/code><\/pre>\n\n\n\n
To verify the data disks have been resized, type:<\/p>\n\n\n\n
stabile-helper runcommand lsblk<\/code><\/pre>\n\n\n\n
Install MinIO operator<\/strong><\/h2>\n\n\n\n
References:
https:\/\/docs.min.io\/minio\/k8s\/reference\/minio-kubectl-plugin.html <\/a>
https:\/\/docs.min.io\/minio\/k8s\/deployment\/deploy-minio-operator.html<\/a><\/p>\n\n\n\n
To install the MinIO operator to your control plane, execute as user “stabile”:<\/p>\n\n\n\n
wget https:\/\/github.com\/minio\/operator\/releases\/download\/v4.2.7\/kubectl-minio_4.2.7_linux_amd64 -O kubectl-minio\nchmod +x kubectl-minio<\/code><\/pre>\n\n\n\n
sudo mv kubectl-minio \/usr\/local\/bin\/<\/code><\/pre>\n\n\n\n
kubectl minio init<\/code><\/pre>\n\n\n\n
Install MinIO client<\/strong><\/h2>\n\n\n\n
The procedure to install the MinIO client is very similar:<\/p>\n\n\n\n
wget https:\/\/dl.min.io\/client\/mc\/release\/linux-amd64\/mc\nchmod +x mc<\/code><\/pre>\n\n\n\n
sudo mv mc \/usr\/local\/bin<\/code><\/pre>\n\n\n\n
Install Krew<\/strong><\/h2>\n\n\n\n
Reference:
https:\/\/krew.sigs.k8s.io\/docs\/user-guide\/setup\/install\/<\/a><\/p>\n\n\n\n
We install the very nice plug-in manager Krew by executing:<\/p>\n\n\n\n
OS=\"$(uname | tr '[:upper:]' '[:lower:]')\" &&\nARCH=\"$(uname -m | sed -e 's\/x86_64\/amd64\/' -e 's\/\\(arm\\)\\(64\\)\\?.*\/\\1\\2\/' -e 's\/aarch64$\/arm64\/')\" &&\nKREW=\"krew-${OS}_${ARCH}\" &&\ncurl -fsSLO \"https:\/\/github.com\/kubernetes-sigs\/krew\/releases\/latest\/download\/${KREW}.tar.gz\" &&\ntar zxvf \"${KREW}.tar.gz\" && .\/\"${KREW}\" install krew<\/code><\/pre>\n\n\n\n
sudo bash -c \"ln -s \/home\/stabile\/.krew\/bin\/kubectl-krew \/usr\/local\/bin\/\"<\/code><\/pre>\n\n\n\n
Fix Kubernetes CA service<\/strong><\/h2>\n\n\n\n
Reference:
https:\/\/github.com\/minio\/operator\/issues\/420<\/a><\/p>\n\n\n\n
The Kubernetes CA service needs a little fix to work properly:<\/p>\n\n\n\n
sudo cp \/etc\/kubernetes\/pki\/ca.crt \/usr\/local\/share\/ca-certificates\/\nsudo update-ca-certificates<\/code><\/pre>\n\n\n\n
Install direct-csi<\/strong><\/h2>\n\n\n\n
Reference:
https:\/\/github.com\/minio\/direct-csi\/blob\/master\/docs\/installation.md<\/a><\/p>\n\n\n\n
We use direct-csi to access the data disks. Install it by executing:<\/p>\n\n\n\n
kubectl krew install direct-csi<\/code><\/pre>\n\n\n\n
Make direct-csi available to kubectl:<\/p>\n\n\n\n
sudo ln -s \/home\/stabile\/.krew\/store\/direct-csi\/v1.4.6\/kubectl-direct_csi \/usr\/local\/bin<\/code><\/pre>\n\n\n\n
Install the driver to the cluster and format the drives:<\/p>\n\n\n\n
kubectl direct-csi install --crd<\/code><\/pre>\n\n\n\n
kubectl direct-csi drives format --drives \/dev\/vdb --nodes minio.*<\/code><\/pre>\n\n\n\n
List the drives being used for MinIO:<\/p>\n\n\n\n
kubectl direct-csi drives list<\/code><\/pre>\n\n\n\n
If all your drives are not listed, repeat the previous format command – you need at least 4 “ready” drives to proceed.<\/p>\n\n\n\n
Install a MinIO tenant<\/strong><\/h2>\n\n\n\n
Reference:
https:\/\/docs.min.io\/minio\/k8s\/tenant-management\/deploy-minio-tenant-using-commandline.html#deploy-minio-tenant-commandline<\/a><\/p>\n\n\n\n
The MinIO tenant is what acually exposes the S3 service etc., so we obviously want to install a tenant.<\/p>\n\n\n\n
kubectl create namespace minio-tenant-1<\/code><\/pre>\n\n\n\n
We now create the MinIO tenant, which provides the actual S3 services. We want to capture the output, since the admin password is only given once – upon tenant creation. However it seems that MinIO writes directly to the TTY, thus disabling regular capture by simply piping stdout – presumably this is a security measure. We bypass this, since we really need this password.<\/p>\n\n\n\n
script -c \"kubectl minio tenant create minio-tenant-1 --servers 4 --volumes 16 --capacity 100G --storage-class direct-csi-min-io --namespace minio-tenant-1\" tenant.out<\/code><\/pre>\n\n\n\n
 cat tenant.out | sed -n -e 's\/.*Password: \/\/p' | grep -oh \"\\S*\" | tee tenant-secret.out 2>&1<\/code><\/pre>\n\n\n\n
Configure MinIO client to connect to our tenant<\/strong><\/h2>\n\n\n\n
We need to add entry to \/etc\/hosts, in order for certificate validation to work properly:<\/p>\n\n\n\n
echo \"`kubectl describe svc minio --namespace minio-tenant-1 | sed -n -e 's\/.*IP:\\s*\/\/p'` minio.minio-tenant-1.svc.cluster.local\" | sudo tee -a \/etc\/hosts<\/code><\/pre>\n\n\n\n
Now configure mc with the password we pub int “minio-secret.out” above:<\/p>\n\n\n\n
mc alias set minio\/ https:\/\/minio.minio-tenant-1.svc.cluster.local admin <tenant-secret.out<\/code><\/pre>\n\n\n\n
If this command fails, please wait a few minutes and try again.<\/p>\n\n\n\n
Then test that mc actually works:<\/p>\n\n\n\n
mc admin info minio\nmc mb minio\/test-bucket<\/code><\/pre>\n\n\n\n
Temporarily port-forward operator console UI<\/strong><\/h2>\n\n\n\n
To access the operator web UI we can temporarily (or permanently in a start-up script) start a proxy:<\/p>\n\n\n\n
kubectl minio proxy -n minio-operator<\/code><\/pre>\n\n\n\n
Temporarily port-forward tenant console UI<\/strong><\/h2>\n\n\n\n
To access the tenant console web UI we can temporarily (or permanently in a start-up script) port-forward traffic to the service:<\/p>\n\n\n\n
kubectl port-forward service\/minio-tenant-1-console 9443:9443 --namespace minio-tenant-1 --address 0.0.0.0<\/code><\/pre>\n\n\n\n
Now point a browser to https:\/\/administration-servers\u2013IP-address:9443 and log in with “admin” and the password found in tenant-secret.out.<\/p>\n\n\n\n
After this it’s a good idea to remove the files containing the password:<\/p>\n\n\n\n
rm tenant*.out<\/code><\/pre>\n\n\n\n
That’s it. You should now have a working MinIO object storage service. Happy S3’ing!<\/p>\n\n\n\n
<\/div>\n\n\n","protected":false},"excerpt":{"rendered":"
In this guide we will install an S3 object storage service using MinIO to a Kubernetes cluster in Origo OS. We will use the Kubernetes stack to quickly fire up a Kubernetes cluster, prepare the virtual disks that will be used by MinIO and then install MinIO to the cluster.<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":7856,"menu_order":17,"comment_status":"open","ping_status":"closed","template":"page-leftsb.php","meta":{"_acf_changed":false,"footnotes":""},"class_list":["post-10696","page","type-page","status-publish","hentry"],"acf":[],"featured_image_src":null,"featured_image_src_square":null,"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/origo.systems\/info\/wp-json\/wp\/v2\/pages\/10696","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/origo.systems\/info\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/origo.systems\/info\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/origo.systems\/info\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/origo.systems\/info\/wp-json\/wp\/v2\/comments?post=10696"}],"version-history":[{"count":147,"href":"https:\/\/origo.systems\/info\/wp-json\/wp\/v2\/pages\/10696\/revisions"}],"predecessor-version":[{"id":12772,"href":"https:\/\/origo.systems\/info\/wp-json\/wp\/v2\/pages\/10696\/revisions\/12772"}],"up":[{"embeddable":true,"href":"https:\/\/origo.systems\/info\/wp-json\/wp\/v2\/pages\/7856"}],"wp:attachment":[{"href":"https:\/\/origo.systems\/info\/wp-json\/wp\/v2\/media?parent=10696"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}