15 | Written on Fri 31 January 2020. Posted in Tutorials | Richard Walker
Another stepping stone towards understanding developing for OpenShift is Source-to-Image.
s2i is a tool provided by Red Hat that enables you to include additional information in your source code project, to build a container image and run your code on-demand.
This guide assumes you have done the deckofcards and handofcards coding projects in this series. This guide also builds on from the previous podman exercises.
The Source-to-Image
s2i buildcommand has a dependency on a local Docker service. This guide uses RHEL 8 and Podman (no Docker) explicitly. Therefore, there is a subtle difference using the--as-dockerfileoption and an extrapodman buildstep.
I'll admit it took some time for me to make sense of Source-to-Image. In a nutshell, it's nothing more than an opinionated approach to building container images.
Starting with a clean base image, s2i builds an application builder image which typically contains standard packages. For example, you might take a minimal RHEL8 base image then create a new image, adding Python, PIP and use PIP to install Django.
This new Python/Django builder image can be then consumed, only needing to add your code and a run script, reducing the time to deploy an application. The less extra stuff to do in the final application image builds, the better.
── Base Image
└── Builder Image
└── Application Image
You can think of the process as a basic build pipeline. Most enterprises already have more sophisticated build pipelines in place. A term often used is an "Image Bakery". Final application images ready for consumption in some image registry.
However, Source-to-Image comes out-of-the-box with OpenShift and provides a convenient method for developers to override assemble and run scripts. The scripts, as mentioned earlier, can be included in a source code repository along with source code. Source-to-Image empowers the developer to include additional packages and custom commands while still consuming standard base or custom build images.
Environment
This guide uses a fresh installation of Red Hat Enterprise Linux 8 as the base operating system.
$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.1 (Ootpa)
Install Podman, Buildah and Skopeo:
sudo dnf install podman buildah skopeo -y
podman --version
podman version 1.4.2-stable3
buildah --version
buildah version 1.9.0 (image-spec 1.0.0, runtime-spec 1.0.0)
skopeo --version
skopeo version 0.1.37
Your client host needs to be registered and attach to a valid subscription for accessing ubi images.
Install Source-to-Image
The s2i tool itself can is a binary that is simply download (https://github.com/openshift/Source-to-Image/releases) and added to your local environment PATH.
sudo yum install wget -y
wget https://github.com/openshift/Source-to-Image/releases/download/v1.2.0/Source-to-Image-v1.2.0-2a579ecd-linux-amd64.tar.gz
tar -xvf Source-to-Image-v1.2.0-2a579ecd-linux-amd64.tar.gz
If not already present, create a bin directory in your users home directory:
mkdir ~/bin
Move the s2i binary in to ~/bin:
mv s2i ~/bin
And refresh you user environment:
source ~/.bashrc
s2i version
s2i v1.2.0
Explore Python UBI
Using valid Red Hat customer portal credentials login to the registry:
podman login registry.redhat.io
Make sure you do have access to
registry.redhat.iovia proxies or VPN etc.
You can search the registry:
podman search registry.redhat.io/ubi8
And pull base images as needed.
podman pull registry.redhat.io/ubi8/python-36
podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.redhat.io/ubi8/python-36 latest 88189ceb8a67 5 weeks ago 825 MB
You could start with the s2i-base image and add, in this case, the needed Python packages. However, as you'll see from the registry search, there are already base images for all the most popular languages.
Our example is concerned with the python-36 image.
You can validate that this is indeed an s2i primed image:
podman inspect --format='{{ index .Config.Labels "io.openshift.s2i.scripts-url"}}' registry.redhat.io/ubi8/python-36
Explore the container:
podman run --name test -it registry.redhat.io/ubi8/python-36 bash
You'll find the following directories and files are present:
/usr/libexec/s2i/assemble
/usr/libexec/s2i/run
/usr/libexec/s2i/usage
And the Python and PIP are installed:
(app-root) python -V
Python 3.6.8
(app-root) pip -V
pip 9.0.3 from /opt/app-root/lib/python3.6/site-packages (python 3.6)
Basic Django example
Starting with a blank canvas, create a new working directory:
mkdir ~/source-to-image
cd ~/source-to-image
Use s2i create <image_name> <directory> to start a new project:
s2i create s2i-django s2i-django
cd s2i-django
The command create the following skeleton structure:
.
├── Dockerfile
├── Makefile
├── README.md
├── s2i
│ └── bin
│ ├── assemble
│ ├── run
│ ├── save-artifacts
│ └── usage
└── test
├── run
└── test-app
└── index.html
You can remove the test directory.
rm -rf test/
The first task is to create your builder image by editing the Dockerfile and including the most common packages required for any subsequent build.
The following Dockerfile is the most basic example. It adds the minimum LABELS used by OpenShift and updates existing packages. Because we're using the python-36 base image, the only addition is the installation of Django, thus making a simple builder image for deploying Django projects.
Example:
vi Dockerfile
FROM ubi8/python-36
LABEL maintainer="Richard Walker <[email protected]>"
LABEL io.k8s.description="Platform for building Django applications" \
io.k8s.display-name="builder 0.0.1" \
io.openshift.expose-services="8000:http" \
io.openshift.tags="builder,python,python36"
USER root
RUN yum -y update && \
yum clean all
RUN pip install --upgrade pip
RUN pip3 install django
COPY ./s2i/bin/ /usr/libexec/s2i
RUN chown -R 1001:1001 /opt/app-root
USER 1001
EXPOSE 8000
CMD ["/usr/libexec/s2i/usage"]
Any tasks added to the assemble script execute for subsequent application builds, done every time you deploy code using this image. Therefore keeping them to a minimum reduces the build time.
For example, say an application you have created needs some additional packages. Our deckofcards example requires the djangorestframework package. You could add the following to the assemble script:
vi s2i/bin/assemble
...
echo "---> Building application from source..."
pip install djangorestframework
Or because we also know in our handofcards example, it needs Django, requests and psycopg2-binary we might be happy to include all four dependencies at this stage. Or maybe not? We know djangorestframework has a dependency on Django. So the only common package is in fact Django.
In that case, the deckofcards project would need to include its own assemble script to include djangorestframework. And the handofcards project would need its own assemble script to install both requests and psycopg2-binary.
Confused?
There is no right or wrong place to do this stuff. Source-to-Image provides flexibility. The more specific you prepare a builder image for your applications, the less time to build and deploy your code in OpenShift.
It might also make sense to make a builder image specifically for each application.
OK, for our purposes of demonstration and learning, we'll make a single image with the deckofcards project in mind:
vi s2i/bin/assemble
...
echo "---> Building application from source..."
pip install djangorestframework
And in this case, we want to keep things simple and keep using the Python development server to run the application, so add the command to the run script:
vi s2i/bin/run
It's important to include the
0.0.0.0for it to work within a container:
...
python manage.py runserver 0.0.0.0:8000
With the Dockerfile, assemble and run scripts in place, build a new builder base image. Remember your local system needs to be correctly registered and attached for the yum/dnf commands to work during the build:
buildah bud -t richardwalker.dev/django-s2i-base-img .
List and view the new image:
podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
richardwalker.dev/django-s2i-base latest dcc66f9625e0 8 minutes ago 905 MB
We now use this new builder base image to build your final application image on-demand.
Time to stop and consider that the next steps using the
s2iCLI tool are essentially demonstrating what OpenShift does under the hood when using Source-To-Image as a deployment strategy. This guide is mainly showing you how to develop custom S2I images.
We use an s2i command to create a new Dockerfile, specifying both the builder image to use and location of the source code.
Create and move into a separate directory as not to overwrite the existing Dockerfile:
mkdir doc-build
cd doc-build
The following build command can be done using either code directly from a Git repository or a file system location:
s2i build https://github.com/richardwalkerdev/deckofcards.git richardwalker.dev/django-s2i-base-img deckofcards-s2i-img --as-dockerfile Dockerfile
or
s2i build code/ richardwalker.dev/django-s2i-base-img deckofcards-s2i-img --as-dockerfile Dockerfile
It's this step in the process that
s2iin the previous OCP3 and on RHEL7 with Docker, would build the final image in one go. Using the newer Podman tooling introduces the--as-dockerfile Dockerfilerequirement and thus the need for an additionalpodman buildstep coming up.
The previous command gathers all the artefacts needed to build the final application image. In a nutshell, this includes the source code and any script to overwrite (for example run or assemble:
.
├── Dockerfile
├── downloads
└── upload
Content to be copied in to the final image are under the upload directory.
Contents of this Dockerfile:
FROM richardwalker.dev/django-s2i-base
LABEL "io.openshift.s2i.build.commit.author"="Richard Walker <[email protected]>" \
"io.openshift.s2i.build.commit.date"="Sat Feb 1 09:28:45 2020 +0000" \
"io.openshift.s2i.build.commit.id"="50934b79c2b27bd8adee027c7fa4e46335db8753" \
"io.openshift.s2i.build.commit.ref"="master" \
"io.openshift.s2i.build.commit.message"="removed database settings" \
"io.openshift.s2i.build.source-location"="https://github.com/richardwalkerdev/deckofcards.git" \
"io.k8s.display-name"="deckofcards-s2i" \
"io.openshift.s2i.build.image"="richardwalker.dev/django-s2i-base"
ENV DISABLE_COLLECTSTATIC="1" \
DISABLE_MIGRATE="1"
USER root
# Copying in source code
COPY upload/src /tmp/src
# Change file ownership to the assemble user. Builder image must support chown command.
RUN chown -R 1001:0 /tmp/src
USER 1001
# Assemble script sourced from builder image based on user input or image metadata.
# If this file does not exist in the image, the build will fail.
RUN /usr/libexec/s2i/assemble
# Run script sourced from builder image based on user input or image metadata.
# If this file does not exist in the image, the build will fail.
CMD /usr/libexec/s2i/run
Examine the FROM image and the COPY, RUN and CMD.
The application container image can now be built:
podman build -t richardwalker.dev/deckofcards-s2i-img .
podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
richardwalker.dev/deckofcards-s2i-img latest 3ed400d347c3 6 seconds ago 907 MB
richardwalker.dev/django-s2i-base-img latest 852b31d3d9dc 13 minutes ago 903 MB
Run a container from the image, take note of the port mapping:
podman run --name deckofcards-s2i-ctr -u 1001 -d -p 7000:8000 richardwalker.dev/deckofcards-s2i-img
You can get into the running container to check things out by obtaining it's CONTAINER ID from podman ps, for example:
podman exec -it c2cb8cf364da bash
or using its name:
podman exec -it deckofcards-s2i-ctr bash
Finally, test the application:
curl 127.0.0.1:8000/deck/
Recap
Let's recap and conceptualise the process once again. A "base image" is used to create a "builder image". I'm calling it a "builder image", but really its just a custom base image.
Base images include the most common of requirements. In our case, we use the Python base image because Django is a Python framework. The "Builder image" adds the next most common requirements. The assemble script can include other package installations, and the run script can include extra commands to start and run the application. Using s2i to build the final "application image" has an awareness of any ./s2i/ files and scripts that may or may not be present in the source code repository. If present, s2i uses them to overwrite existing scripts in the "builder image". If absent, it keeps the original scripts provided in the "builder image".
The convenience of Source-to-Image is that it puts some control back into the hands of the developer. run and assemble scripts can be included with their source code to override those provided in the builder image. In our django-s2i-base-img image, we added a pip install in assemble and python manage.py runserver in the run file.
dnf install qemu-kvm qemu-img libvirt virt-install libvirt-client virt-manager
Because the handofcards is now configured to use PostgreSQL, we can use the same image but include additional database migration commands to the run script. Let's do that next.
Deploying handofcards
Create another working directory:
mkdir ~/source-to-image/s2i-django/hoc-build
cd ~/source-to-image/s2i-django/hoc-build
Create a directory to copy the source code into:
mkdir code
And copy your source code into it:
cp -R ~/Django/handofcards-master/handofcards-project/* code/
We know that the handofcards project has different package requirements and it need to do the migration commands when starting.
Create a .s2i/bin directory under code/:
mkdir -p code/.s2i/bin
Add an assemble script:
vi code/.s2i/bin/assemble
#!/bin/bash -e
if [[ "$1" == "-h" ]]; then
exec /usr/libexec/s2i/usage
fi
if [ "$(ls /tmp/artifacts/ 2>/dev/null)" ]; then
echo "---> Restoring build artifacts..."
mv /tmp/artifacts/. ./
fi
echo "---> Installing application source..."
cp -Rf /tmp/src/. ./
echo "---> Fixing persmissions..."
chmod -R 775 ./
echo "---> Building application from source..."
pip install django
pip install psycopg2-binary
pip install requests
And add a run script:
vi code/.s2i/bin/run
#!/bin/bash -e
python manage.py makemigrations
python manage.py migrate
python manage.py runserver 0.0.0.0:8000
Now we'll use s2i to generate the Dockerfile using our original django-s2i-base-img
s2i build code/ richardwalker.dev/django-s2i-base-img handofcards-s2i-img --as-dockerfile Dockerfile
You'll now see that the previous command included the assemble and run scripts:
ll upload/scripts/
assemble
run
And finally, create the new application image:
podman build -t richardwalker.dev/handofcards-s2i-img .
Testing
Before testing the handofcards-s2i-img image, we'll need our postgres container running.
To start up a postgres container independent of our previous podman pod examples use the following:
podman run --name pgdb-standalone -e POSTGRES_PASSWORD=password -d -v "/home/user/volumes/postgres/django:/var/lib/postgresql/data:z" -p 5432:5432 postgres
I'm demonstrating a subtle difference while running these containers. We've started them up individually, that is, not a member of the same pod. Therefore the IP Address needs to reflect the IP Address of you're localhost:
podman run --name handofcards-s2i-ctr -u 1001 -d -p 8000:8000 -e DECKOFCARDS_URL=http://192.168.122.21:7000/deck/ -e DATABASE_NAME=cardsdb -e DATABASE_USER=dbuser -e DATABASE_PASSWORD=changeme -e DATABASE_HOST=192.168.122.21 -e DATABASE_PORT=5432 richardwalker.dev/handofcards-s2i-img
````
### Generic S2I image
To build a more universal image it might make more sense to utilize `requirements.txt`.
Why not add the following into your base image:
```text
if [[ -f requirements.txt ]]; then
echo "---> Installing dependencies ..."
pip install -r requirements.txt
fi
Logging - a side note
It's standard practice to have application direct their logs to standard out. One trick is to add RUN tasks in Dockerfile that creates soft links. For example:
# forward request and error logs to collector
RUN ln -sf /dev/stdout /var/log/nginx/access.log
RUN ln -sf /dev/stderr /var/log/nginx/error.log
Conclusion
It might be argued that Source-to-Image doesn't do anything particularly special other than stage a series of regular buildah commands.
That said, it does provide a standard approach in how to prepare and use Source-to-Image deployment strategies with OpenShift.
I guess, creating your library of S2I builder images with good standard package inclusion would result in more efficient S2I builds.
I feel having this grounding, at this level helps prepare for better understanding when working with deployments in OpenShift.
Using S2I in OpenShift does have some auto-detection logic, providing source code includes standard features, it can detect if source code is Java or Python, for example, selecting the appropriate s2i base images accordingly.
However, I believe it's better to have sound knowledge of building and to customise your images and be in control of what is happening under the hood.
Next up, it's time to deploy an OpenShift cluster either in AWS or using local CodeReady containers.